Hellzapoppin’ – Retimed, Enhanced, now in 4K

Video upscaling software seems to be getting very popular these days – and I was blissfully unaware until the folks at Estiloswing blew everyone away with an incredible-looking upscaled version of the HD Hellzapoppin’ routine version which Atilio Menéndez found a while back. Of course… there was nothing for it but to revisit my retiming project, and see if I could use similar techniques to enhance it.

All I needed to do was find out what tools to use, run the original video through it, drop that into my existing project! Easy, yes?

Cue the deranged laughter. No. Nothing is ever that simple. Many hours later, many tens of hours of dedicated computer processing time, at least four start-from-scratch versions, over a dozen “I think I’ve finally finished” versions, the lucky unearthing of another HD copy of the full film, infinite patience on the part of my wonderful and long-suffering partner, a few more lost hair follicles….

And yes – I believe I do – finally – have something to show for it.

It all Started…

Going back a little – back in 2018, I got it into my head that there was something wrong with the Hellzapoppin’ routine, and that I needed to fix it. A few things bothered me about it – starting with the dancers all clapping to the music on the on-beat, some of the big moves in the routine not quite hitting the main beats of the music, dancers coming on a slightly odd parts of a phrase.

Frankie had the following to say about the music – which was changed by the studio from the original choice of Basie’s Jumpin’ at the Woodside:

On the second day [of studio rehearsals], the musical director at Universal started sitting in on rehearsals to listen to the music and watch us dance. His job was to create a composition for our number so the studio wouldn’t have to use Count Basie’s song. As we went through the routine repeatedly, he started writing an original score that fit the choreography and really emphasised certain movements that we did. Within a week, he had finished, and the studio made a recording of his composition, which is what we rehearsed to from then on. We used that when we filmed the scene and, as far as I can tell, that’s what’s on the soundtrack of the movie.

Frankie Manning – Ambassador of Lindy Hop

So it’s clear that they danced very closely with the music we hear. This list went on – and if you need a “smoking gun” to point to it being hacked around somewhat, here’s two adjacent frames from the last section:

These two frames are literally side-by-side. Clearly something is missing – and sadly this actually means that the scene isn’t really truly fixable – you can’t add footage that isn’t there. So had to get a little creative.

Disclaimer – what I’ve produced is not exactly what was danced. I don’t believe that’s possible without magically finding everything that was left on the cutting room floor (side note – Frankie said that the takes that made it into the film, incredible though they were were not in his opinion the best of the takes that were filmed – I daydream that somehow that all the footage that was shot but didn’t make the final cut was saved somewhere, and somehow sees the light of day again, impossible thought that may seem. Hand that to me, and I’ll be spending a long time in an editing room somewhere).

TLDR: I’ve just tried to get as close to what feels right as I can.

My Goals

I really started off with two goals – get everyone clapping on the correct beats – how the filmmakers decided that clapping on the on-beat was the right way to go is just beyond me – and if I could accomplish that little miracle, have them all start on phrase – which was something Frankie specifically talks about in his autobiography. Beyond that, where possible, try to make the dancing fit the music – so match stand-out moments with appropriate musical moments. And finally – coming to the point of the latest set of work – make it all look as good as I possibly could.

Evolution of Hellzapoppin’ Quality.

In the beginning was the DVD version. Which was a terrible quality DVD (I have a copy, and it’s such bad transfer that these days I can’t bear to watch it) – I used this the first time I did a retime of the routine – because that’s all we had.

Then a miracle happened. Atilio Menédez managed to get a fantastic quality version at 1080p. The difference was incredible, and we were able to see a whole host of details in the video which we’d not spotted before – it made such a difference to the routine… and, obviously, it was compulsory for me to retime that one.

Finally, the folks at Estiloswing started playing with upscaling technology, and produced a version which was twice the resolution, and double the frame rate – so it looked much clearer, and far smoother. There was nothing else for it, but to try and use similar techniques for my own retimed version. I think the video below should illustrate that evolution pretty well.

Left hand side – as taken from the DVD. Middle – Atilio’s initial 1080p discovery. Right – my own final 4K / 60fps version – which you should see is much clearer, and much smoother (you might need to watch on YouTube for full effect).

Rise of the Machines

What makes this possible at all is the advancing use of Machine Learning (often, of course referred to as “artificial intelligence”). To upscale a video, one is effectively creating parts of it that don’t currently exist. This is difficult. The modern techniques involve (simplified description) teaching a computer to do it, by giving it a whole bunch of “before” and “after” examples that it can learn from, then telling it to do the same thing on the video you want to upscale. Luckily, the first, hardest part has been done for us, leaving us with a choice of software and methods to use. In theory, all I needed was a really powerful computer (check) with the right graphics card (oops) and a huge amount of time (lockdown is at least good for something).

Resolution (Making it Bigger)

I ended up going with a superb program: Video Enhance AI from Topaz. This piece of software is incredible. You feed it your video, choose the right mode, tweak a few things, preview the result, wait a few hours. Then look at the result, swear a bit, choose a different mode, tweak a few things, preview it, wait. Rinse and repeat.

When it goes right, it’s incredible.

• 

• 

When it goes wrong…. it doesn’t. Get things wrong, and it can pull tricks like faces into plasticine models, or worse. I realised that even though the original 1080p copy was incredible, it would be nice to have an even better start point – and as the original one was a result of a screen capture, maybe there was a better way of capturing the scene. So I started looking, yet again, for a full HD copy of the film – a generally fruitless task, since it’s never been available as a blu ray, and only appeared in viewable form online on one site, for about a month.

To my amazement, I found one. I have managed to buy a copy of the full film, in high definition.

Even stranger – it seems to be a different transfer from the original that Atilio obtained, with different good and bad points. To cut a very long story short, after trying to upscale both copies I got in touch with Atilio, and he suggested combining the two versions to get the best of both. He then did exactly that, and cleaned up the image quality, tweaked the lighting, and produced a small miracle. Essentially, he combined this:

With this:

To produce this.

And we were in business.

Framerates

The other big part of the upscaling adventure was to improve the frame rates. The scene was, of course, filmed at 24 frames per second (well – 23.97) – the standard for cinema. The goal was to view it at a much smoother 60fps. To do this, you have to generate “intermediate” frames – and the main methods for doing this boil down to:

1. Don’t.
2. Frame blending – make each intermediate frame a blend of the ones either side of it.
3. Optical flow – have the machine try to guess what the intermediate frames should be.
4. AI interpolation – the same, but with machine learning.

When it works, option 4 is the best, and there are two main methods I tried.

1. DAIN – Depth-Aware video frame INterpolation. This does the best job of guessing what those intermediate frames should be. Except when it doesn’t. It detests, for example, some of the curtains in the Hellzapoppin’ dance clip. It’s also very very expensive to run – and I have nothing at home capable of running it. Luckily, it’s possible to rent time on machines to do stuff like this.
2. RIFE – Real-time Intermediate Flow Estimation. Much faster (hence cheaper) to run, smoother results, perfect for the job – except the results aren’t quite as good as DAIN, and it can’t stand really fast moving objects or fast camera movements – and it really hates the two combined. Guess what Hellzapoppin’ is filled with? Really fast moving people, sometimes combined with fast camera movements.

Curtains

Did I mention that DAIN doesn’t like the curtains? Not all curtains. Just these curtains.

RIFE loves the curtains. Fast moving objects / people / cameras? Not so much…. this slowed down RIFE interpolated clip shows off RIFE’s shortcomings in a wonderfully freaky way.

Just as well there’s no sections in the routine with fast-moving cameras and the dodgy curtains, you might think…. (but you’d be wrong – just don’t get me started on that one)

To cut a long story short, the way to address stuff like this was to run pretty much all the FPS upscaling methods, and merge them all where appropriate – there’s even moments when all of them are being used at the same time. The result is mostly done by DAIN (an estimated 18-20 hours computer time to process the whole routine) – but all of them play a role.

Putting it all Together

To finally cut to the chase. This took a while. A few other little things probably aren’t worth going into detail over – like hand editing of eight specific frames to fix a probably underwear malfunction (no, I’m not putting up before and after pictures of that one – you can find it yourself it you really want) or the discovery that my monitor’s brightness was turned up way too much, meaning I’ve been churning out videos that are much too dark – there’s a never ending list, but by now, you probably just want to see the video…

Overall though, it has been fun. There’s a lot of truth to the saying that a project like this is never finished – it just reaches a point you stop working on it… but I think this one has at least reached a good point to stop at.

I’ll let you be the judges. For now, or until someone finds an 8K version of the film, I think this project can rest.

A huge thank you to Atilio Menédez, for finding that first HD version in the first place, and for his patient advice and ninja editing skills. Also to Nick Rossi for keeping me straight on the history side of things – and to all the other folks who’ve helped out along the way, watched the endless versions and lent me their advice. It has all been very much appreciated.