YouTuber uses neural networks to upscale 1896 short film to 4K 60 fps

nanoguy

TS Addict
Staff member

One of the hot topics in tech for the last few years has been AI, with business leaders throwing lots of buzzwords around - machine learning, deep learning, reinforcement learning, neural networks, computer vision, natural language processing.

These are all different approaches that companies implement and mix together to create intelligent services and breathe additional "smarts" into consumer electronics. Examples abound, but you may have heard about that AI developed by Google that is able to detect breast cancer, or that other one that Instagram is using to fight abuse on its platform.

AI's complicated history can be traced back to software developed several decades ago with the purpose of imitating how the human brain learns. Today, many companies use something called neural networks, which are essentially layers of densely interconnected nodes that can be "trained" to recognize patterns by throwing large amounts of data at them and fine-tuning their parameters.

If you're wondering what can be done with them other than say, improving accuracy in autonomous cars, look no further than the above video by YouTuber Denis Shiryaev, who did a cool experiment of turning the famous Lumière Brothers' 1896 short clip "Arrival of a Train at La Ciotat" into a 4K, 60 fps video that looks a lot better than your standard upscaled content.

The original is a modest 640 by 480 video at just 20 frames per second, and is the first example of a number of common cinematic techniques, not to mention one of the earliest attempts at 3D film. As you can see, it's not exactly impressive by today's standards, so Denis used a mix of neural networks from Gigapixel AI and a technique called depth-aware video frame interpolation to not only upscale the resolution of the video, but also increase its frame rate to something that looks a lot smoother to the human eye.

Update: As pointed out by many readers, the upscaled video shown above did not use the original film as the source, but an already upscaled, digitalized version of the same film that had been curated. Here's also a "deoldified" version of the video that adds color.

Supposedly, there is a myth that people who had the privilege of watching it in theaters at the time were startled by the sight of a fast-approaching train that seemed to almost come out of the screen. However, the upscaled version made by Denis showcases this effect quite well, even though it's miles ahead of the original in terms of subjective quality.

The Gigapixel AI used for the upscaling process is developed by Topaz Labs and is being trained to "fill in" information in an image using patterns and structures from a large pool of source images that are downscaled so that results can be compared to the originals. Traditional wisdom is that upscaled content usually loses sharpness and clarity, and also doesn't have any added detail since you're limited by the source image, but Gigapixel AI doesn't rely on interpolation like standard techniques to add in the missing pixels.

It can be argued that this new method might introduce erroneous details and is also much slower (according to Topaz Labs) than conventional techniques (bi-cubic, Lanczos, spline). However, the point is that neural networks can be trained to colorize images and add in various missing patterns in things like floors, windows, vegetation, etc.

There is even an experiment to teach neural networks how to paint, such as one conducted by Libre AI's Ernesto Diaz Aviles, who used PyTorch and Fast.ai libraries to achieve just that. The idea here is to get computers to draw landscapes in much the same way that Bob Ross did in his famous program "The Joy of Painting" where he taught viewers how to use simple techniques in a fun way.

And we've already seen examples of what can be achieved using neural networks to visually overhaul old games like Doom and give game characters more natural and convincing animations.

AI is so big that companies like Google, Apple, Intel, Microsoft, Amazon, and others are all racing to buy or invest in every new and obscure startup in the field to foster talent for their various projects. It's even the subject of ample discussion on the role AI in copyright and trademark law moving forward. It will be interesting to see what's in store for the next decade, and, thanks to Nvidia, we might have a hint - AI-rendered, interactive virtual worlds based on massive libraries of real-world footage.

Permalink to story.

 

Uncle Al

TS Evangelist
Incredible, resolution-wise. But in terms of the frame-rate / smoothness, this can improve further, some movements appear a bit jerky.
True, but there are other applications that can "smooth" that out. There are a whole host of video's out there that can be vastly improved like the first video's of the moon landing and walking on the moon just to name one. I think one of the acid tests would be the B&W of "The Great Train Robbery" that had a huge problem with shadows from being shot in bright sunlight .....
 
  • Like
Reactions: SolarisGuru

Naysayer

TS Rookie
Ok, interesting...but the "original" video linked (which is the only "arrival of a train at la ciotat station 1896" I was able to find" is a completely different film than this "upscaled" example.

it appears to be the same station and a similar train, but with a different set of cars and people. which kind of calls the entire thing into question
 

Polycount

TS Evangelist
Staff member
It's a very impressive achievement. There's obviously some clear artifacts, particularly on faces, but the smoothness is quite good in my opinion. I've seen much, much worse.

The reflections, in particular, look very good.
 
Ignore this post. Pure Troll.

Wonder if that would work on Congress? Artificial Intelligence and that 'upscaling' thing. Make them look like they're doing some work? I mean '1896'. Some of the faction leaders are five or ten years younger than thaaat.'(sic)
 

Lionvibez

TS Evangelist
I'm surprised this story was posted without the comparison video's and only the final project?

here is a digitally updated version not the same as the one posted above.


And here is the original.

 

HyperSloth

TS Rookie
It should be noted that the original quality of the Lumieres brothers film was substantially better than appears in the video. It is only within the past decade that digital has equaled film in quality and a pristine print can be indistinguishable from (or even better than) 4K in quality. In fact, this ai version is probably WORSE than the original projected version because it is based off a damaged and degraded film print. While this is a neat project, the historical take of this article is very wrong.