YouTuber Denis Shiryaev has unveiled his most spectacular AI video restoration yet: a new montage of footage shot by the Lumière brothers.
Shiryaev spent months using machine learning to colorize, upscale, and enhance a collection of films by the French cinema pioneers.
He began by using an algorithm developed by his company, Neural Love, to remove duplicate frames from the footage and restore the film’s original frame rate.
“This is a crucial step to achieve the effect of fluid movement when we do a version of the video at 60 frames per second (fps),” Shiryaev explained on YouTube.
[Read: Meet the 4 scale-ups using data to save the planet]
He also applied stabilization to the footage, removed noise where required, and corrected some of the damage to the films.
Shiryaev then used neural networks to upscale every frame to 4K resolution, generate more frames until he reached 60 fps, and restore a realistic playback speed to the video.
He also colorized the footage and added detail to the faces with a generative adversarial network (GAN) based algorithm.
Finally, he added new sounds to the film:
The sounds are not real, and sometimes I did a terrible job. But I really like this ambient effect when, for example, you can see and hear a dog starting to bark on screen and experience the moment as if you were there.
You can check it out for yourself in the video below:
The video joins Shiryaev’s growing filmography of AI-powered restorations. But not everyone approves of his work.
Academics warn that it creates a distorted image of the past and undermines the choices of the original photographer. They fear that every added effect also takes something away from what was captured at the time.
Shiryaev acknowledges that the videos are not historically accurate, and only intended for entertainment or artistic purposes:
Neural networks can only guess colors when we’re doing colorization, or new pixels when we’re doing upscale, and faces when working with some facial restoration.
Nonetheless, the artificial sights and sounds provide a new window to the past — even if it is a pretty cloudy one.
Published January 5, 2021 — 14:56 UTC