As AI programs continue to demonstrate their ability to manipulate media, the line between real videos and fake ones continues to blur. Deepfake movies with alternate (and creepy) cast members, for example, show how seamlessly AI can manipulate faces. Now, there’s a new AI that’s able to take videos and, purely based on what it sees, add sci-fi objects like alien spaceships and floating sky castles.
The AI tool, which offers “dynamic sky replacement,” is the brainchild of AI researcher Zhengxia Zou. Zou, a postdoc research fellow at the University of Michigan, Ann Arbor, says his program is able to execute “sky replacement and harmonization” in order to seamlessly generate realistic and “dramatic” sky backgrounds.
The program utilizes neural networks—algorithms capable of gleaning relevant patterns from large datasets—in order to perform the aerial magic. In other words, the sky replacement program has learned what a “sky” looks like in videos, and is able to identify and track it. It’s then able to replace it with new sky, which not only tracks perfectly, but is lit perfectly as well.
Zhengxia Zou
In practice, Zou’s AI is able to replace, it seems, any sky in any video with an alternate one. In Zou’s video up top, he shows how he can add not just spaceships and flying castles, but pretty much anything to source videos. Planets are seemingly no problem, and dynamic skies—such as ones filled with lightning—are also possible.
One of the most critical details of Zou’s program is that it is completely vision based. This means the program only needs a source video, and nothing else, in order to make its edits. This is in contrast to other programs that can only do sky replacements in stills; or can edit moving-sky videos, but only if inertial measurements are available. (YouTuber Two Minute Papers explains this in further detail in the video below.)
Moving forward, it’s easy to imagine all kinds of applications for Zou’s AI. Not only will it probably be a social media-app feature soon, but Zou says it’ll help train self-driving cars. And if there’s one thing every self-driving car should know, it’s how to spot a spaceship. It is 2020, after all.
Feature image: Zhengxia Zou
