What just happened? Top-tier video editing suites can seamlessly remove objects from scenes, even generating realistic shadows and reflections for the freshly removed elements. However, these tools fall short when the deleted object involves significant interactions, such as collisions. In such cases, existing solutions often struggle to produce plausible results. Netflix is addressing this shortcoming with a new video object removal framework called Void.
Short for Video Object and Interaction Deletion, the model can effectively delete an object from a scene and adjust for its absence.
For example, erasing a car crash from a scene will also modify the remaining elements accordingly as if the accident never happened. This means that flying debris, fire, and damage to nearby props will be removed as if the crash never occurred. Similarly, in a scene involving someone cannonballing into a pool, removing the person would leave the pool water naturally undisturbed.
To train the model, its creators used Kubric and Humoto to generate a new paired dataset of counterfactual object removals. During the inference stage, a vision-language model is used to identify parts of the scene impacted by a removed object which serves as a guide for the diffusion model to fill in the blanks with the counterfactual data.
Void sounds like a powerful video editing tool that could afford producers lots of flexibility long after filming has wrapped up. Not having to reshoot a scene would save an immense amount of time and money – assuming of course that the effect is seamless and doesn't look like AI slop.
Interested parties can learn more about Void over on GitHub. Its creators – Saman Motamed, William Harvey, Luc Van Gool, Benjamin Klein, Ta-Ying Cheng, and Zhuoning Yuan – have also published a 19-page pre-print (PDF) on the subject.
As The Register highlights, the model isn't exclusive to Netflix. The streaming giant has also made it available on Hugging Face, meaning anyone can install and use it. And while Void isn't the first of its kind, it might be the best currently available.
In a survey of 25 individuals cited by The Register, Void was reportedly preferred over rivals like ProPainter, Rose, DiffuEraser, and Generative Omnimatte nearly 65 percent of the time.