In context: Deepfake technology does not necessarily have to be used for nefarious purposes. Facebook is adapting the technique to retouch photos where the subject blinked to make it look like their eyes were open.
As machine learning continues to advance, we see it used in more creative ways each day. Whether it is improving the quality of our voice assistants, smoothing out a slow-motion video, or predicting heart disease, machine learning applications are on the rise. Not one to be left behind, Facebook has been working on algorithms that can alter a selfie ruined by an accidental blink.
Facebook researchers Brian Dolhansky and Cristian Canton Ferrer published a paper titled “Eye In-Painting with Exemplar Generative Adversarial Networks,” which outlines a machine-learning technique that can open the eyes on a picture of someone who blinked when the photo was snapped. The researchers point out that there are plenty of photo-retouching tools that fix things like red eye, and blemishes. Making one that takes care of poorly timed blinks would be a nice addition to photo editing tools.
The technique uses generative adversarial networks (GANs), or more specifically "exemplar GANs" (ExGANs) to essentially paint on (or in-paint) a subject’s eyes drawing from a number of photos of the person with their eyes open. The network “trains” on these photos to recreate photo-realistic looking eyes on the picture.
“We show that ExGANs can produce photo-realistic personalized in-painting results that are both perceptually and semantically plausible by applying them to the task of closed-to-open eye in-painting in natural pictures,” Dolhansky and Ferrer write.
The results are most effective with straight on shots. However, the system is a work-in-progress and still struggles with some aspects of certain photos such as eyeglasses, profile or semi-profile shots, and long fringes. I'm not entirely convinced though. Most of the above examples have an unnatural look that I can't quite put my finger on.
The system is only in the research phase right now, and the social media giant does not have a marketing plan for it yet. Other companies such as Adobe and Pixelmator are using machine-learning techniques as well, but it is unclear if they have “blink fixers” in the works.
On the one hand, these advances are making it easier to edit our photos and video. On the other hand, it is also making it easier to tamper with evidence and create fake images and video that didn’t before exist.