Amazon's Prime Video is using AI to make dubbed actors' mouths match the dialogue

Skye Jacobs

Posts: 2,178   +62
Staff
First look: Prime Video is using AI to make English dubs look less like dubs. The company is starting with Maxton Hall, using AI and visual-effects tools to match actors' lip movements to the show's English dub. The rollout gives Amazon an early test of whether post-production AI can make foreign-language programming feel more natural to viewers without replacing traditional dubbing.

The feature is available worldwide for the show's first two seasons and will be included with the third and final season, which premieres in early December.

The technology works alongside human-recorded dubbing. Prime Video said it combines AI and visual-effects tools to adjust lip movements after translating and recording the dialogue. The result is meant to make the English audio align more closely with what viewers see on screen.

For now, Maxton Hall is the only series using the feature. Prime Video said it plans to bring lip-synced dubbing to additional titles but did not say which programs are next or offer details about the AI systems behind it.

The rollout gives Amazon a high-profile test case. Maxton Hall, a German-language romance, is Prime Video's most-watched international original series. Its upcoming third season is based on Mona Kasten's 2018 novel Save Me.

Prime Video began testing AI-assisted dubbing last year across 12 movies and series in English and Spanish. Those titles included El Cid: La Leyenda, Mi Mamá Lora, and Long Lost. The Maxton Hall release goes further by changing the video itself to better fit the translated audio.

"With lip-syncing, we're creating a more seamless and immersive way for customers to enjoy global content," Raf Soltanovich, vice president of technology at Prime Video and Amazon MGM Studios, said in a statement. "And for creators, we're making it easier for their stories to reach an even broader and global audience, all done under creative oversight to ensure the integrity of their artistic vision is preserved."

The company did not explain how much of the process is automated or what specific AI technology it uses. It said the feature was developed using a combination of AI and VFX tools and will operate under creative oversight.

Prime Video is entering a growing market for AI translation and dubbing tools. Meta and YouTube have each introduced AI dubbing features that can translate creator videos into other languages. Both platforms also offer lip-syncing options designed to make translated speech appear more natural.

Amazon's approach differs in one important way: It is pairing the visual adjustments with human dubbing for a scripted series. The company is not replacing voice actors with automated voices. Instead, it is using AI and post-production techniques to address the visible gap between translated dialogue and the performances originally filmed in German.

Permalink to story:

 
And yet, we still have ignorant people saying that A.I is not taking people's jobs away....
Can you articulate which job, exactly, this is taking away? Re doing lip syncing to match dubs is not something that has been done in any sort of regular capacity before. And since this is new, and someone has to make the AI do it, you could argue this AI has CREATED a job.

Perhaps it is time to address your AIDS (A. I. Derrangement Syndrome) before you make yourself look like more of a fool.
 
So is the AI reshaping their mouths in the video or somehow making the speech match the mouth more?
Yes - it's dubbed so the lips wouldn't sink to the speech properly at all. So they are making the mouths match the dubbed dialog. This is a perfect use for AI. Doing something that couldn't be done properly any other way.
 
Nice, now do the news using geofenced models to control the flow of information regionally.

Oops, pulled back the curtain too soon!
 
Who listens to dubs? Listen to it in its native tongue and read the subtitles like an adult.
Only children cry about dubs, thinking that reading makes them more mature.
Nice, now do the news using geofenced models to control the flow of information regionally.

Oops, pulled back the curtain too soon!
What does that have to do with AI changing lip movement? This is a total non-sequitur.
 
Yes - it's dubbed so the lips wouldn't sink to the speech properly at all. So they are making the mouths match the dubbed dialog. This is a perfect use for AI. Doing something that couldn't be done properly any other way.
Seems like a good use for AI.
Assuming it looks right. Otherwise, it's just trading one weird mouth movement for another.
 
Back