The takeaway: Microsoft, Sony, and other companies have filed several patents in the past year that use generative AI to alter content in real time. Patents don't always become products, but the frequency of the filings demonstrates strong interest in real-time AI editing. Microsoft says its latest patent is only for cleaning up and translating broadcasts, but the basic concept raises concerns about accuracy and misinformation.
AI-edited and AI-generated videos have been among the most popular generative AI applications since the technology's initial explosion in popularity. However, a Microsoft patent filing from late last month, recently uncovered by Patentlyze, aims to enhance how AI can edit real-time streaming.
The first claim describes an AI that listens for certain words in a broadcast's audio track and replaces them with new words based on information from the viewer's profile, such as age, location, and native language. Microsoft claims the AI would remove pauses and filler words to clean up the presentation, but it could also censor profanity and replace words based on internet searches.
Although the claim could also support instant fact-checking, it doesn't say whether it would tell users which edits were made or why. Furthermore, its ability to hide edits by adjusting lip movements in the intervening frames suggests viewers could end up watching AI-edited streams without realizing it. Real-time AI face swapping is already popular with scammers, so the ability to alter what is being said in a livestream could take misinformation and fraud to new levels.
Meanwhile, a separate, independent claim describes using the technology for translation. YouTube already supports AI audio translation, but Microsoft's patent would translate livestreams with matching lip sync. Moreover, unlike the first claim, the translator retains the original recording as a "ground truth."
These patents share a basic idea with others that involve changing the user experience in real time with AI. For example, an ambitious claim Microsoft filed in July aims to rewrite a game's code to add content based on player behavior.
Meanwhile, Sony recently filed multiple patents that would enhance game performance and reduce input latency by using AI to predict upcoming frames and which button a user will press next. Another would use AI to control game time progression, difficulty, and other aspects, or even fill in for players when they temporarily leave online sessions.
Microsoft patent filing describes AI that rewrites what people say in livestreams in real time
