AI innovation: Game development is an incredibly time-consuming and expensive process, with art and animation budgets often eating up a hefty chunk of a team's cash reserves. Believable facial animation, in particular, is critical for cutscene-heavy titles. That's why Nvidia is working on an AI-based tool that can read audio files and create matching facial animations in real-time; no mocap is required.
This tech, called "Audio2Face," has been in beta for several months now. It didn't seem to get much attention until more recently, despite its potentially-revolutionary implications for game developers (or simply animators as a whole).
As you'd probably expect from any technology that is both powered by AI and still in beta, Audio2Face isn't perfect. The quality of the source audio will heavily impact the quality of the tech's lip-syncing, and it doesn't seem to do a very good job of capturing facial emotions. No matter what sentence you throw at Audio2Face's default "Digital Mark" character, the eyes, cheeks, ears, and nose all remain fairly static. There is some movement, but it's generally more subdued than the lip animations, which are clearly the main focus.
But perhaps that's a good thing. Conveying accurate emotions in the face of a 3D character is what animators train for years to accomplish. Depending on how easy this tool is to implement into a given developer's workflow, it could provide serviceable, perhaps placeholder lip-sync animations while letting animators focus on other parts of a character's face.
Some of our readers, or simply hardcore fans of Cyberpunk 2077, may recall that CDPR's title used similar technology called "JALI." It used AI to automatically lip-sync dialogue across all of the game's main supported languages (those with both subtitles and voice acting), lifting the burden from the animators themselves.
Audio2Face doesn't have that capability, as far as we can tell, but it still looks useful. If we hear of any instances where a developer has taken advantage of the tech, we'll let you know. If you want to give it a spin yourself, the open beta is available for download, but just know that you'll need an RTX GPU of some kind for it to function properly.