A crew at Tencent’s Hunyuan lab has created a brand new AI, ‘Hunyuan Video-Foley,’ that lastly brings lifelike audio to generated video. It’s designed to hearken to movies and generate a high-quality soundtrack that’s completely in sync with the motion on display.
Ever watched an AI-generated video and felt like one thing was lacking? The visuals is likely to be beautiful, however they typically have an eerie silence that breaks the spell. Within the movie business, the sound that fills that silence – the rustle of leaves, the clap of thunder, the clink of a glass – is named Foley artwork, and it’s a painstaking craft carried out by specialists.
Matching that stage of element is a large problem for AI. For years, automated techniques have struggled to create plausible sounds for movies.
How is Tencent fixing the AI-generated audio for video downside?
One of many largest causes video-to-audio (V2A) fashions typically fell brief within the sound division was what the researchers name “modality imbalance”. Basically, the AI was listening extra to the textual content prompts it was given than it was watching the precise video.
As an example, for those who gave a mannequin a video of a busy seaside with folks strolling and seagulls flying, however the textual content immediate solely stated “the sound of ocean waves,” you’d doubtless simply get the sound of waves. The AI would utterly ignore the footsteps within the sand and the calls of the birds, making the scene really feel lifeless.
On high of that, the standard of the audio was typically subpar, and there merely wasn’t sufficient high-quality video with sound to coach the fashions successfully.
Tencent’s Hunyuan crew tackled these issues from three completely different angles:
- Tencent realised the AI wanted a greater schooling, in order that they constructed a large, 100,000-hour library of video, audio, and textual content descriptions for it to be taught from. They created an automatic pipeline that filtered out low-quality content material from the web, eliminating clips with lengthy silences or compressed, fuzzy audio, making certain the AI discovered from the very best materials.
- They designed a wiser structure for the AI. Consider it like instructing the mannequin to correctly multitask. The system first pays extremely shut consideration to the visual-audio hyperlink to get the timing good—like matching the thump of a footstep to the precise second a shoe hits the pavement. As soon as it has that timing locked down, it then incorporates the textual content immediate to know the general temper and context of the scene. This twin strategy ensures the particular particulars of the video are by no means neglected.
- To ensure the sound was high-quality, they used a coaching technique known as Illustration Alignment (REPA). That is like having an knowledgeable audio engineer continuously wanting over the AI’s shoulder throughout its coaching. It compares the AI’s work to options from a pre-trained, professional-grade audio mannequin to information it in the direction of producing cleaner, richer, and extra secure sound.
The outcomes converse sound for themselves
When Tencent examined Hunyuan Video-Foley in opposition to different main AI fashions, the audio outcomes had been clear. It wasn’t simply that the computer-based metrics had been higher; human listeners persistently rated its output as larger high quality, higher matched to the video, and extra precisely timed.
Throughout the board, the AI delivered enhancements in making the sound match the on-screen motion, each by way of content material and timing. The outcomes throughout a number of analysis datasets assist this:
Tencent’s work helps to shut the hole between silent AI movies and an immersive viewing expertise with high quality audio. It’s bringing the magic of Foley artwork to the world of automated content material creation, which could possibly be a robust functionality for filmmakers, animators, and creators in all places.
See additionally: Google Vids will get AI avatars and image-to-video instruments
Need to be taught extra about AI and large knowledge from business leaders? Try AI & Big Data Expo going down in Amsterdam, California, and London. The great occasion is a part of TechEx and is co-located with different main expertise occasions, click on here for extra data.
AI Information is powered by TechForge Media. Discover different upcoming enterprise expertise occasions and webinars here.
