The next generation of AI-powered videos is about to hit the public domain as Google announces a new tool that can automatically create unique soundtracks.
Several AI-generated video creators have impressed users for years, such as OpenAI’s Sora, Runway Gen-3 Alpha, and Luma AI’s Dream Machine.
But none of these magic makers could generate a decent soundtrack to go along with videos – until now.
Google announced the new video-to-audio tool for its DeepMind AI generator on Monday.
“Video Generation models are advancing at an incredible pace, but many current systems can only generate silent output. One of the next one major steps to bringing generated movies to life is creating soundtracks for these silent videos,” Google writes.
“Today we’re sharing the progress of our video-to-audio (V2A) technologymaking synchronized audiovisual generation possible.”
“V2A combines video pixels with natural ones language text prompts to generate a rich soundscape for the on-screen action,” they explained.
The tool can be combined with video generation models such as Veo to create dramatic soundtracks that perfectly match any scene.
The AI will produce music that works with character dialogue and other tonal elements to create the right sonic atmosphere.
“It can also generate soundtracks for a range of traditional footage, including archival footage, silent films and more – opening up a wider range of creative possibilities,” DeepMind said.
Google shared impressive examples of the new technology in action, including clips of a Western-style soundtrack that accompanies a cowboy on a horse and a wild wolf howling at the moon.
FULL CREATIVE CONTROL
Google’s new V2A tool will give creators the power to let AI generate a soundtrack based on a clip’s visual input and linguistic prompts, or design a soundtrack themselves.
Users can provide editing prompts and pointers to the tool to direct its output in the desired direction.
One set of directions reads: “Audio Prompt: Cinematic, Thriller, Horror, Music, Suspense, Atmospheric, Footsteps on Concrete.”
The scene shows a man walking through a ruined building before ending with a view of the same man on an eerie bridge.
AI creates an appropriate soundtrack for the clip that matches the tone and pace of the narrative.
ENDLESS SOUNDTRACK OPTIONS
DeepMind’s V2A can also generate an unlimited number of soundtrack ideas.
An example prompt reads: “Audio prompt: Spaceship floating through the vastness of space, stars passing by, high speed, science fiction.”
The video showed a spacecraft soaring through the vastness of space with the light of a star shining in the distance.
The first soundtrack generated by the V2A tool was an uplifting, orchestral piece that matched the image and prompt.
The second AI-generated soundtrack from the same prompt was darker and slower.
What is Google DeepMind?
Google’s DeepMind project was born in 2010.
“Google DeepMind brings together two of the world’s leading artificial intelligence laboratories – Google Brain and DeepMind – into a single, focused team led by our CEO Demis Hassabis,” according to Google.
“Over the past decade, the two teams have been responsible for some of the biggest research breakthroughs in AI, many of which underpin the thriving AI industry we see today.”
The organization aims to unlock the enormous potential of AI for everyone.
“We are a team of scientists, engineers, ethicists and others working to build the next generation of AI systems safely and responsibly,” they wrote.
“By solving some of the most difficult scientific and engineering challenges of our time, we are working to create breakthrough technologies that could advance science, transform work, serve diverse communities – and improve the lives of billions of people.” “
SOURCE: GOOGLE DEEPMIND
Using Audio Prompt: Ethereal Cello Atmosphere changed things even more.
This third soundtrack immediately set a sadder and more brooding tone.
IT ONLY GET BETTER
Google said these updates are just the latest effort to upgrade its full suite of AI-generated content providers.
They hope to improve some issues in future versions.
“Because the quality of the audio output depends on the quality of the video input, artifacts or distortions in the video that are outside the model’s training distribution can cause a noticeable drop in audio quality,” Google said.
“We’re also improving lip sync for videos that include speech. V2A attempts to generate speech from the input transcripts and synchronize it with the characters’ lip movements.”
“But the paired video generation model may not be driven by transcripts. This creates a mismatch that often results in abnormal lip sync because the video model does not generate mouth movements that match the transcript,” they added.
