The race for high-quality AI-generated videos is heating up.
On Monday, Runway, a company building generative AI tools aimed at film and graphic content creators, unveiled the Gen-3 Alpha. The company’s latest AI model generates videos from text descriptions and still images. Runway says the model provides a “major” improvement in generation speed and precision over Runway’s previous flagship video model, the Gen-2, as well as fine-grained controls over the structure, style and movement of the videos it creates.
The Gen-3 will be available in the coming days to Runway subscribers, including corporate clients and creatives in Runway’s Creative Partner Program.
“Gen-3 Alpha excels at generating expressive human characters with a wide range of actions, gestures and emotions,” Runway wrote in a blog post. “It was designed to interpret a wide range of styles and cinematic terminology [and enable] imaginative transitions and precise key framing of elements in the scene.”
The Gen-3 Alpha has its limitations, including the fact that its frames max out at 10 seconds. However, Runway co-founder Anastasis Germanidis promises that the Gen-3 is just the first—and the smallest—of several video generation models to come in a family of next-generation models trained on an upgraded infrastructure.
“The model can struggle with complex interactions between characters and objects, and generations don’t always follow the laws of physics exactly,” Germanidis told TechCrunch this morning in an interview. “This initial deployment will support high-resolution 5- and 10-second generations, with noticeably faster generation times than Gen-2. A 5-second clip takes 45 seconds to generate, and a 10-second clip takes 90 seconds to generate.”
Gen-3 Alpha, like all video generation models, was trained on a huge number of video examples – and images – so it could “learn” the models in those examples to generate new videos. Where does the training data come from? Track would not say. Few generative AI vendors volunteer such information these days, in part because they see training data as a competitive advantage and thus keep it and the information related to it close to their chest.
“We have an in-house research team that oversees all of our training, and we use curated in-house datasets to train our models,” Germanidis said. He left it at that.
The details of the training data are also a potential source of IP-related lawsuits if the vendor trained on public data, including copyrighted network data – and so another deterrent to disclosing much. Several cases making their way through the courts have rejected providers’ fair use training data protections, arguing that generative AI tools reproduce artists’ styles without the artists’ permission and allow users to generate new works resembling artists’ originals for which the artists they don’t get paid.
Runway addressed the copyright issue somewhat, saying it consulted with artists when developing the pattern. (Which artists? Not clear.) This echoes what Germanidis told me during a fireside at TechCrunch’s 2023 Disrupt conference:
“We’re working closely with artists to find out what the best approaches are to deal with this,” he said. “We’re exploring different data partnerships so we can grow further … and build the next generation of models.”
Runway also says it plans to launch Gen-3 with a new set of safeguards, including a moderation system to block attempts to generate videos from copyrighted images and content that doesn’t comply with the terms of service on Runway. Also in the works is a provenance system – compatible with the C2PA standard supported by Microsoft, Adobe, OpenAI and others – to identify that videos come from Gen-3.
“Our new and improved internal visual and text moderation system uses automatic oversight to filter out inappropriate or harmful content,” Germanidis said. “C2PA authentication verifies the origin and authenticity of media created with all Gen-3 models. As the model’s capabilities and the ability to generate high-quality content increase, we will continue to invest significantly in our alignment and safety efforts.”
Runway also revealed that it is partnering and collaborating with “leading entertainment and media organizations” to create custom versions of Gen-3 that allow for more “stylistically controlled” and consistent characters that target “specific artistic and narrative requirements “. The company adds: “This means that generated characters, backgrounds and elements can maintain a consistent appearance and behavior across scenes.”
A major unsolved problem with video generation models is control—that is, getting a model to consistently generate video consistent with the creator’s artistic intent. As my colleague Devin Coldway recently wrote, simple questions in traditional filmmaking, such as choosing the color of a character’s clothing, require workarounds with generative models because each shot is created independently of the others. Sometimes even workarounds don’t work – leaving extensive manual work for editors.
Runway has raised over $236.5 million from investors including Google (with whom it has cloud computing credits) and Nvidia, as well as VCs such as Amplify Partners, Felicis and Coatue. The company has aligned itself closely with the creative industry as its investments in generative AI technologies grow. Runway operates Runway Studios, an entertainment division that serves as a production partner for corporate clientele and hosts the AI Film Festival, one of the first events dedicated to showcasing films produced entirely — or in part — by AI.
But the competition is getting fiercer.
Generative AI startup Luma last week announced Dream Machine, a video generator that went viral for its ability to animate memes. And just a few months ago, Adobe revealed that it was developing its own video generation model trained on content in the Adobe Stock media library.
Elsewhere, there are companies operating like OpenAI’s Sora, which remains strictly limited but which OpenAI is seeding with marketing agencies and indie and Hollywood filmmakers. (OpenAI CTO Mira Muratti attended the 2024 Cannes Film Festival.) This year’s Tribeca festival — which also has a partnership with Runway to curate films made using AI tools — featured short films produced with Sora by filmmakers who received early access.
Google has also put its image generation model, Veo, in the hands of select creators, including Donald Glover (aka Childish Gambino) and his creative agency Gilga, as it works to bring Veo to products like YouTube Shorts.
As much as the various collaborations shake out, one thing is clear: Generative AI video tools threaten to upend the film and television industry as we know it.
Director Tyler Perry recently said he halted a planned $800 million expansion of his production studio after seeing what Sora could do. Joe Russo, the director of Marvel films such as Avengers: Endgame, predicts that within a year AI will be able to create a full-fledged movie.
A 2024 study commissioned by the Animation Guild, a union representing Hollywood animators and cartoonists, found that 75 percent of film production companies that have adopted AI have cut, consolidated or eliminated jobs since incorporating the technology. The study also estimates that by 2026, more than 100,000 jobs in the US entertainment sector will be disrupted by generative AI.
Some serious labor protection measures will be needed to ensure that video generation tools do not follow in the footsteps of other generative AI technologies and lead to a sharp decline in the demand for creative work.
