Welcome to the new surreal: how AI-generated video is changing film

Fast and cheap

Artists are often the first to experiment with new technologies. But the immediate future of generative video is shaped by the advertising industry. Trail made The Frost to explore how generative AI could be incorporated into its products. The company creates video creation tools for businesses looking for a quick and affordable way to make commercials. Waymark is one of several startups, along with companies like Softcube and Vedia AI, offering tailored video ads to customers with just a few clicks.

Waymark’s current technology, which launched earlier this year, brings together several AI techniques, including large speech models, image recognition and speech synthesis, to generate a video ad on the fly. Waymark also drew on its large dataset of non-AI-generated commercials created for previous clients. « We have hundreds of thousands of videos, » says CEO Alex Persky-Stern. « We mined the best of those and trained him in what a good video should look like. »

To use Waymark’s tool, which it offers as part of a tiered subscription service starting at $25 a month, users provide the web address or social media accounts for their activity, and it shuts down and collects all the text and images it can find. It then uses that data to generate a commercial, using OpenAI’s GPT-3 to write a script that is read aloud by a synthesized voice over selected images highlighting the activity. A one-minute commercial can be generated in seconds. Users can change the result if they want, by changing the script, changing the images, choosing a different voice and so on. Waymark says more than 100,000 people have used its tool so far.

The problem is, not every company has a website or images to draw from, Parker says. « An accountant or a therapist might not have any assets, » she says.

Waymark’s next idea is to use generative AI to create images and videos for businesses that either don’t have any yet or don’t want to use the ones they do have. “This is the drive to create The Frost”, Parker says. « Create a world, an atmosphere. »

The Frost it has an atmosphere, for sure. But it’s also extravagant. « It’s still not a perfect medium by any means, » says Rubin. “It was a little difficult to get certain things from DALL-E, like the emotional responses in the faces. But other times, he has delighted us. We were like, ‘Oh my God, this is magic happening before our eyes.’”

This erratic process will improve as technology improves. DALL-E 2, which Waymark manufactured The Frost, was released just a year ago. Video generation tools that generate short clips have only been around for a few months.

The most revolutionary aspect of the technology is the ability to generate new shots whenever you want, says Rubin: « With 15 minutes of trial and error, you get that shot you wanted that fits perfectly into a sequence. » He recalls editing the film together and needing special shots, like a close-up of a boot on a mountainside. With DALL-E, he could just call it back. « It’s amazing, » he says. « That’s when it started to be a really eye-opening experience as a director. »