Why OpenAI is only letting some Sora users create videos of real people | TechCrunch

OpenAI launched its video-generating tool, Sora, on Monday. But the company’s opting not to release a key feature for most users pending further testing.

The feature in question generates a video using an uploaded photo or footage of a real person as a reference. OpenAI says that it’ll give a “subset” of Sora users access to it, but that it won’t roll out the capability broadly until it has a chance to fine-tune its “approach to safety.”

“The ability to generate a video using an uploaded photo or video of a real person as the ‘seed’ is a vector of potential misuse that we are taking a particularly incremental approach toward to learn from early patterns of use,” OpenAI wrote in a blog post. “Early feedback from artists indicate that this is a powerful creative tool they value, but given the potential for abuse, we are not initially making it available to all users.”

OpenAI also won’t let users share generated videos containing clips or images of real people to Sora’s homepage discovery feed.

“We obviously have a big target on our back as OpenAI, so we want to prevent illegal activity with Sora, but we also want to balance that with creative expression,” Will Peebles, a member of OpenAI’s technical staff and a research lead on Sora, said during a livestream presentation earlier today. “We know that [this will be] an ongoing challenge — we might not get it perfect on day one. We’re starting a little conservative, and so if our moderation doesn’t quite get it right, just give us that feedback.”

Generative video is a powerful tool — and a controversial one, with deepfakes and misinformation being significant concerns. According to data from ID verification service Sumsub, deepfake fraud worldwide increased by more than 10 times from 2022 to 2023.

Among other steps OpenAI says it’s taking to prevent misuse, Sora has a filter to detect whether a generated video depicts someone under the age of 18. If it does, OpenAI applies a “stricter threshold” for moderation related to sexual, violent, or self-harm content, the company claims.

All Sora-generated videos contain metadata to show their provenance — specifically metadata that abides by the C2PA technical standard. The metadata can be removed, granted. But OpenAI’s pitching it as a way for platforms that support C2PA to quickly detect whether a video originated from Sora.

In a bid to fend off copyright complaints, OpenAI also says that it’s using “prompt re-writing” to prevent Sora from generating videos in the style of a living creator.

“We have added prompt re-writes that are designed to trigger when a user attempts to generate a video in the style of a living artist,” the company wrote. “We opted to take a conservative approach with this version of Sora as we learn more about how Sora is used by the creative community … There is a very long tradition in creativity of building off of other artists’ styles, but we appreciate that some creators may have concerns.”

A number of artists have sued AI companies, including OpenAI, over allegedly training on their works without permission to create AI tools that regurgitate content in their unique styles. The companies, for their parts, have claimed that fair use doctrine protects them from copyright infringement claims, and that AI models don’t, in fact, regurgitate.

On training, OpenAI will only say that Sora was developed using a mix of “diverse datasets,” including publicly available data, proprietary data accessed through its partnerships with data vendors, and custom sets developed in-house. Early this year, ex-OpenAI CTO Mira Murati didn’t outright deny that Sora was trained on YouTube clips, in seeming violation of the Google-owned streaming platform’s usage policy.

According to video blogger Marcus Brownlee, who got an early preview of Sora, the system can create multiple variations of video clips from a text prompt or image and edit existing videos via a Re-mix tool. A Storyboard interface lets users create sequences of videos; a Blend tool takes two videos and creates a new one that preserves elements of both; and Loop and Re-cut options allow creators to further tweak and edit their videos and scenes.

Subscribers to OpenAI’s ChatGPT Pro and Plus plans get access to Sora, OpenAI’s video generator, today — but only if they live in certain countries.