The Weird World of AI Voice Replicas

It’s a good bet that the generative-AI era will be stranger than anyone expects. In a new feature for The Atlantic, my colleague Charlie Warzel profiles ElevenLabs, an AI company that specializes in replicating voices.

“It’s easy, when you play around with the ElevenLabs software, to envision a world in which you can listen to all the text on the internet in voices as rich as those in any audiobook,” Charlie writes. “But it’s just as easy to imagine the potential carnage: scammers targeting parents by using their children’s voice to ask for money, a nefarious October surprise from a dirty political trickster. I tested the tool to see how convincingly it could replicate my voice saying outrageous things. Soon, I had high-quality audio of my voice clone urging people not to vote, blaming ‘the globalists’ for COVID, and confessing to all kinds of journalistic malpractice. It was enough to make me check with my bank to make sure any potential voice-authentication features were disabled.”

You may have already encountered ElevenLabs’ technology without realizing it. The Atlantic and The Washington Post use the software to produce audio versions of some stories. Nike cloned the NBA star Luka Dončić’s voice with the software for a recent marketing campaign. New York City Mayor Eric Adams’s office used it to mimic the politician’s voice for multilingual robocalls. And a gun-control nonprofit used it to re-create the voices of children killed in the Parkland school shooting.

The company has established safeguards in an attempt to head off nefarious usage, but it’s reasonable to expect surprises. As Charlie writes, “There are simply too many motivated people constantly searching for ways to use these tools in strange, unexpected, even dangerous ways.”

— Damon Beres, senior editor

ElevenLabs Is Building an Army of Voice Clones

By Charlie Warzel

Read the full article.

What to Read Next

P.S.

Apple received a ton of blowback this week for a new iPad commercial that, to many viewers, seemed like an uncomfortable reminder of generative AI’s threats to human creativity: It depicts a variety of artistic tools getting crushed in a hydraulic press, leaving a new tablet behind. “Apple wants to show you that the bulk of human ingenuity and history can be compressed into an iPad, and thereby wants you to believe that the device is a desirable entry point to both the consumption of culture and the creation of it … But good Lord, Apple, read the room,” I wrote with Charlie this week. (The company apologized yesterday for the ad.)

— Damon