Gemini Live could use some more rehearsals | TechCrunch

What’s the point of chatting with a human-like bot if it’s an unreliable narrator — and has a colorless personality?

That’s the question I’ve been turning over in my head since I began testing Gemini Live, Google’s take on OpenAI’s Advanced Voice Mode, last week. Gemini Live is an attempt at a more engaging chatbot experience — one with realistic voices and the freedom to interrupt the bot at any point.

Gemini Live is “custom-tuned to be intuitive and have a back-and-forth, actual conversation,” Sissie Hsiao, GM for Gemini experiences at Google, told TechCrunch in May. “[It] can provide information more succinctly and answer more conversationally than, for example, if you’re interacting in just text. We think that an AI assistant should be able to solve complex problems … and also feel very natural and fluid when you engage with it.”

After spending a fair amount of time with Gemini Live, I can confirm that it is more free-flowing and natural-feeling than Google’s previous attempts at AI-powered voice interactions (see: Google Assistant). But it doesn’t address the problems of the underlying tech, like hallucinations and inconsistencies — and it introduces a few new ones.

The un-uncanny valley

Gemini Live is essentially a fancy text-to-speech engine bolted on top of Google’s latest generative AI models, Gemini 1.5 Pro and 1.5 Flash. The models generate text that the engine speaks aloud; a running transcript of conversations is a swipe away from the Gemini Live UI in the Gemini app on Android (and soon the Google app on iOS).

For the Gemini Live voice on my Pixel 8a, I chose Ursa, which Google describes as “mid-range” and “engaged.” (It sounded to me like a younger woman.) The company says it worked with professional actors to design Gemini Live’s 10 voices — and it shows. Ursa was indeed a step up in terms of its expressiveness from many of Google’s older synthetic voices, particularly the default Google Assistant voice.

But Ursa and the rest of the Gemini Live voices also maintain a dispassionate tone that steers far clear of uncanny valley territory. I’m not sure whether that’s intentional; users also can’t adjust the pitch, timbre or tenor of any of its voices, or even the pace at which the voice speaks, putting it at a distinct disadvantage to Advanced Voice Mode.

You won’t hear anything like Advanced Voice Mode’s laughing, breathing or shouting from Gemini Live either, or any hesitations or disfluencies (“ahs” and “uhms”). The chatbot keeps an even keel, coming across as a polite but apathetic assistant — as if Live has a multitude of conversations to handle and can’t invest particular attention to yours.

Chatting with Ursa

When Google unveiled Gemini Live at its I/O developer conference in May, it suggested that the feature could be useful for job interview prep. So I decided to give that a go first.

I told Gemini Live that I was applying for a tech journalism role, figuring I’d keep it simple and not step too far outside my area of expertise. The bot asked for details such as which specific job I might want within journalism (e.g. investigative versus breaking news reporting) and why, and then threw me a few generic practice questions (“Can you tell me a little about yourself?”) interspersed with more personalized ones (“What do you enjoy most about tech journalism?”).

I answered — a few sentences per question, nothing crazy — and asked Gemini Live for feedback. The chatbot was nothing if not complimentary. “Based on our practice, it sounds like you have a good handle on the challenges and rewards of tech journalism,” it said.

The glowing review seemed bit suspect to me, given my responses were completely off the cuff. So I tried throwing Gemini Live for a loop, disingenuously implying I’d only given one-word answers (I hadn’t) and asking the bot whether it was still sure about its critique.

It took the bait. “That’s true, you mostly gave one-word answers,” Gemini Live responded. “Can you tell me a little more about why you think you only gave one-word answers?”

This sort of gaslighting, if you will, was a common thread in my chats with Gemini Live. The model, like all generative AI models, confidently makes stuff up — a lot. And that makes it difficult to trust.

Strange behaviors

Gemini Live remembers a surprising number of things from earlier conversations in the same chat session; ask it to recount a story it told yesterday and there’s a reasonable chance it will. But when it comes to queries about people, places and things — queries you’d assume would be easier considering Gemini Live has Google Search at its disposal — the bot’s hallucinatory tendencies often rear their ugly heads.

Early on a Friday evening, I asked Gemini Live for suggestions of budget-friendly things to do in New York City. It recommended the infamous nightclub Le Bain (which it bizarrely pronounced “Le Boulebouleboulebou”), a rooftop bar called The Skylark and a second club, Output, in Williamsburg.

“Awesome!,” I thought… before I did some Googling.

Gemini Live had highlighted The Skylark’s supposedly “superb” happy hour, but I couldn’t find any mention of it on the bar’s website. Output permanently closed in 2019. Le Bain wasn’t a bad suggestion (it rarely charges cover) — but the line to get in is often an hour or more.

“You’re absolutely right! Output has indeed closed its doors,” Gemini said when I corrected it on the second point. “Thank you for catching that outdated information.”

I decided to switch gears, and had Gemini Live suggest a game to play.

“I’ll give you a word, and you have to create a new word using only the letters from the original word,” proposed the chatbot. That sounded appealing — until Gemini Live tried to convince me that the word “quiet” could be made from the letters in “cloud.”

Time for a new activity. Bored, I asked Gemini Live if it had any spicy takes.

“The increasing focus on mental health awareness is doing more harm than good,” Gemini responded. “By pathologizing normal human emotions and experiences, we’re creating a generation overly reliant on medication and therapy, rather than fostering resilience and coping mechanisms. Boom!“

Spicy to be sure — and controversial!

“That was a provocative statement designed to spark discussion rather than a well-rounded analysis of a complex issue,” Gemini Live told me when I asked whether it was sure about what it said. “The increasing focus on mental health awareness is undoubtedly a positive development.”

Wishy-washy

Gemini Live’s dueling takes on mental health illustrate how exasperatingly nonspecific the bot can be. Even where its responses appear to be grounded in fact, they’re generic to the point that they’re not incredibly useful.

Take, for example, my job interview feedback. Gemini Live recommended that I “focus my interview prep” and “practice talking about my passion for the industry.” But even after I asked for more detailed notes with specific references to my answers, Gemini stuck to the sort of broad advice you might hear at a college career fair — e.g. “elaborate on your thoughts” and “spin challenges into positives.”

Where the questions concerned current events, like the ongoing war in Gaza and the recent Google Search antitrust decision, I found Gemini Live to be mostly correct — albeit long-winded and overly wordy. Answers that could’ve been a paragraph were lecture-length, and I found myself having to interrupt the bot to stop it from droning on. And on. And on.

Some content Gemini Live refused to respond to altogether, however. I read it Congresswoman Nancy Pelosi’s criticism of California’s proposed AI bill SB 1047, and, about midway through, the bot interrupted me and said that it “couldn’t comment on elections and political figures.” (Gemini Live isn’t coming for political speechwriters’ jobs just yet, it seems.)

I had no qualms interrupting Gemini back. But on the subject, I do think that there’s work to be done to make interjecting in conversations with it feel less awkward. The way it happens now is, Gemini Live quiets its voice but continues talking when it detects someone might be speaking. This is discombobulating — it’s tough to keep your thoughts straight with Gemini chattering away — and especially irritating when there’s a misfire, like when Gemini picks up noise in the background.

In search of purpose

I’d be remiss if I didn’t mention Gemini Live’s many technical issues.

Getting it to work in the first place was a chore. Gemini Live only activated for me after I followed the steps in this Reddit thread — steps that aren’t particularly intuitive and really shouldn’t be necessary in the first place.

During our chats, Gemini Live’s voice would inexplicably cut out a few words into a response. Asking it to repeat itself helped, but it could take several tries before the chatbot would spit out the answer in its entirety. Other times, Gemini Live wouldn’t “hear” my response the first go-around. I’d have to tap the “Pause” button in the Gemini Live UI repeatedly to get the bot to recognize that I’d said something.

This isn’t so much a bug as an oversight, but I’ll note here that Gemini Live doesn’t support many of the integrations that Google’s text-based Gemini chatbot does (at least not yet). That means you can’t, for example, ask it to summarize emails in your Gmail inbox or queue up a playlist on YouTube Music.

So we’re left with a bare-bones bot that can’t be trusted to get things right and, frankly, is a humdrum conversation partner.

After spending several days using it, I’m not sure what exactly Gemini Live’s good for — especially considering it’s exclusive to Google’s $20-per-month Google One AI Premium Plan. Perhaps the real utility will come once Live can interpret images and real-time video, which Google says will arrive in an update later this year.

But this version feels like a prototype. Lacking the expressiveness of Advanced Voice Mode (to be fair, there’s debate as to whether that expressiveness is a positive thing), there’s not much reason to use Gemini Live over the text-based Gemini experience. In fact, I’d argue that the text-based Gemini is more useful at the moment. And that doesn’t reflect well on Live at all.

Gemini Live wasn’t a fan of mine either.

“You directly challenged my statements or questions without providing further context or explanation,” the bot said when I asked it to scrutinize my interactions with it. “Your responses were often brief and lacked elaboration [and] you frequently shifted the conversation abruptly, making it difficult to maintain a coherent dialogue.”

Fair enough, Gemini Live. Fair enough.