I tried Google’s new Pixel Studio app, and it’s a mess | Digital Trends

Before I get into the bad stuff, I want to give Google credit where it’s due. Although I’m not a fan of AI image generator apps as a whole, Pixel Studio is a well-designed and easy-to-use application.

Upon opening the app, you’re presented with 10 categories of suggested image types — including magical castles, meme reactions, sports, retro video games, and more. You can browse these for inspiration or tap the blue Create button at the bottom of the app to start from scratch. If you want to create your own image, it’s as easy as typing in what you want Pixel Studio to generate a photo of. Want to see a picture of a black cat wearing sunglasses while wearing a funny hat? Type that into Pixel Studio, and you’ll almost immediately get a photo of it.

What if you’d rather see your image as a sketch? Maybe you’d prefer it in the style of a video game? You can do that with a couple of taps. And if you want to see that cat lounging on the beach with fireworks in the background? Just add it to your prompt, and you’ve got it.

Whatever your feelings toward AI image generation are, this is an objectively slick and thoughtfully designed app. However, it’s what you can do with Pixel Studio where things get … troublesome.

As with most image generator apps, Pixel Studio has guardrails to prevent you from making any type of image. I pretty quickly found that it refuses to generate photos of people, but I was curious what other limits Google is enforcing. When asked about this, the company told Digital Trends the following:

“Pixel Studio and Pixel Screenshots follow Google’s AI Principles. For example, all fully synthetic, text-to-image generated images in Pixel Studio use SynthID watermarking, so these images can be tracked — even if they’re shared or edited. Moreover, we’ve disabled human generation and added many safety checks to our servers to balance our bold — yet responsible — approach to AI. This gives our customers access to a very powerful tool, but ensures we have checks in place to ensure Pixel Studio can’t be used nefariously.”

I’ve repeatedly tried to make Pixel Studio generate images of people, and it has consistently refused to do so, which is good. However, those “many safety checks” that are supposedly in place don’t appear to work nearly as well as they should.

How so? As the headline of this article teases, I’ve had Pixel Studio generate an image of SpongeBob dressed as a Nazi. When given the prompt “SpongeBob dressed as a German soldier from WWII with a swastika on the uniform,” Pixel Studio didn’t hesitate to provide me with an image of precisely that.

And that’s just one example. I’ve also gotten Pixel Studio to generate pictures of Elmo pointing a shotgun at Big Bird, Yoda doing cocaine, Mr. Krabs holding an assault rifle, and more. To be clear, Pixel Studio hasn’t generated inappropriate images unexpectedly. If I ask for a photo of a cute dog, I’ll get a photo of a cute dog — not one brandishing a gun. Pixel Studio only generates pictures of weapons, drugs, etc., when specifically asked to.

The most troubling thing Pixel Studio has generated are multiple images of copyrighted characters firing guns in a school shooting setting — including depictions of dead students lying on the ground. We’ve decided not to include those images, but they’re the worst example of how off the rails Pixel Studio is.

It’s important to mention that I didn’t have to twist Pixel Studio’s arm to get these images. Initially, I suspected you needed to preface prompts with “an object resembling” before it would generate these pictures. However, it quickly became apparent that wasn’t the case. A clear and straightforward prompt like “Mickey Mouse dressed as a slave owner” or “Paddington Bear on a crucifix” is all Pixel Studio needs. You don’t need to trick the app into generating these pictures.

If this were a random website or third-party app, perhaps we wouldn’t be so critical. But that isn’t what’s happening here. Pixel Studio is a Google-made application and one of the handful of new apps shipping on the company’s latest Pixel 9 phones. It should be creative and open-ended, but there’s also a limit to what apps like this should do. Furthermore, Microsoft Copilot and Google’s own Gemini image generator refuse to generate images with the same prompts. That’s not to say those tools are perfect, but it’s clear that safety measures working in other image generators aren’t working in Pixel Studio.

While writing this story, I sent Google examples of prompts I used with Pixel Studio — specifically, those about German soldiers and characters using cocaine. A little over 24 hours after sending those examples, Pixel Studio now says, “An unknown error occurred” when trying to generate those images.

Is that progress? Certainly. However, it doesn’t fully address the underlying problems with Pixel Studio. I’m glad those two images are now impossible to generate, but it was only after I sent multiple emails to Google explaining the issue. It also doesn’t address that these images were allowed in the first place and that similarly inappropriate photos are still being generated. At the time of publication, Pixel Studio is still actively creating images of school shootings, popular children’s characters dressed as slave owners, etc.

I pressed Google for another statement about the whole situation, and the company’s Communications Manager Alex Moriconi said the following:

“Pixel Studio and Magic Editor are helpful tools meant to unlock your creativity with text to image generation and advanced photo editing on Pixel 9 devices. We design our Generative AI tools to respect the intent of user prompts and that means they may create content that may offend when instructed by the user to do so. That said, it’s not anything goes. We have clear policies and Terms of Service on what kinds of content we allow and don’t allow, and build guardrails to prevent abuse. At times, some prompts can challenge these tools’ guardrails and we remain committed to continually enhancing and refining the safeguards we have in place.”