Google infusing AI into Android with Gemini app and on-device Nano

Google wants to reimagine Android with AI in two distinct ways that both leverage Gemini. At I/O 2024, Google is splitting the OS news into two days, with AI up first and more on Android 15 coming tomorrow. 

Following the launch in February, the Gemini app on Android is “getting even better at understanding the context of what’s on your screen and what app you’re using.” Google says that context and integration makes Android the best place to use Gemini.

For starters, Gemini will soon exist as an overlay panel even when delivering results. Previously, anything after your initial command would open in a fullscreen UI. In addition to preserving context, it will allow you to drag-and-drop an image Gemini generated into a conversation. 

The other big integration is how activating Gemini for Android in YouTube will show an “Ask this video” button. Gemini can answer your questions about this video. It will work for billions of videos, with things like captions being used. Meanwhile, those subscribed to Gemini Advanced, with its long context window, will get an “Ask this PDF” button to do the same. This update is rolling out “over the next few months” to hundreds of millions of Android devices. 

Meanwhile, Google shared today that Circle to Search is now available on over 100 million devices with plans to double that by year’s end. Google is adding a homework helper to it powered by LearnLM, a “new family of models fine-tuned for learning.” Like Google Lens, you can circle a math or physics world problem to get step-by-step instructions. An upcoming update will let CTS solve symbolic formulas, diagrams, graphs, and other more complex problems. 

In the future, activating Gemini will show Dynamic Suggestions. This will use Gemini Nano to understand what’s on your screen. For example, if you activate Gemini in a conversation talking about pickleball, suggestions might include “Find pickleball clubs near me” and “Pickleball rules for beginners.”

Google introduced Gemini Nano late last year on the Pixel 8 Pro before expanding to the Galaxy S24. The next major update to the on-device foundation model is Gemini Nano with Multimodality, specifically “sights, sounds and spoken language.” This will launch on Pixel “later this year.”

Besides Gemini Dynamic Suggestions, Gemini Nano will be used by TalkBack to create rich descriptions for unlabeled images. No internet connection is required with this happening quickly on your device.  

This update will help fill in missing information — whether it’s more details about what’s in a photo that family or friends sent or the style and cut of clothes when shopping online.

Meanwhile, Android is going to use Gemini Nano to deliver “real-time alerts during a call if it detects conversation patterns commonly associated with scams.” Google will look for telltale signs like asking for personal information. This happens entirely on device and will be an opt-in feature. Google will share more details later this year.

For example, you would receive an alert if a “bank representative” asks you to urgently transfer funds, make a payment with a gift card or requests personal information like card PINs or passwords, which are uncommon bank requests.