Google’s AI Mode search can now answer questions about images

Google started cramming AI features into search in 2024, but last month marked an escalation. With the release of AI Mode, Google previewed a future in which searching the web does not return a list of 10 blue links. Google says it's getting positive feedback on AI Mode from users, so it's forging ahead by adding multimodal functionality to its robotic results. AI Mode relies on a custom version of the Gemini large language model (LLM) to produce results. Google confirms that this model now supports multimodal input, which means you can now show images to AI Mode when conducting a search. As this change rolls out, the search bar in AI Mode will gain a new button that lets you snap a photo or upload an image. The updated Gemini model can interpret the content of images, but it gets a little help from Google Lens. Google notes that Lens can identify specific objects in the images you upload, passing that context along so AI Mode can make multiple sub-queries, known as a "fan-out technique." Google illustrates how this could work in the example below. The user shows AI Mode a few books, asking questions about similar titles. Lens identifies each individual title, allowing AI Mode to incorporate the specifics of the books into its response. This is key to the model's ability to suggest similar books and make suggestions based on the user's follow-up question.

Google sees AI Mode as a key way to maintain its role as the primary Internet directory. As the company has explained in the past, many people are using traditional search to find answers to specific questions. For those people, AI Mode can be a quicker and more effective way to find what they need. Google now says that its early telemetry from AI Mode shows that people are putting about twice as much text in their searches compared to traditional web search. Google frames this as a good thing, but that could also indicate that users feel they need to provide more context to the robot. It's likely you still haven't seen AI Mode, even though it has been available for weeks. Google launched this feature exclusively for Google One AI Premium subscribers, and it needed to be enabled in Google Labs. Now, Google says it's expanding AI Mode to "millions more Labs users in the US" who aren't paying for AI features. While you must still opt-in, the day might be coming when AI Mode is an option for all searches. And perhaps sometime after that, it'll be the default way Google wants you to search the web.