In brief: This week, Google unveiled plans for a way to search that combines images and text to give more context to search queries. The method can use a smartphone's camera in combination with AI, attempting to intuitively refine and expand search results.
At its Search On event this week, Google revealed details about how it plans to use a technology it calls Multitask Unified Model (MUM), which should intelligently figure out what a user is searching for based on images and text, as well as give users more ways to search for things.
While Google didn't give a specific date, its blog post stated the feature should roll out "in the coming months." Users will be able to point at something with a phone camera, tap an icon which Google calls Lens, and ask Google something related to what they're looking at. The blog post theorizes scenarios like taking a picture of a bicycle part you don't know the name of and asking Google how to fix it, or taking a picture of a pattern and trying to find socks with the same pattern.
Google initially introduced MUM back in May where it theorized more scenarios in which the AI might help expand and refine searches. If a user asks about climbing Mt. Fuji for instance, MUM might bring up results with information about the weather, what gear one might need, the mountain's height, and so-on.
A user should also be able to use MUM to take a picture of a piece of equipment or clothing and ask if it's suitable for climbing Mt. Fuji. MUM should additionally be able to deliver information it learns from sources in many different languages other than the one the user searched in.