Google's Game-Changer: Equipping Robots with a 'New Brain'

Google unleashes a killer move: the Gemini Robotics 1.5 series model takes another solid step toward general embodied AI.

On September 25 local time, DeepMind, Google's core AI research division, released the Gemini Robotics 1.5 series of models designed for robots. The series includes two models:

• Gemini Robotics 1.5: A VLA (Vision-Language-Action) model that converts visual information and instructions into motion commands for robots to execute tasks. Unlike traditional VLA models that simply follow commands, it first thinks and demonstrates the process before executing actions, thereby evaluating and completing complex tasks. It also possesses strong cross-morphology learning capabilities, requiring no adjustments based on different robot morphologies.
• Gemini Robotics-ER 1.5: A VLM (Vision-Language Model) capable of reasoning about the physical world, calling digital tools, and creating detailed multi-step plans to complete tasks. This model has achieved state-of-the-art performance in both academic and internal benchmarks.

Both models are built upon the core Gemini series models. While they have distinct positioning, their combined collaborative operation unlocks an AI agent experience for robots when executing specific tasks.

In practical applications, Gemini Robotics-ER 1.5 serves as the robot's 'strategic commander.' It interacts with humans via natural language, autonomously assesses task progress and success rates, and can call external tools like Google Search to obtain outside information. It formulates detailed multi-step plans for complex tasks and provides natural language instructions for each step.

Subsequently, Gemini Robotics 1.5 utilizes its pre-action thinking capability to generate internal reasoning processes in natural language while executing tasks, better handling semantically complex tasks. It can also break down long tasks into short steps, improving execution success rates and environmental adaptability.

In the demonstration video, Aloha and Apollo robots equipped with the Gemini Robotics 1.5 series models performed several specific tasks.

For the task assigned to Aloha—sorting garbage—it first used the Gemini Robotics-ER 1.5 model to call a search tool to obtain San Francisco's garbage classification rules. Through collaborative processing by the model, it placed wet waste such as fruit peels into the corresponding green bins, put cola cans into blue bins representing recyclables, and disposed of general waste in black bins, successfully completing the garbage sorting task.

In the luggage packing task performed by Apollo, not only did it complete packing for a trip to London, but it also independently queried the local weather in London (without any human instruction). Upon learning that it would rain in London, Apollo packed an umbrella into the luggage.

The integration of the Gemini Robotics 1.5 series models makes robots smarter when executing specific tasks, enabling targeted handling based on actual situations. Especially the ability to think before executing a task makes the execution of specific tasks more reliable.

Compared to traditional VLA models, the Gemini Robotics 1.5 model possesses deep understanding capabilities for tasks rather than simply converting commands directly into actions. It even demonstrated other abilities beyond the task itself, such as preparing an umbrella for the user in the second task mentioned above.

In the subsequent third task, Apollo sorted clothes: white clothes were to be placed in white storage boxes. When Apollo picked up the clothes to put them into the box, staff members swapped the target box with another box. During this process, the model helped Apollo react promptly, ultimately completing the task smoothly.

In multi-level thinking processes, the model can break down longer tasks into simple segments that robots can successfully execute, and it can also help the model generalize to solve new tasks, making it more stable in response to environmental changes.

Gemini Robotics 1.5 also boasts excellent cross-morphology learning capabilities.

During the implementation of Embodied AI models, because different robot bodies have different morphologies and sizes, as well as varying sensing capabilities and degrees of freedom, it is very cumbersome to transplant actions learned on one body to another.

However, the emergence of Gemini Robotics 1.5 effectively addresses this difficulty without requiring additional specialized adjustments for the model based on different bodies.

For example, when asked to perform tasks trained only on the Aloha robot, the previously unexposed Apollo robot can also complete them well through skill transfer learning.

This cross-morphology learning capability is a crucial factor for the true application of Embodied AI in the future. It alleviates the efficiency dilemma of "one robot, one training" in traditional Embodied AI and is a key element for Embodied AI to truly integrate into the physical world.

For the release of this model, DeepMind stated that it will make Gemini Robotics-ER 1.5 available to developers via the Gemini API in Google AI Studio, while Gemini Robotics 1.5 is currently only available to select partners.