中文

Google DeepMind Unveils Genie 4, the First World Model in the Series to Generate Native Audio

By Jiang Si Qi

GenieSimulationGenerationRoboticsIntroduction

Google DeepMind has released Genie 4, a new real-time interactive world model that natively generates audio, prompting questions about how close such sounding, simulated environments are to being used for robot training.

In the early hours of October 28, Beijing time, Google DeepMind released the next-generation world model Genie 4 on its official blog. Google published a synchronized blog post detailing the release, and Google CEO Sundar Pichai announced it on X. How close are we to using real-time, interactive, and audio-equipped simulated worlds for robot training?

This image is a related illustration for Google's release of Genie 4, featuring a transparent sphere containing a hummingbird and orange flowers, set against a blurred natural background with rich, soft colors. Used as the header image for the Genie 4 official blog, it visually presents the Genie series' ability to generate simulated worlds with sound, showcasing a vivid natural simulation and echoing Genie 4's core feature of building real-time interactive simulated worlds with audio.

Genie 4 is currently available as a research preview to trusted testers. According to the Google blog, Genie 4 will soon be accessible to Google AI Ultra subscribers within Project Genie, alongside early partners of the Gemini API and Google AI Studio; users can join the waitlist through Project Genie. The DeepMind blog post is authored by Raia Hadsell, Diego Rivas, and Hang Qi.

The underlying capabilities of Genie 4 are powered by the Gemini series of models. According to Google, Gemini Omni ensures the simulated scenes closely mirror the real world in both appearance and behavior, while other Gemini models provide the complex reasoning needed to maintain simulation coherence and continuity. Users first define an initial event, and Genie 4 simulates the subsequent sequence, including the world's response, the accompanying audio, and how the characters in the scene move, react, and speak.

Regarding interaction, users control movement and camera angles using a standard keyboard and can enter text commands at any time to alter the scene—for example, by changing the weather, panning the camera, or introducing new characters or objects. According to Google, Gemini contextualizes the user's commands based on the surrounding environment. Between inputs, Gemini continuously monitors and guides the scene, ensuring the world evolves in a coherent direction.

Audio generation is a newly introduced capability in Genie 4. According to Google, it is the first real-time world model in the Genie series to natively generate audio. Each generated world features ambient sounds tailored to the environment, such as wind rustling through trees or waves crashing against the shore, and can also produce music. Genie 4 also incorporates dynamic, real-time dialogue, allowing characters in the scene to converse and react based on user input or in-world events. Google specified in the blog post that all audio in the demonstration video was generated directly by Genie 4. Meanwhile, TechCrunch reported that Runway’s second-generation world model, GWM Worlds 2, released a few months prior, is similarly capable of responding to user input in real time and generating audio.

The realism of physics and character animation is another major focus of this release. According to Google, Genie 4 leverages an understanding of the physical world to simulate real-world physical phenomena such as aerodynamics and collisions. Google notes that the most significant improvement lies in human simulation: the generated characters move naturally, react to their environment, and can be manipulated in real time. Official demonstrations showcase scenarios such as a watermelon being smashed and maple seeds spiraling down from a tree onto the grass.

The image shows Google Genie 4 simulating a marble bust smashing into a watermelon. A green watermelon sits on a stone pillar in a courtyard surrounded by greenery. Below the image is a command input box displaying 'Drop a heavy white marble classical bust on top of the watermelon!' with a '→' button. The bottom left features directional keys 'W A S D', and the bottom right has arrow keys. This image relates to Genie 4's simulation of multi-step complex events, intuitively demonstrating its audio generation capabilities.

Figure | Following a user command, Genie 4 simulates a marble bust smashing into a watermelon (Source: Google DeepMind)

The integration of Gemini and Genie 4 is used to simulate complex, multi-step events. According to Google, Gemini 3.8 Flash acts as a "guiding agent in the loop": while Genie 4 generates the world in real time, Gemini continuously monitors the visual output, breaks the objective down into a sequence of steps, and evaluates the newly generated frames to track progress and verify the completion of each step.

When the environment changes or a user intervenes, Gemini adjusts its plan on the fly. According to Google, this equips the simulated world with reliable causal relationships, procedural behaviors, and multi-step reasoning capabilities, allowing researchers to test how AI systems respond to various environments before deploying them in the real world. One of the official demonstrations features a Universal Robots UR5 robotic arm holding a pipette and dispensing liquid into a lab plate.

This image presents a robotic arm operating in a laboratory setting. The main subject is a Universal Robots UR5 robotic arm holding a pipette above a transparent experimental plate, corresponding to Google Genie 4's official demonstration scenario. This demonstration is an instance of Genie 4 generating a world in real time, utilizing Gemini 3.8 Flash as a guiding agent in the loop to complete the simulation of the robotic arm dispensing liquid into the experimental plate.

Image | Genie 4 simulating a UR5 robotic arm using a pipette to dispense liquid into a lab plate (Source: Google DeepMind)

In robotics, Google is combining research on Gemini 4 Argon, Genie 4, and Gemini Robotics. According to Google, in one experiment, Argon breaks down a user-assigned task into multiple steps, feeding them to Genie 4 as text descriptions during the simulation rendering process. The simulation can then be adjusted in real time as the task progresses.

In the demonstration video, the two models work together to simulate Apptronik's Apollo humanoid robot playing Chinese checkers. Google states that this provides a pathway for simulating a vast number of scenarios within an imagined world, which the official model page refers to as an "infinite playground" for robotic simulation.

The image shows the reasoning steps of Gemini 4 Argon and a scene of Genie 4 simulating a robot playing checkers. On the left are the reasoning steps of Gemini 4 Argon, showing Robot A's metallic silver mechanical hand operating at the sixth step, with its right index finger pushing a white chess piece. On the right is the scene simulated by Genie 4, where the robot's mechanical hand operates on the chessboard with a human assisting nearby. This image closely relates to the context, intuitively presenting the process of Genie 4 executing steps in simulated rendering after Gemini 4 Argon formulates a multi-step plan.

Image | Left: reasoning steps broken down by Gemini 4 Argon; Right: Genie 4 simulating the Apollo humanoid robot playing Chinese checkers (Source: Google DeepMind)

The robotic examples for Genie 4 are text-driven. According to the Google DeepMind model page, Gemini 4 Argon first formulates a multi-step plan for a task, then feeds the steps to Genie 4 as text descriptions during the simulation rendering process. In contrast, another class of world models in robotics uses actions as input conditions.

According to 1X, its world model is trained on thousands of hours of video and action data collected by the EVE humanoid robot in homes and offices. Starting from the same set of initial frames, the model can generate different outcomes based on varying action commands, which are used to evaluate robot policies.

According to Meta, V-JEPA 2 first undergoes action-free pre-training on more than 1 million hours of video and 1 million images, followed by action-conditioned training on 62 hours of robot data. The prediction outputs are generated in the representation space, which can be used for zero-shot robot planning.

According to NVIDIA, Cosmos 3 can generate future frames from text, image, video, ambient audio, and action inputs, serving as a foundation model for world-action models in robot policy learning. On Google's currently published page, Genie 4 accepts keyboard inputs and text instructions.

Google stated in a blog post that understanding and simulating the world is a crucial step toward building AGI. Genie 4 can generate safe, infinitely variable simulated environments to train robots and agents, as well as to create new interactive experiences. According to Google, beyond robotics research, enterprise partners across multiple industries are evaluating Genie 4 to rapidly generate scenes from scratch. Trusted testers also span the creative, journalism, education, gaming, and engineering fields. One film and television team is using it for pre-visualization, while a photojournalist is experimenting with turning a historical photograph into an interactive simulated environment.

Google also outlined the current limitations of Genie 4. According to the company, Genie 4 maintains environmental coherence over short periods, but visual details and object permanence may drift as simulation time increases, complex instructions are modified repeatedly, or the viewpoint moves through the scene. Complex physical interactions, real-time audio accuracy, clear text rendering, spatial accuracy, and long-term consistency remain active areas of research. Google's official model page states that the model supports "a few minutes" of continuous interaction and does not support extended sessions, without specifying an exact duration.

According to reports from The Verge and TechCrunch, Genie 4 can maintain scene coherence for up to approximately two minutes, twice the duration of Genie 3. All video and audio generated by Genie 4 are embedded with Google's SynthID digital watermarks.