ACE Robotics and Nanyang Technological University Release Mundo, an Interactive World Model

On October 30, ACE Robotics and the S-Lab at Nanyang Technological University released Mundo, an environment-specific world model that converts a small number of images or videos into explorable, interactive 3D scenes for robot navigation and manipulation training. The system integrates 3D scene representation, physical simulation, and video generation with Astra to enable spatial understanding.

On October 30, ACE Robotics and the S-Lab at Nanyang Technological University unveiled Mundo. Designed for specific environments, this world model transforms a small set of images or videos into explorable, interactive 3D scenes, supporting robot navigation and manipulation training.

Mundo integrates 3D scene representation, physics simulation, and video generation, leveraging Astra to build spatial understanding. The system identifies scenes and objects from input imagery to construct 3D proxies, allowing users to edit object positions, attributes, and actions via code. A physics engine computes object motion, contacts, and collisions, while a video model generates corresponding frames based on the input imagery, creating an end-to-end pipeline from scene reconstruction to interactive simulation.

Mundo can reconstruct an indoor space using just 1 to 12 images and generate novel views along specified camera paths. A single set of images can support multiple exploration trajectories, such as moving from a desk to a window or transitioning from a residential courtyard into the house. For regions not directly visible in the input imagery, the model infers geometry based on the existing spatial structure. Adding more reference views further grounds the scene in physical reality.

Figure | Office scene reconstruction and view generation

For larger environments, Mundo can utilize dozens of context images to continuously generate exploration videos ranging from tens of seconds to several minutes in length. The system generates video in segments along the same 3D proxy, selecting relevant reference images for each segment and blending overlapping frames to seamlessly connect adjacent clips, ensuring all segments share a consistent spatial foundation. Mundo also supports constructing explorable scenes from a single image or painting.

Figure | Residential scene exploration

Scenes constructed by Mundo are fully interactive. Users can define object properties such as name, dimensions, position, mass, and friction via code, then trigger actions or apply external forces. The physics engine computes object dynamics under gravity and contact constraints, outputting states like motion trajectories, contact points, and collision forces. The system then synthesizes the interaction results using the environmental imagery.

Figure | Scene code and physics simulation

Users can push monitors, drag trash bins, or open drawers within the scene, and adjust object materials and gravity conditions to observe the resulting changes. When spheres of varying masses are dropped from the same height, they exhibit different bouncing and collision outcomes, with their corresponding physical states rendered in real time.

Figure | Collision simulation of spheres with varying masses

Built on the same scene framework, Mundo also supports robot navigation and manipulation. During navigation tasks, a 3D agent provides information on object categories, positions, obstacles, and traversable areas. The path planner uses this data to generate routes, rendering either first-person or third-person views of the robot along its path.

Figure | Robot scene navigation

During manipulation tasks, the robot can attempt to place an apple into a bowl, set a can on a tray, or stack blocks within the same environment. The system simultaneously generates external, head-camera, and wrist-camera views, while recording task trajectories and physical states for researchers to analyze and compare strategies.

Figure | Simulation of a robot grasping an apple

Mundo also enables the observation of dynamic events from novel viewpoints. Researchers can pause the simulation at a specific moment and move the camera to inspect the exact same instant from different angles, examining motion details and occlusion relationships. In a hotel scenario, the team used Mundo to reconstruct floors and laundry areas, providing spatial and semantic information for navigation and simulating interactions such as opening a washing machine door.

Within ACE Robotics' Kairos-centric world model technology stack, Mundo adds capabilities for specific environment modeling, training data generation, and simulation-based training. By converting real-world footage into editable, interactive 3D environments, Mundo supports robots in scene adaptation, navigation, and manipulation training.