Yunzhu Li: World Models Must Be Controllable to Train and Evaluate Robots | CoRL 2026
By Yu Yue
At the "Do Robots Need World Models?" workshop at CoRL 2026 on November 12, Columbia University's Yunzhu Li argued that world models must be highly controllable to serve as scalable data sources for robot training and policy evaluation.
On the morning of November 12, 2026, the Conference on Robot Learning (CoRL) hosted a half-day workshop titled "Do Robots Need World Models?" in the Brazos Room at the JW Marriott in Austin.
According to the workshop's official description, a world model refers to a class of predictive systems that enable robots to anticipate how their actions will alter the environment. The debate centers on whether this predictive structure should be explicitly constructed to facilitate planning, counterfactual reasoning, data efficiency, safety, and generalization, or whether it can be absorbed into an end-to-end policy that directly maps observations and goals to actions.
Yunzhu Li, an assistant professor in the Department of Computer Science at Columbia University and a researcher at World Labs, delivered the final invited talk of the day and moderated the subsequent debate. The third motion listed for the workshop was precisely that explicit world models are necessary for reproducible and scalable policy evaluation.
He framed the theme of his talk around building world models as scalable data sources for training and evaluating robotic systems.
Clarifying Which Type of Model
Li opened by reviewing the current state of the industry: massive amounts of capital and time are being invested in building general-purpose robots, yet many of these teams are doing almost nothing but large-scale imitation.
When discussing the relationship between world models and imitation policies, people refer to both as "models": some say dynamics models or world models, while others say policy models. He believes that when using the term "model," one must be highly precise and first clarify exactly which type is being referred to.
A world model or dynamics model performs forward prediction: it takes actions as input to predict the future. A policy is the inverse process: given the current state and a desired outcome, it determines the action. Both types of models have many different implementations; one is forward and the other is inverse, complementing each other.
He noted that while many in the industry are scaling up policy learning, they still rely primarily on imitation. Yet, looking back at the history of robotics, systems that actually function in the real world are almost always underpinned by a model—whether for airplanes, rockets, drones, or robots. They move and fly because they are built on good, precise models. Even legged robots can navigate their surroundings because they rely on a sufficiently good model.
This model does not need to be perfect: no one simulates snow, nor do they simulate bushes. With sufficient randomization and sim-to-real transfer, it is possible to obtain policies that run robustly in real-world environments. Therefore, Li Yunzhu’s research group has dedicated significant effort to figuring out how to build a sufficiently good world model for manipulation.
A spectrum from data to physics
Modeling for manipulation involves not just objects and not just robots, but also the interactions between robots and their environments. He acknowledges that this is difficult, as it requires dealing with a messy world, but he believes this should not be a reason to avoid trying.
The workflow followed by his research group involves bringing real-world environments into the digital world and then transferring them back to real-world scenarios. The digital world can be constructed using learning, using physics, or a combination of both.
He connects this to human intuition. Humans possess an intuitive model capable of predicting how the environment will change after an action is applied, such as how dough will deform. Although these models are never perfectly precise, they still yield useful and important signals that guide decision-making.
He then outlined a spectrum. At one end are models learned purely from data, citing highly impressive work such as the Genie series. He argued that for such models, even with additional data and compute, issues with overall consistency persist, particularly during complex and difficult interactions.
At the other end is the use of physics. NVIDIA Newton, MuJoCo, and Isaac are excellent examples of using physics to model environments, offering very high consistency. However, building a digital environment still requires complete state information about the environment, which is sometimes difficult to obtain and occasionally impossible to acquire.
For years, his research group has focused on Graph-Based Neural Dynamics (GBND), which represents an environment as a set of nodes connected by edges to model interactions. The group has also produced a series of studies on manipulating deformable objects, such as kneading dough.
He noted that this body of work culminated in a review paper published last year in Science Robotics, where he served as the corresponding author. The paper surveys a decade of learning-based dynamics models for robotic manipulation. Its overall structure was built upon the framework of his doctoral dissertation. He explained that this reflects the group's core priorities and deep insights when constructing such learning-based dynamics models.
Based on the published record, this review is titled "A review of learning-based dynamics models for robotic manipulation," released on September 17, 2025.
Following this series of studies, the group began considering its next steps: modeling more complex tasks and objects, handling not only homogeneous but also heterogeneous interactions, and extending the approach to open environments.
This led him to propose a new perspective on data. What is required for large-scale imitation learning? Action-conditioned robot interaction data. What is needed to build a world model? Again, action-conditioned robot interaction data.
He argues that using this data solely for imitation learning is a waste. The data contains rich signals that can be used to build models, which in turn can be applied to train and evaluate robotic systems.
The physics side: training in simulation, error-free for an hour
Returning to that spectrum, a major effort the group has pushed forward leans toward the physics side: applying hybrid modeling to the environment, and precisely modeling dynamics and interactions. He noted that this work was conducted at World Labs.
He believes that results of this caliber remain, to date, among the best achievable when transferring real-world environments into simulation. Both in appearance and dynamics, the simulation aligns closely with reality.
He showed several open-loop playback videos: the same pre-recorded motion executed on the left in the real world and on the right in simulation. The model captured the elasticity and plasticity at the box joints, and could also reproduce inserting a panel into a very tight slot, again in open-loop simulation.
Beyond these elastoplastic boxes, they also modeled cables. He noted that almost all cables in the real world bend and twist rather than remaining perfectly straight, and their framework models these bends and creases with high precision, achieving excellent sim-to-real alignment. The same framework can also model more rigid objects, such as inserting a tube into a very tight slot. Because it is simulation-based, swapping out the robot, the environment, or the task is highly convenient.
He believes the primary advantage of simulation is its controllability: object combinations, clutter, physical parameters, robot states, and camera settings can all be adjusted individually. Systematically varying the environment along these dimensions enables comprehensive coverage of the states a robot might encounter in the real world.
One of the results he presented was a mobile manipulation policy trained entirely in simulation and then transferred to the real world zero-shot. He specifically emphasized that the training data contains absolutely no real-world data; it is entirely simulation-generated.
Anticipating that the audience might suspect the video was cherry-picked, given how many similar examples they have already seen, they have consistently strived to prove that highly robust policies can be achieved in the real world: they recorded a full hour of the robot continuously executing the task, with zero failures and no human assistance.
When someone casually tossed a box onto the table, the robot simply kept working without failing. He used this result to demonstrate that robust sim-to-real transfer is genuinely achievable, not just for the sake of a demo.
Because the policy was exposed to a wide range of variations in simulation, it also remains robust to changes in ambient lighting: they altered the room lighting and even shone a flashlight directly at the scene, and the policy handled these changes without issue. This framework has also been applied to various robot types in the lab. For tabletop manipulation, it handles tight-tolerance insertions and object manipulation; it also works for mobile robots, such as inserting a water pipe into a port with equally tight tolerances, achieving highly reliable results relying entirely on simulation.
Using Simulation to Identify the Boundaries Between Success and Failure
Beyond policy training, he considers policy evaluation to be another highly valuable capability. When conducting an evaluation, the core question is where success ends and failure begins, and exactly where the boundary between the two lies.
To demonstrate this, he deployed a policy trained exclusively on real-world data to perform an object handover task, running it in a closed loop in both the real world and simulation. The first row shows the real-world environment, and the second row shows the simulation. By design, the robot is intended to grasp the middle of the object, as this region lies near the boundary of the data used for their post-training.
In successful trials, the qualitative behavior in simulation and the real world matches remarkably well. When the object is placed outside the post-training distribution, both systems fail, and their failure modes are strikingly similar. This helps illustrate how simulation can reproduce failure modes.
Quantitatively, he notes that, like others in the field, they plotted a correlation graph with the real-world success rate on the x-axis and the simulated success rate on the y-axis. He describes this as a fairly standard practice, but they took it two steps further.
The first step is to use the simulation environment as an evaluation engine for selecting checkpoints, plotting the number of training steps on the x-axis and the success rate on the y-axis. He presented the curves for π0.5 and GR00T N1.6, showing that most checkpoints can be filtered out in simulation first, eliminating the need to test them all in the real world.
The second step is to map the boundary between success and failure. In the resulting plot, the large dots in the foreground represent real-world evaluations, with the task executed hundreds of times in the physical environment; the small dots in the background represent simulation evaluations, run thousands of times. The simulation and real-world results align closely, revealing very clear boundaries between success and failure, as well as between generalizable and non-generalizable cases.
He gives the example that if the budget only allows for collecting 1,000 more demonstrations, this plot makes it immediately clear which ones to collect. Relying solely on real-world evaluations to generate such a plot would be a massive burden for anyone who has conducted large-scale evaluations. He is highly enthusiastic about these results.
sourceContext
adjacentContext
entityGlossary
task
output
blocks
Exactly one English string per input block, in the same order. Do not merge, split or omit blocks; preserve every ZXQPROTECTED token
In line with published work, the practice of evaluating policies in simulation and analyzing the correlation between simulated and real-world success rates at checkpoints across different training stages is also featured in Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions (November 2025), a paper co-authored by Li Yunzhu.
Combining reconstruction and generation
He noted that the previous day, Fei-Fei Li of Stanford University and World Labs introduced Atlas, a world model that unifies reconstruction and generation: it performs reconstruction when observations are available, and generation when they are not, all within a video prediction-like framework. He added that his own work explores how such models can be made useful for robots.
For navigation, simply taking casual photos of large-scale environments enables 3D reconstruction to generate point clouds, which can then be used to realistically render the robot's movements and the camera views it would observe. Operating in a digital world also allows for appearance estimation and enables the robot to be transferred into different environments. Consequently, a single data collection session can be leveraged to place the robot into these digital environments for large-scale training and evaluation.
For manipulation, a few casual videos of a robot performing tasks can serve as the starting point. Using such models for novel view synthesis yields more accurate and comprehensive state estimation and shape reconstruction. Improved state estimation enhances the system identification of dynamic models, leading to better alignment between simulation and reality. Finally, systematically varying configurations generates scalable training and evaluation data.
He categorizes these approaches as falling on the mixed, slightly more physics-oriented side of the spectrum.
The learning side: 3D world models
On the learning side, he has also conducted extensive research. He introduced a 3D world model released by his research group last week, the 3D World Simulator, which can learn 3D dynamics and perform world modeling of environments.
The model takes actions as input to predict tactile signals, point clouds, and standard camera images. Its 3D nature ensures controllability: users can alter individual scene instances, extract necessary geometric information, and swap foregrounds, backgrounds, and objects, thereby providing a scalable source of training data for robots.
He presented a live demonstration, noting that the audience could visit the lab's website to interact with it themselves. Using a mouse, they can move the robot through the 3D environment, control it precisely, and open or close the gripper, with the model predicting interactions between the gripper and cloth in real time. This 3D representation is also applicable to common pushing tasks and to understanding object pose changes between fingers for in-hand manipulation.
He explained that the model's pipeline is fairly standard: once the input is received, it gathers all necessary information within a unified framework, learns a dynamics model in the 3D data space, and then renders images from any desired camera viewpoint.
He noted that many people immediately associate world models with video generation. He wants to challenge this notion, as he is a staunch advocate for 3D and his team is focused on 3D approaches.
In his view, a key advantage of the 3D space is its controllability. Here is the complete output of their model. Operating in 3D allows for the easy combination of different simulation components: adding the joint structure of a robotic arm or a static background, thereby concentrating the learning capacity and computational budget on the elements that truly require fine-grained understanding.
As he mentioned earlier, the 3D space allows control over environmental changes, enabling the plug-and-play use of neural, articulated, and static components—all within the same framework for scalable experience generation. Furthermore, this space allows for the re-rendering of tactile signals as well as multi-view camera observations.
A highly direct application remains evaluation. He ran policies trained exclusively on real-world data within this model: the first row shows successful policies, while the second row shows failed ones. As observed, the model excels at distinguishing between successful and failed policies. This 3D world model can also supply the necessary observations for the policies.
Beyond evaluation, his research group is also exploring training: they are training policies using data derived exclusively from this world model.
Cross-referencing publicly available work, this section aligns with "Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation," which was uploaded to arXiv on November 3, 2026.
The paper introduces a 3D world simulator that uses a hybrid particle-voxel autoencoder to compress point clouds into 3D voxel latent variables. Conditioned on actions, it predicts appearance, geometry, and tactile feedback. The simulator can be integrated with static backgrounds and kinematic robot models, and it provides browser-based interactive demonstrations for policy training and evaluation. The first author is Jimmy Wang, affiliated with Columbia University and the University of California, Berkeley, with Yunzhu Li as a co-author.
Q&A: Advantages of the learning approach and the role of agents
An audience member asked about the advantages of learning-based world models over physical simulation. Physical methods can also generate large volumes of images with decent accuracy, so what exactly is the benefit of the learning-based approach?
Yunzhu Li replied that while he has conducted extensive research on the physical modeling of environments, he still cannot claim to have a physical simulator capable of modeling everything in the surrounding world. Instantiating a simulator to match the messy real world remains a highly challenging task, even with the agent tools available today.
Therefore, he remains optimistic about this learning-based, data-driven approach; however, Yunzhu Li's research group strongly emphasizes that the model must be as controllable as possible. Controllability gives you variation; controllability gives you coverage, which is why it can still support robust evaluation and training of robot policies.
Another attendee asked: given how powerful agents are today, could we use agents to control the simulator and the robot, explore the environment, perform system identification, close the sim-to-real gap, and then train a policy?
He said it was a great question. Yunzhu Li's research group is already heavily utilizing agents in its workflows. For instance, in previous work on the real-to-sim pipeline, they can now employ a more agent-centric workflow to scale and automate these processes.
He argues that rather than competing with or fighting against agents, researchers should work alongside them and delegate specific steps in the pipeline to them. In his view, agents are highly valuable for concrete tasks such as building world models, validating them, generating data within these models, and executing the subsequent transfer back to the real environment.
He concluded his talk with a single takeaway: world models can serve as a scalable data source for training and evaluating robotic systems.
Li Yunzhu’s stance in this talk aligns closely with the third initiative of the workshop: he positions the primary value of world models in providing training data and evaluation for policies. His argument rests on two pillars. The first is data: the action-conditioned interaction data required for imitation learning serves as the exact raw material for building world models. The second is controllability: only by independently adjusting lighting, objects, robots, and cameras can a simulation systematically cover state spaces, and only then can evaluation accurately map the boundaries between success and failure.
He also acknowledged the shortcomings on both ends: purely learned models still struggle with overall physical consistency, while physics simulators cannot yet simulate everything. The direction he proposes is the development of 3D, composable, and controllable world models.
Among the results he presented, policies trained entirely in simulation operated without error for a continuous hour, and simulation-based evaluations matched hundreds of real-world evaluations—all built upon meticulous real-to-sim modeling. The next step is to determine whether learning-based 3D world models can deliver equally reliable evaluation results across more objects, more robots, and longer-horizon tasks, thereby giving researchers the confidence to use these results to decide what data to collect next.
Do Robots Need World Models? (CoRL 2026 Workshop Official Website, November 12, 2026)