Do Robots Need World Models? Live Vote Reverses After Finn and Max Li Debate at CoRL 2026

At noon on November 12, 2026, the 'Do Robots Need World Models?' workshop at the Conference on Robot Learning (CoRL) concluded its final debate session in the Brazos room of the JW Marriott in Austin.

At noon on November 12, 2026, the "Do Robots Need World Models?" workshop at the Conference on Robot Learning (CoRL) entered its final debate session in the Brazos room at the JW Marriott in Austin.

The workshop website broke the debate down into four motions: whether end-to-end policies can internalize the predictive structures required for robot behavior; whether general-purpose policies are easier to learn than general-purpose world models; whether explicit world models are necessary for reproducible and scalable policy evaluation; and whether existing benchmarks can tell us if world models are necessary. On-site, however, these were consolidated into a single question for the audience vote: Do robots need world models?

The affirmative position stated: Predicting the consequences of actions with world models is indispensable for reliable, general-purpose robot behavior. The opposing position stated: Reliable, general-purpose behavior can emerge from policies that use observation history, context and/or memory.

The teams were divided as follows: the world models and structured approaches side featured Max Zhaoshuo Li (NVIDIA) and Georgia Chalvatzaki (Technical University of Darmstadt); the end-to-end and implicit approaches side featured Chelsea Finn (Stanford University / Physical Intelligence) and Vincent Sitzmann (Massachusetts Institute of Technology / Rhoda AI). The moderator was Yunzhu Li (Columbia University / SceniX / World Labs). This is a debate with assigned positions, which do not necessarily reflect the panelists' personal views. The website noted that the grouping was designed to make the discussion more rigorous and does not necessarily represent each individual's personal stance on every motion.

Yunzhu Li opened with two examples to illustrate that both sides make valid points. A world model for Go is straightforward to build, as the rules themselves constitute the model; modeling the flow of water for pouring a glass requires intractable equations, yet the act of pouring is not that difficult—you just need to watch where the water comes out.

Before the debate began, an on-site vote showed that the majority believed robots need world models.

Opening: Predict first or optimize directly

Max Zhaoshuo Li spoke first. He argued that world models are foundational: you must be able to imagine the outcome before you can decide what to do. He anticipated the opposing side would argue that as long as a policy is conditioned on history, memory, and similar context, the action distribution can narrow and become more concentrated, allowing it to directly execute the correct action. He did not believe this reflects reality, especially when dealing with complex objects.

He drew an analogy to recent coding agents: after writing code, they execute the program and observe the results. He believes robots will operate the same way.

Chelsea Finn began by reflecting on her own experience. She worked on world models as early as 2016. What excited her at the time was the apparent path to generalization: policies trained back then could typically perform only a single task, and switching to a new one meant starting from scratch and retraining, which was highly unsatisfactory. Action-conditioned video models, however, showed early signs of life, generalizing to novel goals unseen during training. Yet by 2026, she argues that a different class of models has achieved zero-shot generalization: foundation models that predict actions conditioned on language, task videos, and other inputs can generalize to new objects, new scenes, and unseen combinations of robots and tasks, and have already been deployed in real-world use cases.

Her second argument centers on optimization: whenever you optimize, you should optimize for what you truly care about. Building a world model is merely a means to an end; the actual goal is for the robot to complete the task. If you directly optimize for task success using reinforcement learning in real-world deployment scenarios, she believes this is the most direct path to achieving reliable robots—and reliability is precisely what is lacking today.

Looking at published work, Finn's line of research around 2016 includes Unsupervised Learning for Physical Interaction through Video Prediction and Deep Visual Foresight for Planning Robot Motion.

Georgia Chalvatzaki followed up by noting that while there are already policies capable of successful deployment, robots are not yet ubiquitous in our surroundings. Her stance is that world models can make robot learning more efficient and effective, particularly in novel situations and under uncertainty. Even if a policy can incorporate memory and historical data, she argues that when the correct action is unclear or errors are difficult to correct, the robot should first evaluate potential outcomes before deciding to execute.

She outlined several applications for world models: generating training data; validating policies; enabling proactive robot intervention when faced with two visually identical objects that differ in mass or friction—such as pushing slightly harder to gather more information; and performing counterfactual reasoning after a failure by asking, "What if I had taken a different action?"

She also argued that uncertainty should dictate how robots allocate compute and distribute the evaluation budget for policies. Since different levels of decision-making require varying degrees of detail, modularity must be reconsidered—not in the traditional sense, but by developing new architectures and algorithms that scale with compute while providing predictions and counterfactuals. She also addressed Finn's points, noting that model-based reinforcement learning has existed since 2016; just because it was bottlenecked by certain limitations in the past does not mean models are useless.

Vincent Sitzmann presented two counterarguments. On the pragmatic side, the key uses of world models are, first, to imagine an action before executing it, much like chain-of-thought reasoning, and second, to train policies in simulation. For the former, he agrees with Chelsea's earlier point that direct end-to-end training might be sufficient; for the latter, he does not believe that real-world reinforcement learning is demonstrably insufficient. On the philosophical side, he noted that world models were previously called dynamics models and are essentially a human construct.

We define a box with specific inputs and outputs: taking actions and past observations as input, and outputting predictions of the future, then plugging it into some harness to execute tasks. He argues that physical intelligence will not develop this way. In the future, AI agents will be optimized to pursue goals far more general than any specific robotic task. If world modeling is useful, it will simply emerge as a component of this broader intelligence, eliminating the need for us to build a little box with human-defined inputs and outputs. Even if you believe chain-of-thought reasoning is useful, that doesn't mean you need to sit down and build a dedicated chain-of-thought box.

Rebuttal: Will post-training narrow the model?

Max Zhaoshuo Li’s rebuttal targets post-training. He states his bias is that post-training narrows and concentrates the output distribution, specializing the model for a specific task, which sacrifices robustness to diverse variations. He argues we cannot bypass the need for a model that both understands task intent and predicts the consequences of an impending action. In his view, this is one of the most fundamental capabilities of a generalist agent.

Chelsea Finn responds that narrowing training to a single task is purely a matter of convenience: it is easier to study algorithms within a specific domain and environment. She believes future post-training will resemble that of language models, operating over a much broader scope that covers numerous tasks and environments while maintaining high performance, all without requiring world models.

She shared an anecdote about experimenting with a policy. The task of assembling a pinwheel was later distilled into the base model. Prior to that, she had performed post-training specifically for the pinwheel task. One step involved inserting a small pin into a tiny hole at the center of the paper. The data she used only demonstrated inserting the pin with the right hand, representing a very narrow, single-policy approach.

While watching the evaluation video, she noticed the robot accidentally dropped the pin on the left side of the table. It then picked up the paper with its right gripper, retrieved the pin with its left gripper, and successfully inserted it. She noted that neither the post-training nor the pre-training data demonstrated this symmetry between the left and right grippers. Yet, even after being post-trained on a very narrow, single-policy dataset, the model still understood that this step could be executed with either the left or the right hand. She argues this demonstrates that even when post-training narrows and concentrates the distribution, the model retains its generalization capabilities.

Georgia Chalvatzaki begins her rebuttal with animals. She references examples shown earlier that day of animals solving puzzles, noting that numerous studies demonstrate animals use forward models to make predictions and base decisions on them. As experience accumulates, uncertainty collapses, and knowledge is internalized into policies that become everyday behavior. Therefore, she argues, the affirmative side is not opposed to policies.

The real question is: as systems become more intelligent, required to interact with humans, and subjected to additional constraints, wouldn't you want a system capable of predicting the effects of its own actions? She believes video models already contain a wealth of knowledge that can guide robotic decision-making at an abstract level and facilitate counterfactual predictions. However, she argues that simply collecting more data and scaling up the model is insufficient; we need to deeply consider how to make the architecture itself more efficient.

Vincent Sitzmann’s rebuttal focused on the coverage of learned world models. Because they can only model what they have seen in the past, learned models are constrained by their generalization capabilities. He argued that deploying world models on real robots requires continuously learning them online from the agent's stream of experience. Yet no current system can achieve this in the real world, and no deep learning model can reliably learn online.

Furthermore, once extensive online learning is required, the world model becomes a function of the policy, since the training data is collected by the policy itself. At that point, it becomes difficult to distinguish where the policy ends and the world model begins. Therefore, he advocates addressing the problem holistically.

An audience member asked Sitzmann: A video planner generates and predicts the outcomes of executing actions in a specific environment—does it count as a world model?

He noted that there is a slight semantic issue here. A video planner is not a pure dynamics model; it is entangled with the policy. To be mathematically precise, one must clarify when the model is a function of the agent's policy and when it is not. By this standard, a video planner is closer to the agent's policy. However, he acknowledged that fine-tuning an action-conditioned world model on top of a video planner works remarkably well. The two share extensive infrastructure, and the video planner itself must model dynamics, meaning it is also fundamentally a dynamics problem.

Evaluation: Sandbox or real environments

Yunzhu Li first addressed the affirmative side. He mentioned that he has been encouraging his students to work on world models, but their counterargument is that world models are simply too difficult to build. Policies trained on demonstration data can already accomplish interesting tasks in real environments, and since evaluation must ultimately take place in the real world anyway, why build world models at all?

Georgia Chalvatzaki agreed, noting that training world models is currently difficult and there is no established recipe yet. She suggested one reason is that the prediction target remains unclear, and explicit prediction may not even be the right approach. She has always believed that models should serve as tools across different levels of the decision-making process, and the level of abstraction currently used for learning these models might simply be incorrect. She offered an example: when taking the elevator back to her room, she can roughly simulate the route in her head. The uncertainty is high, but it is sufficient to form a plan, allowing her to update her judgments as she goes.

Therefore, building a true digital twin for the entire process is not necessarily a prerequisite for making informed decisions. She views chain of thought as one way of modeling within a certain space, but integrating semantic understanding with geometric outcomes to compose behavior remains an open problem, meaning representations need to be rethought.

Max Zhaoshuo Li responded that world models and policies model the exact same state transitions, meaning they do not need to be separate systems and can instead be unified into a single framework. Using language models as an example, he argued that every text token generated conditioned on a task is effectively modeling this transition.

Yunzhu Li then directed a question to the opposing side. He recalled that two years ago, a friend working in natural language processing started building agents, which suddenly exposed him to the core pain points of embodied research: agents must interact with their environment, and evaluation must occur in a closed loop. Today, teams developing large language model agents have built extensive closed-loop workflows and sandboxes specifically for evaluation.

He noted that when trying to distinguish between two checkpoints with 90% and 92% success rates, for instance, the efficiency of evaluation in a real environment versus a sandbox differs by several orders of magnitude. Given how successful world model-based sandboxes have been for agent evaluation, how does the opposing side view this?

Chelsea Finn responded that to use a learned world model for evaluation and ultimately achieve efficiency gains, the world model must be able to predict the consequences of arbitrary action distributions, requiring broad coverage across diverse action distributions. For example, pouring wine into a glass requires modeling how the liquid reacts, while accidentally knocking the glass to the floor requires modeling how every shard scatters. Training a policy, by contrast, only needs to implicitly learn the consequences of a specific subset of actions.

During reinforcement learning, the agent tries both successful and failed actions, but in a highly targeted manner. It quickly converges on successful actions within the optimization loop, eliminating the need to evaluate every possible action in the real world. Therefore, taking a detour through a world model—and requiring it to model every possible action—will be slower than directly optimizing the policy in the real world, which only considers the actions actually encountered during the optimization process. She attributes the core issue to the fact that world models are generally much harder to learn than policies, with the key difference being whether the model must account for all possible actions or only those encountered during the learning process.

Vincent Sitzmann stated that his argument aligns perfectly with Chelsea's. Ultimately, what needs to be modeled depends entirely on what is being learned and what the objective is. He believes that drawing a line in advance to dictate which actions and states the model must remain robust to is a losing game; instead, the agent should decide for itself what the world model needs to capture and which actions to take.

Definition: Does predicting consequences count as a world model?

The audience raised numerous questions regarding definitions. Yunzhu Li relayed one: does predicting the consequences of actions in pixel space or latent space count as a world model? And what if it predicts the actions simultaneously—does that still count?

Max Zhaoshuo Li spoke first, noting that whoever answers this question first will likely face criticism later. His answer: as long as you are modeling consequences, it counts. When learning policies, we are often actually modeling consequences; if you probe the model, you can see signs of this—it avoids certain policies and tries other approaches.

In his view, this is the definition of a world model: understanding consequences. He also noted that, for him, building a world model is much harder than taking actions, because it requires understanding the consequences of both good and bad actions, which he finds to be the more interesting problem.

Georgia Chalvatzaki reiterated that world models are far from easy and that she still believes we are learning them the wrong way. Humans spend a lot of time playing in their earliest years; this play is not goal-directed but mostly driven by curiosity, which in turn stems from a pursuit of knowledge and information. She believes that letting robotic intelligence grow out of this kind of interaction with the world is a very exciting direction. Reinforcement learning can enable agents to solve tasks, but it yields greedy policies and offers few alternatives. If we can train models through intervention, exploring alternatives, and in a goal-agnostic manner, we can then build on that foundation to explore other approaches.

Vincent Sitzmann returned to the issue of coverage. Modeling outcomes for every possible action, he believes, is something humans themselves cannot do, and the required training data would be disproportionately massive. In the deep learning paradigm, every capability of a model comes from data, which includes the methods used to generate that data; therefore, our work is largely about curating data or designing algorithms to select data. If we take a uniform prior over all possible actions, we would have to sample all the data, which he considers highly intractable.

Wrap-up: essential or useful?

Finally, Yunzhu Li asked each panelist to briefly share their genuine views.

Max Zhaoshuo Li said that his conception of a world model refers to truly understanding the transition dynamics of every action, which is entirely distinct from an action-conditioned video generation model; in his view, this is the foundation for any policy evaluation.

Georgia Chalvatzaki said that video models might provide a good starting point, but the key is to truly understand physics; only after solving this can we proceed to counterfactual prediction. This is a difficult problem that requires deeper thought; meanwhile, researchers are pushing the limits of vision-language-action (VLA) models to see how far they can go, while another line of research must also be pursued in parallel.

Vincent Sitzmann argued that from a technical standpoint, the focus should be on whatever works, as it can aid evaluation and solve broader problems; this is ultimately applied science and an engineering challenge. Philosophically, however, he does not believe that the advent of physical intelligence will be accompanied by a world model where we first define the state space and then make action-conditioned predictions. Instead, if agents can predict the consequences of their own actions, as he expects, that capability will simply emerge as a side effect.

Chelsea Finn noted that world models are a worthwhile and fascinating research topic, with certain aspects proving highly useful for model development. For instance, simulations that do not need to capture every possible interaction could be invaluable for policy evaluation and might even improve policies to some degree. However, she considers it unlikely that they are strictly necessary; they may simply help us get there faster.

The debate's core disagreement ultimately boils down to a testable question: which actions should be modeled. Finn and Sitzmann argue that requiring a world model to predict reliable outcomes for any arbitrary action incurs prohibitive costs in coverage and data. They favor directly optimizing only the actions considered during the optimization process, suggesting that if world-modeling capabilities are useful, they will naturally emerge from more general objectives. Conversely, Max Zhaoshuo Li and Chalvatzaki argue that policies capable of merely executing the correct action fall short precisely in novel, high-uncertainty situations that require comparing outcomes and reasoning counterfactually. Chalvatzaki also acknowledged that there is currently no definitive recipe for this, noting that the challenges likely stem from the level of abstraction and the learning algorithms used.

The two sides found the most common ground on evaluation. The moderator highlighted the efficiency gap in sandbox evaluations, Max Zhaoshuo Li described world models as the foundation of policy evaluation, and Finn concluded by agreeing that simulations not requiring the capture of all interactions would be highly useful for evaluating policies. The divergence lies in whether such models are merely a tool or a strict prerequisite: the opposition views them as an accelerant, while the affirmative sees them as an essential capability for generalist agents. Settling this debate will require empirical evidence showing which approach is more reliable under identical data and compute constraints—systems with explicit prediction or directly optimized policies—when deployed with novel objects in real-world environments.

Following the debate, a revote was held, and the votes flipped.