Chelsea Finn: End-to-End Policies Can Handle Long-Horizon Tasks Without Explicit World Models

At the CoRL 2026 workshop 'Do Robots Need World Models?' in Austin on November 12, Chelsea Finn presented research showing that end-to-end policies equipped with implicit supervision, multi-scale memory, and detailed multimodal prompting can manage long-horizon tasks without building explicit world models.

On the morning of November 12, the Conference on Robot Learning (CoRL) hosted a half-day workshop titled "Do Robots Need World Models?" in the Brazos Room at the JW Marriott in Austin. According to the workshop's official description, a world model is defined as a class of predictive systems that enable robots to anticipate how their actions will alter the environment.

The core debate centers on whether this predictive structure must be explicitly constructed to facilitate planning, counterfactual reasoning, data efficiency, safety, and generalization, or whether it can be absorbed into an end-to-end policy that directly maps observations and goals to actions.

Chelsea Finn, an assistant professor of computer science and electrical engineering at Stanford University and co-founder of Physical Intelligence, delivered the first invited talk of the day, assigned to the side advocating for end-to-end, implicit policies. This stance was assigned by the organizers and does not necessarily reflect her personal views. She opened her talk by stating that she actually stands on the other side of the table. The first motion listed for the workshop was precisely that: end-to-end policies can internalize the predictive structures required for robotic behavior without attempting to explicitly model the world.

Finn began by noting that although her earlier work leaned toward explicit models, her experience shows that robots can already perform highly complex tasks without an explicit predictive model of the world. These tasks include reasoning over long context histories, completing long-horizon tasks, and operating in scenarios that require generalizing the consequences of actions.

Her central question is how to deploy robots in complex scenarios and whether it is truly possible to learn policies capable of handling such complexity without explicitly modeling the world.

Her talk was divided into three parts: handling long contexts, such as tracking past events; completing full long-horizon tasks, such as cleaning a kitchen; and reasoning in novel situations, such as transferring an existing task to a different robot.

Adding Memory Can Sometimes Be Worse Than Having None

She first pointed out that many powerful robot foundation models actually lack memory or context. While this is unnecessary for repetitive, short-horizon motor skills, memory of the past becomes critical once the system needs to track past events or handle long-horizon tasks. Yet, existing models still frequently lack memory. She then outlined two main challenges.

One challenge is the learning signal itself. Equipping a model with extensive memory can introduce spurious correlations between past observations and future actions. As the context grows and more information is fed into the model, the number of spurious correlations between inputs and outputs increases, making it harder for the model to learn generalizable action predictions. She noted that this issue is severe enough that adding memory sometimes degrades performance compared to having no memory at all.

The second challenge is computation, which applies to both training and inference. Rather than processing just one or two images, these systems aim to ingest hundreds of frames, increasing the computational load by several orders of magnitude. This issue becomes more pronounced as the context size grows. Inference must also be performed in real time, making training even more demanding. Existing work addresses this by using only a few strided frames, as in PTP by Torne, Tang, Liu, et al. at CoRL 2025; using keyframes, as in MemER by Sridhar, Pan, et al. at ICLR 2026; and employing test-time adaptation, as in RoboTTT by Jiang et al. at CoRL 2026.

While these approaches do enable policies to act based on past events, they somewhat sidestep the computational burden. She argues that in the long run, robots will still require highly capable context. The first question to address is whether the memory of a vision-language-action (VLA) model can be scaled to hundreds of frames, and whether a usable training signal can be extracted from that volume of images.

To tackle the problem of spurious correlations and the difficulty of learning from massive numbers of frames, they found a highly effective approach: adding implicit supervision to the model alongside action prediction. This involves predicting both the current task being performed and generating a narration of what is happening in memory. This continuous narration helps the model learn features and better understand past frames and video sequences.

Specifically, they use a vision-language model (VLM) as the backbone, training it to process longer videos. The backbone is Qwen3.5-4B, a 4-billion-parameter model with a context length of 262,000 tokens. The model features three output heads: one for reasoning, which provides a chronological narration of events so far (e.g., 'At second 2, the ball was placed under the middle cup. At second 5, the middle and right cups were swapped'); one for subtasks (e.g., 'pick up the left cup'); and one for action tokens. The narration head employs parallel decoding, requiring only a single forward pass. The action tokens use OAT, grouped as 1, 1, 2, 4, 8, and 16.

Computationally, they avoid repeatedly recalculating already-processed past frames. Instead of recomputing the past video from scratch during each inference step, they cache existing computations and stream the video into the model, eliminating the need to repeatedly recalculate activations for past frames. Because autoregressive generation of narration text would be slow, they implement parallel decoding for the narration, allowing the model to predict actions at test time without first generating the narration. On the action side, they use discretized actions and group them to reduce the number of forward passes.

Integrating these components, they evaluated the approach on memory-intensive tasks like cup swapping, which requires tracking object locations. The subtask in the demonstration is 'pick up the middle one,' requiring the model to chronologically track when cups are covered and which two cups are swapped to ultimately select the correct one. She noted that this is not her favorite task, as it does not represent a real-world robotic necessity, but it serves as an excellent sanity check: the model cannot simply retrieve a single frame but must reason over the entire video sequence to succeed. The model was fine-tuned for this task and demonstrated generalization to unseen slots or combinations during training. She also emphasized that the system processes 1x speed video; despite the extensive memory context, the inference speed does not slow down the robot's physical movements, and the resulting actions remain smooth.

Quantitative results cover three tasks: cup swapping, drawer, and visual cue. Baseline methods include π0.7, MemER, and two ablation studies: predicting only subtasks, and predicting subtasks plus narration. Their proposed method combines subtasks, narration, and latent narration. The context length of π0.7 is insufficient for these tasks.

For the cup swapping task, their method achieved a 100% success rate on the 5 seen mixtures during training, 93.3% on the 5 unseen mixtures, and 73.3% on the 6 unseen mixtures, all outperforming π0.7, MemER, and the two ablations. On the drawer task, π0.7 and MemER had very low success rates, at 6.7% and 16.7% respectively in the training setting, while their method achieved 83.3% in both the training and unseen settings. In the visual prompt task with 3 training objects, π0.7 achieved 70%, while MemER and "subtask plus narration" both reached 100%; on the 5 unseen objects, π0.7 scored 20%, MemER 80%, and "subtask plus narration" 90%.

They also quantitatively analyzed the relationship between the number of context tokens and latency, comparing the curves for computing from scratch versus using caching on H100 and B200 GPUs. Without caching past computations, the latency curve rises sharply; adding caching significantly improves this. Inference for 900 frames can be completed in under 0.5 seconds, corresponding to 64,500 historical visual tokens.

Her conclusion for this section is that by incorporating task and narration predictions into end-to-end training and handling the computation efficiently, one can achieve highly effective long-context reasoning in robot policies without needing to build a separate explicit world model.

Cleaning an Entire Kitchen

Next, she turned to longer-term memory. The relatively dense visual memory discussed previously is reasonable for many current scenarios; however, if the goal is to cover hours or days, one might not want the memory to remain strictly visual and that dense. Medium-term visual memory is dense and observation-based, whereas longer-term memory is sparser and more semantic.

Examples she cited include remembering what items to buy at the store and following a recipe to cook in the kitchen. The idea, therefore, is to create a multi-scale memory model: using video memory for short- to medium-term horizons, which can follow the approach from the previous section; and using more compressed text to represent longer-term memory spanning many minutes. Both scales of memory are used within an end-to-end policy.

This architecture is called Multi-Scale Embodied Memory. The upper layer is a vision-language model that holds long-term text memory and cyclically updates memory summaries; the lower layer is a vision-language-action model that uses short-term video memory and outputs robot actions through an action module.

For text memory, she described two approaches. A simple approach is to feed the language of subtasks completed or attempted so far directly back into the model as a compressed representation of the past. This method is straightforward but often causes distribution shift between training and deployment—especially when the execution methods and speeds are inconsistent with the data. For instance, if the actual execution takes longer, certain content will appear repeatedly in the input far more times than during training.

They found a more effective approach: compressing and summarizing past events into a new summary based on the previous summary, the current subtask, and the current visual observations, then feeding this summary into the model instead of the raw subtask list. In other words, the model is trained to update this text memory and compress it into a summary. This helps mitigate distribution shift, yields a more compact representation, and is better suited for longer time horizons. The pre-trained vision-language model for this component comprises SigLIP (400 million parameters) and Gemma (4 billion parameters).

With this text memory, they extended the robot's task horizon to approximately 15 minutes in this work. The tasks involved cleaning a kitchen and preparing a recipe. She showed a video where the bottom of the screen displays what the robot has accomplished so far. Memory is crucial here because the tasks involve many steps. The process she described includes wiping the counter with a sponge, drying it with a paper towel, updating the summary, throwing the paper towel in the trash, putting items back in the refrigerator, and eventually washing a plate and putting away the air-dried dishes. She noted that the video was quite long and there was not enough time to watch it in full during the presentation.

Quantitatively, they evaluated π0.6-MEM—a hybrid of this visual memory and very long-horizon memory—against π0.6 without memory, naive text-plus-video memory, video memory alone, and text memory alone. They evaluated two tasks: preparing a recipe and cleaning the kitchen, using task progress as the metric. The main conclusion is that for such long-horizon tasks, especially kitchen cleaning, π0.6-MEM performs significantly better than the version without text memory and also outperforms the more naive text memory approach. She believes this compressible summary approach still has room to further extend the task horizon.

Compared to subsequently published work, this component aligns with the Multi-Scale Embodied Memory (MEM) framework released by Physical Intelligence in March 2026: using video memory for short horizons and language summaries for long horizons, validated by a kitchen-cleaning task lasting approximately 15 minutes.

Compositional Generalization in Novel Contexts

She noted that another major source of complexity for robot models is generalization to novel contexts, especially out-of-distribution scenarios, and particularly compositional generalization. Researchers are often tempted to hardcode structural inductive biases into their algorithms to facilitate generalization. Instead, they wanted to see if this capability could emerge organically simply by training a sufficiently high-capacity model on diverse data. The basic recipe is straightforward: sufficiently large and diverse data paired with a model of adequate capacity.

Regarding data, they leveraged every available source, including diverse robot demonstrations—which encompass low-quality demonstrations—robot policy rollout data, human videos, and web data. Diverse data alone, however, is not enough. They found that providing the model with detailed contextual prompts is crucial and yields significant improvements; even with lower-quality data, this context helps the model predict actions accurately. Low-quality demonstrations can remain in the training set, provided the model is given sufficient information about what is being attempted in that specific episode, how well it is being executed, and the exact nature of the current step.

They trained the π0.7 vision-language-action (VLA) model using detailed multimodal prompts. The pre-trained backbone comprises SigLIP (400 million parameters) and Gemma (4 billion parameters), accompanied by an 800-million-parameter action module. The inputs include observation memory; task instructions, such as 'clean the kitchen'; finer-grained subtask instructions, such as 'pick up the knife'; subgoal images; and execution metadata, such as data quality. During testing, subtask instructions can be provided by a human or predicted by a high-level policy, which is also pre-trained using the SigLIP and Gemma backbone. The subgoal images are generated by a world model pre-trained as BAGEL (14 billion parameters), taking the current observation, subtask instruction, and metadata as inputs.

After training on diverse data, the model can directly perform fairly complex tasks, such as buttoning a shirt, driving screws that require high precision, and taking out the trash. The team also evaluated it in out-of-distribution scenarios. Finn noted that the most interesting case involved a dual-arm UR5e robot that had never been used to collect laundry or clothes-folding data, yet was tasked with folding clothes. The folding data had been collected on a fixed dual-arm robot with two forward-facing arms, which is relatively inexpensive. Testing, however, was conducted on the dual-arm UR5e, which is significantly larger and has a distinctly different kinematic configuration. Despite this highly out-of-distribution control setting, the model demonstrated strong compositional generalization, successfully transferring skills learned on one robot to another.

Quantitatively, the evaluation was divided into towel folding and shirt folding, with π0.5, π0.6, π0.7, and π0.7 (GC) serving as baselines. π0.5 achieved near-zero task progress on both tasks; π0.7 outperformed π0.6; and π0.7 (GC) achieved the highest score. For the shirt-folding task, a human performance baseline was also established, and π0.7 (GC) approached this line. Despite the lack of dedicated clothes-folding data for this specific robot, its performance was close to that of human teleoperation.

Compared with subsequently published work, this section aligns with Physical Intelligence's π0.7 release in April 2026: the model was trained using detailed context, subtask instructions, execution metadata, and subgoal images, and evaluated on clothes-folding tasks using a robot that lacked corresponding folding data.

After the presentation, an audience member asked whether memory should be compressed through some mechanism for tasks spanning longer time horizons—for instance, placing a plate on a specific shelf today and still knowing its location the next day. This question directly followed up on her earlier approach of partitioning memory into denser visual representations and compressed text.

Finn responded that in many scenarios, particularly everyday, multi-day tasks, memory compression is indeed necessary. On the other hand, having the capability to feed fairly dense context directly into the model is highly advantageous, as people often do not know exactly what kind of memory representation they want. Implementing compression typically requires humans to first decide what to compress the data into and how to do it. By directly feeding in video along with some additional supervision, the model gains more flexibility in how it utilizes this information. While supervision does not entirely eliminate the challenges of representation and compression, it enables the model to leverage all available information rather than just a pre-selected subset.

She concluded with a core judgment: As long as the data, supervision, and context are properly utilized, these end-to-end models can handle highly complex situations—including those requiring long contexts, the completion of lengthy, multi-step tasks, and reasoning in novel scenarios—without needing to explicitly integrate world model mechanisms into the architecture, or specifically construct a dedicated world or physical model. Simpler recipes are also generally easier to scale across diverse data and complex scenarios. She also noted that these models have already begun entering real-world use cases, such as logistics.

However, she acknowledged that longer time horizons remain an open challenge. For memory spanning hours or days, she believes it no longer needs to be kept so dense or purely visual; remembering which shelf a plate is on across multiple days, for instance, should involve compression. Yet, compression again typically requires humans to first define the target representation. What truly remains to be verified is whether this approach—without relying on explicit world models—can still retain necessary facts and stably transfer skills on tasks that extend beyond kitchen cleaning and on entirely unseen robots, all without hardcoding the memory representation or reintroducing explicit predictive models.