NVIDIA and MIT Researchers Show Long-Context Robotics Requires Autoregressive World Models
Long-WAM found that "having access to history" is not the same as "knowing how to use history": only when the video foundation model first learns to predict the future from the past in an autoregressive manner can extending the context truly drive performance gains. Based on this conclusion, the team built this world action model into a system capable of real-time operation on edge devices. After expanding the context duration from zero seconds to 19.2 seconds, RoboCasa GR-1's task success rate rose from 63.3% to 78.7%; its success rate in the real-robot dynamic cup-stacking task reached 95%, whereas the control groups π0.5 and Fast-WAM failed all twenty tests.
On October 7, a joint team from NVIDIA, Massachusetts Institute of Technology, The University of Hong Kong, and University of California San Diego published the paper "Long-WAM: Scaling the Context of World-Action Models." The paper has sixteen authors; Wei Huang and Bohan Zhang are co-first authors. Jim Fan, co-lead of NVIDIA GEAR, is a core participant in this research. As a doctoral graduate under Fei-Fei Li at Stanford University, Jim Fan is a representative Chinese scholar in embodied AI who has led well-known projects such as VIMA, MineDojo, Voyager, and Eureka, with long-term focus on general-purpose robot agents and world models. Song Han from Massachusetts Institute of Technology and Xiaojuan Qi from The University of Hong Kong also appear on the author list.
World action models integrate video prediction into the robot control closed loop, making visual history a natural information source for action decision-making. However, reading longer histories slows down response speed, raising the core question: to what extent can robot control extend visual context, and how should performance gains be preserved under real-time constraints? Recent research mostly relies on causal caching or memory modules to retain history, but this team identified a deeper key point: "being able to access history" and "learning to predict from history" are two entirely different things. Existing solutions either adapt bidirectional pre-trained video generators into causal video-action predictors, or jointly pre-train causal videos with learned latent actions. Long-WAM opens a third path: first completing the learning of autoregressive video prediction independently, then connecting it to the action module, while fully preserving the temporal sequence structure from "history to future."
01 Teaching Models to Leverage History
The research starting point for Long-WAM came from a set of controlled experiments. The team connected three video foundation models to the same world action model and gradually extended the context window from 0 seconds to 19.2 seconds on the RoboCasa GR-1 platform: the bidirectional pre-trained foundation model showed almost no net gain in adding history, with performance rising slightly from 61.7% at 2.4 seconds to 64.1% at 9.6 seconds, before falling back to 61.6% at 19.2 seconds; whereas the success rate of two autoregressive foundation models continued to rise. Among them, the foundation model pre-trained for the robotics field saw its task success rate increase from 63.3% to 78.7% as the context expanded from 0 seconds to 19.2 seconds. Its performance advantage over the bidirectional foundation model also grew from 3.3 percentage points under zero-history conditions to 17.1 percentage points at 19.2 seconds. Getting longer history and learning to predict from it are two different things; only by first teaching the foundation model to "use the past to push the future" does the extra history translate into score gains.
This foundation model is named LongLive2.0-Robot. The team continued training on an off-the-shelf 16-second autoregressive video model, using robot videos and first-person-view videos equivalent to approximately ten thousand windows as training data. The data sources include five datasets: RoVid-X, AgiBot World, EgoDex, EgoVerse, and VITRA. Training uses only video as the supervision signal and does not rely on action labels, allowing the model to be compatible with data from various robot bodies without requiring a unified action space. Pre-training targets sequences up to 30 seconds in length and employs block causal attention; each noisy block is supervised by a ground-truth prefix, while condition images remain clean and are excluded from loss calculation.
The core of integrating the action module lies in preserving the causal temporal sequence learned during pre-training. The video and action branches are coupled through an asymmetric interface: the video query reads only its own and earlier visual blocks, without accessing action information; the action query can read both the observation history and some denoised future information, before reading the complete noisy action block. In this way, the "past-to-future" temporal relationship is already learned during the pre-training phase, rather than being constructed ad hoc after connecting the action module. When outputting actions, the video branch first predicts subsequent latent variables to a specified noise level, providing the corresponding key-value cache for reuse by the action experts. The team has named this "predict first, then generate actions" paradigm inverse dynamics modeling.
One detail warrants separate emphasis: during inference, the video branch does not decode future latent variables into pixel frames; it retains them at an intermediate noise level and supplies the cache to the action module for denoising. Here, prediction is no longer a task of generating complete video but serves as an intermediate representation for robot control. Ablation experiments also corroborate the value of this design: the approach that predicts the future before outputting actions achieves a 99.5% success rate on the LIBERO-Long task; the joint denoising scheme reaches 97.8%; and the baseline version without future prediction scores only 94.5%.
Viewing the approach within the broader technology roadmap brings clarity. Existing solutions fall into two categories: one modifies bidirectional pre-trained video generators to transform them into causal video-action predictors; the other jointly pre-trains causal videos with learned latent actions. Long-WAM takes a third path: it first independently learns autoregressive video prediction, then integrates an action module while preserving the native causal structure throughout the integration process. The team explains that autoregressive pre-training embeds the ability to "infer the future from history" into the representation. When integrating actions, this mechanism remains intact, allowing newly added historical information to flow smoothly into the control pipeline. In contrast, bidirectionally initialized models have not acquired this temporal logic; no matter how much historical data is stored, they cannot effectively utilize it.
Pre-training and action alignment are split into two independent stages. The first stage learns only the motion patterns in video frames; the second stage integrates the action module, employing a two-round alternating training process: first matching the video stream based on clean historical data, then matching the action stream based on historical data and noisy ground-truth future states. In the second step, the visual cache is disconnected, and the action loss updates only the action expert and the body adapter. The advantage of this split design is that action learning does not interfere with the causal prediction capabilities already learned during the pre-training stage.
02 Fitting Long Contexts into Real-Time Control
Once long-horizon capabilities generate value, the next challenge is efficient computation. The pre-training phase consumed approximately 30,720 GPU hours, completed across 64 NVIDIA H100 GPUs; the downstream world action model was trained on 16 GB200 units using the AdamW optimizer, with a peak learning rate of 1e-4, cosine decay, 5% learning rate warmup, and BF16 precision. The team considers context length and response time within the same dimension: longer prefixes simultaneously increase prefilling, attention computation, and cache overhead. Therefore, context is treated as an execution resource that incurs costs, rather than something to be maximized simply for being longer.
On the real-time inference side, the team adopted a scheme where inference and robot execution overlap. Control steps are triggered at fixed intervals, with inference and execution proceeding in parallel; during module handoffs, expired prefixes are discarded, and suffixes aligned to new blocks are executed, waiting for synchronization if latency occurs. This purely asynchronous mechanism could originally disrupt motion continuity, with conventional solutions either adding guidance constraints during inference or action conditions during training. The team found that as long as the model relies on temporally continuous video prediction as a condition, outputs from adjacent blocks naturally converge in overlapping regions, allowing stable operation using only the pure asynchronous mechanism. In RoboTwin 2.0 tests, Long-WAM showed almost no performance degradation when switching from synchronous to asynchronous mode; by contrast, Fast-WAM dropped by 15.4 percentage points, and LingBot-VA dropped by 45.5 percentage points.
Another speedup design appears in the image encoding stage. The streaming causal VAE begins encoding as frames arrive, completing prefix observation encoding before inference triggers; once inference starts, it only processes remaining frames. In contrast, whole-window encoding requires waiting for a complete window, continuously accumulating robot idle time. Treating context as a consumable resource, combined with asynchronous execution and streaming encoding co-design, is the prerequisite for deploying long history in real-time control.
For edge devices, the team adopts a global shared optimization plus device-specific tuning approach. The video expert’s linear layers use NVFP4 four-bit weight and activation quantization during generation and key-value prefilling; the action branch maintains BF16 precision—quantizing the action module yields limited acceleration benefits and risks compromising control accuracy. Pairing NVFP4 with CUDA Graph replay and compilation techniques reduces the startup overhead of quantized kernels. With multiple optimizations stacked, performance on RTX 5090, DGX Spark, and Jetson AGX Thor hardware improves by 3.2 to 4.1 times compared to native BF16. Including future video latent variable prediction, each action block takes 107.4 milliseconds, corresponding to 4-step denoising and a single-expert architecture, with full observation VAE encoding included. The trade-off is that extending context increases computational cost: when history length expands eightfold, latency on the RTX 5090 rises to 341.0 milliseconds.
03 Context Is Not Always Better When Longer
First, consider several standard benchmarks; the main experiment uses a 2.4-second context. On the LIBERO dataset, this "predict future then act" scheme achieves an average task success rate of 99.5%; on the long-task subset LIBERO-Long, it also reaches 99.5%, outperforming LingBot-VA by 1.0 percentage points and Fast-WAM by 4.3 percentage points. On RoboTwin 2.0, the average success rate is 94.4%: 94.7% under clean conditions and 94.2% under randomized conditions, placing it in the same tier as current state-of-the-art models and slightly ahead. The DOMINO dataset introduces dynamic moving objects; after fine-tuning the model on dynamic data, it achieves a 34.9% task success rate and a manipulation score of 45.1; compared to the strongest baseline Fast-WAM, success is higher by 15 percentage points, and compared to PUMA, the manipulation score is higher by 10.1.
The team conducted the most detailed tests on context variables using the GR-1 platform, where every task involves multiple object transfer operations. When context extends from 2.4 seconds to 19.2 seconds, task success rates rise from 66.3% to 78.7%, exceeding Cosmos Policy (67.1%), the strongest baseline on GR-1, by 11.6 percentage points. LIBERO-Long shows a different pattern: adding just 2.4 seconds of history boosts success from 94.5% to 99.5%; further extending context yields diminishing returns. How much improvement context brings depends on the task itself; tasks manageable with short history do not require forced window extension, whereas multi-stage sequential tasks demand longer memory.
Performance does not increase indefinitely with context length. When the context window is extended to 38.4 seconds, success rates actually drop back to 75.2% and 74.2%. This window length reaches three times the average training trajectory length of 12.1 seconds, with 80.4% of historical frames belonging to padding content. The team judges that the root cause of the performance decline is insufficient coverage of real historical samples, rather than memory capacity reaching its limit. Ablation experiments also verify: the approach of "predicting the future before generating actions" outperforms baselines that use joint denoising without predicting the future, with the gap being most prominent in long-task scenarios. This indicates that pre-estimating the evolution of interaction can help subsequent sequences of sub-actions align with each other.
04 Real-World Validation: Catching a Cup from the Conveyor Belt
In the real-robot validation phase, the team used a Unitree G1 and YAM dual-arm manipulation platform to test eight model configurations. Tasks included grabbing cups on a conveyor belt and long-horizon household operations. Dynamic tasks best exposed performance gaps between models. As the conveyor belt speed increased from 3 centimeters per second to 7.5 centimeters per second, Fast-WAM's success rate dropped from 75% to 0%, and π0.5 fell from 15% to 0%. Long-WAM remained stable at 90% to 100%, with success rates of 100%, 100%, 95%, and 90% across the four speed settings. In dynamic tasks stacking green cups into blue cups on the moving belt, Long-WAM succeeded in nineteen out of twenty tests, while both baseline groups failed every time. "The core capability to catch high-speed moving objects lies in the model's ability to predict subsequent multi-step interaction states in advance."
Long-horizon capabilities are equally valuable for long-horizon tasks. On the YAM platform, the average duration of a single run across three tasks exceeds 40 seconds; Long-WAM achieved success rates of 80%, 80%, and 85%—averaging 81.7%—for sorting blocks by color, placing dumplings into a frying pan, and stacking bowls, respectively. This expands evaluation scenarios from "rapid interception" to "sustained execution."
The model can also serve as the low-level executor for a high-level planning module. The team set up fifty tasks on RoboCasa365, categorized into seen atomic tasks, seen composite tasks, and unseen composite tasks, connecting Long-WAM with the high-level planner GPT-6 Astra in series. When running independently, Long-WAM achieved a success rate of 67.9% for seen atomic tasks, but only 15.8% and 6.1% for the two types of composite tasks; after integrating the planner, the metrics for the three task categories improved to 85.6%, 38.8%, and 35.0%, respectively, raising the overall average from 31.4% to 54.4%. With the same planner paired with π0.5, the overall score was only 30.5%, while π0.5 running independently scored 25.2%. The performance gain brought by the planner to Long-WAM is significantly higher than that to π0.5, indicating that stronger low-level execution capabilities make it easier for upper-level planning strategies to be implemented. The low-level controller is responsible for managing short-timescale physical interaction history, while hour-level goals, logical reasoning, and task decomposition are handed over to the upper-level planner; the two form a complementarity. The length of the memory window should also be determined jointly by task requirements and computing power.