DreamZero's First Author Reflects on a Year of World Action Models

On November 9, NVIDIA released GR00T 2 at the CoRL conference in Austin. In a published test suite, a bimanual robot achieved a 40% average success rate on 50 tasks with no prior demonstrations for the target robot using only its pre-trained capabilities; adding five demonstrations per task for fine-tuning increased the success rate to 95%.

This four-panel composite image captures scenes from the CoRL Conference on Robot Learning. The top-left panel is labeled "CoRL: 42HOW Robotics On Site." The other three panels depict various robot simulation and real-world operation scenarios: the top-right shows a simulation of a robotic arm equipped with counterweights performing a stacking task; the bottom-left displays a cluster of multiple quadruped robots training synchronously on a workbench; and the bottom-right presents a real-world demonstration of a humanoid dual-arm robot performing fine manipulation at an operating table, visually illustrating the practical deployment of robot learning technologies related to the World Action Model focused at the conference.

On November 9, NVIDIA unveiled GR00T 2 at the Conference on Robot Learning (CoRL) in Austin. In a set of published tests, a bimanual robot faced 50 tasks for which no demonstrations had been provided on the target robot. Relying solely on its pre-trained capabilities, it achieved an average success rate of 40%. After supplementing each task with five demonstrations and fine-tuning, the success rate improved to 95%. These tasks had previously appeared in human videos or data from other robots, and are now beginning to be transferable to a new physical embodiment.

This statistical chart presents the test results of a bimanual robot on 50 tasks without prior target demonstrations. The left bar chart shows that the average task success rate is 40% at 0-shot (no demonstrations), increases to 71% at 1-shot (one demonstration provided), and reaches 95% at 5-shot (fine-tuned with five additional demonstrations per task). The right stacked bar chart divides the task distribution by success rate intervals, with different shades of green corresponding to the 0%, 10-50%, 60-90%, and 100% success rate intervals. This visually demonstrates that as the number of provided demonstrations increases, the number of high-success-rate tasks significantly grows, corroborating the conclusion that fine-tuning with additional demonstrations can substantially improve robot manipulation success rates.

GR00T 2 continues the World Action Model (WAM) approach explored by DreamZero in February this year: the model simultaneously predicts future visual frames and robot actions, incorporating "how the world will change next" into action learning. For this release, NVIDIA further integrated human videos, data from handheld capture devices, and data from various robots into a single training pipeline. Through few-shot demonstrations and reinforcement learning, it continues to improve the precision and reliability of specific manipulation tasks.

This diagram illustrates the structure of GR00T 2 as a World Action Model, intuitively presenting the model's input and output logic. The model's inputs are categorized into three types: first, language instruction goals, with the example "Flip GROOT 1 to 2"; second, multi-view videos, covering first-person, left wrist, and right wrist perspectives; and third, time-series data of robot states. After processing by GR00T 2, the model outputs two types of content: multi-view future predicted videos and the corresponding time-series robot action sequences, clearly demonstrating the model's core capability of simultaneously predicting future frames and robot actions.

Its division of labor within the robotic system is also clearer. A high-level reasoning agent breaks down tasks, monitors progress, and handles exceptions, invoking GR00T 2 through an interface. GR00T 2 itself handles the physical manipulation requiring continuous visual feedback and fine-grained control. How many skills the model can learn, how much data is needed to adapt to new tasks, and how it recovers after failure have all become key questions this release aims to address.

This provides more practical context for the discussion around WAM. Seonghyeon Ye, the first author of DreamZero, originally focused on whether video models could endow robots with manipulation capabilities they had not been explicitly demonstrated. More than eight months later, models developed along this trajectory are beginning to tackle more complex tasks, and researchers must now further determine which parts of the future are most useful to predict, which computations can be omitted, and which experiences must still be recovered from real-world manipulation.

These questions also feature on the agenda for this year's CoRL. A workshop titled "Do Robots Need World Models?", scheduled for November 12, will discuss the boundary between explicit prediction and end-to-end policies: whether future changes need to be predicted separately or can be internalized into the policy through training, and whether existing evaluations can distinguish the value of the two in terms of generalization, efficiency, and failure recovery. Another workshop, Scaling H2R, co-organized by Ye, asks how human data can be scaled effectively—whether the benefits come from more hours and more diverse tasks, or from manipulation experiences that better match the target robot.

This is a presentation slide for a debate topic, with the debate theme "Do Robots Need World Models?" prominently displayed at the top. The page is divided into three sections based on viewpoints: the left "YES" section lists two participants supporting the view, Max Zhaoshuo Li from NVIDIA and Georgia Chalvatzaki from TU Darmstadt; the middle "NO" section lists two participants opposing the view, Chelsea Finn from Stanford Physical Intelligence and Vincent Sitzmann from MIT Rhoda AI; the right "Moderator" section lists the host of the debate, who is from Columbia ScenicX/W

Ye is situated at the intersection of both sets of questions. He is the first author and one of the project leads for DreamZero, as well as a co-author of the recent DreamFree.

DreamFree attempts to distill future knowledge into model representations, eliminating the need for future video generation during execution. He believes in the importance of world modeling and is researching whether this capability can be preserved in another form.

In an interview with 42HOW Robotics, Ye first clarified that DreamZero was his internship project at NVIDIA. After the internship ended, he left the company, did not participate in GR00T 2, and is not privy to the company's high-level planning. He acknowledges the progress in this release and is particularly focused on how to retain generalization capabilities in fine-grained manipulation. However, regarding how to continue developing WAM, he does not believe a definitive answer has been found yet.

The ideal foundation for manipulation is not necessarily today's video generation models, which prioritize visual fidelity. While scaling human data is worthwhile, contact and force information still requires further research. And once a model can complete a new task, the next challenge is finding a way to make it do so reliably. "On the robotics side, the learning recipe, optimal architecture, and data collection have not yet converged," he said.

This is a formal portrait photo with Seonghyeon Ye as the main subject. He is an AI researcher at KAIST and the interviewee for this conversation. He has fluffy, short black curly hair, wears a solid black collared shirt, faces the camera with a gentle smile, and looks calmly straight ahead. The entire photo features a clean, light gray background, making the image simple and neat with clear, soft subject outlines. This portrait intuitively presents the image of this researcher who is deeply engaged in general-purpose robot policies and World Action Model-related fields, and is also the first author of DreamZero.

Seonghyeon Ye is an artificial intelligence researcher at KAIST. He previously interned at the NVIDIA GEAR Lab, where he researched general robot policies, world-action models, and the transfer of sensorized human data to robotic manipulation. His notable works include LAPA, a video latent action pretraining method (co-first author; awarded Best Paper at the CoRL 2024 LangRob Workshop and published at ICLR 2025); DreamGen (CoRL 2025, co-first author); DreamZero (CoRL 2026, first author and co-lead of the project); and DreamFree (co-author).

How many untaught actions can video models unlock

Explaining his shift toward world models, Ye pointed to chimpanzees: they do not understand human language, yet they can manipulate objects and anticipate what will happen next. This led him to question whether starting with a vision-language model and then training it to output actions is truly the most effective learning paradigm for robotic manipulation.

Many vision-language-action (VLA) models inherit the ability to identify objects, understand instructions, and grasp semantic relationships from vision-language models. This knowledge helps a robot determine which object to manipulate, but executing a previously untrained action still requires experience with motion and object state changes. Ye wants the model to learn these physical changes alongside control during training.

This idea predates DreamZero. In LAPA, a project he contributed to, the model learns latent actions from frame-to-frame changes in video and then bridges to actual control using robot data with action labels. For Ye, the significance of this approach is that the model can learn from unannotated videos—those lacking robot action labels—by leveraging the objective of predicting world changes. He became convinced then that future prediction should play a more central role in robot policies.

Video generation models offer a more direct starting point. Trained on large-scale video datasets, these models already learn how objects and scenes change, accumulating a degree of transferable experience. Rather than building this capability for robots from scratch, the team began exploring the use of existing pretrained models.

Initially, this capability was used to generate training data. In 2025, DreamGen adapted video models to the robotics domain to generate novel behaviors and scenes, subsequently extracting action information from the generated videos to augment policy training data. Ye recalls that observing the generalization capabilities of these video models at this stage prompted their next step: since the model could already generate the process of a robot executing a new task, could it be directly converted into a policy?

Released on February 17, DreamZero takes a step forward in addressing this question. Building on a 14-billion-parameter video model, the team integrated video generation and action learning into a single architecture. During training, the model is tasked with predicting both scene dynamics and the corresponding control actions. Future prediction itself is now directly involved in policy learning and execution.

This is a diagram demonstrating the technical architecture and capabilities of DreamZero, divided into four functional blocks. The top block displays two types of pre-training data for robot manipulation: bimanual mobile manipulation and single-arm manipulation. It highlights that the data features "diversity and non-repetition," pointing to the pre-training attributes of task diversity and low repetitiveness. The middle block highlights the core World Action Model with a cloud shape, noting that it integrates video, language, and proprioceptive information to output future video predictions and continuous actions. This intuitively presents the model's core positioning of integrating video generation and action learning, while both sides display the model's generalization capability after fine-tuning and its cross-embodiment transfer characteristics. The bottom block demonstrates the model's core practical capabilities: zero-shot generalization for unknown tasks in unknown environments, and few-shot adaptation for new embodiments.

If given the chance to start DreamZero over, Ye would make the exact same choice. "Because I think it was the fastest path to the goal." Having gone through extensive iterations from model training to real-robot evaluation, he does not believe the team should have taken a different route.

A practical challenge this research addresses is how to collect robot data. Given limited collection time, repeating the same task numerous times versus exposing the robot to as many different objects, environments, and manipulations as possible yields two vastly different types of data. The former makes it easier to refine a specific action, while the latter provides an opportunity to cover variations beyond the training tasks.

Ye notes that the DreamZero team put considerable thought into increasing diversity during fine-tuning. They aimed to preserve the existing capabilities of the video foundation model while exposing it to a wider range of variations. "While repetitive data might be effective for accuracy, it comes at the cost of some generalization." In his experience, pursuing the ability to handle novel tasks requires making strategic choices right from the data collection stage; you cannot wait until the model is fully trained and then expect it to make up for missing experience on its own.

A set of tests on AgiBot demonstrates the benefits the team aimed to achieve. Across ten unseen tasks, including actions like untying shoelaces and shaking hands, DreamZero achieved an average task progress of 39.5%, compared to 16.3% for the baseline pre-trained VLA. The paper also notes that when faced with varying instructions, some VLAs still tend to reach for objects and attempt to grasp them, falling back on common actions seen during training. What is being compared here is the extent of task completion, not the overall success rate of the complete task.

These results also capture DreamZero's current standing: it demonstrates a stronger ability to attempt novel actions, while still leaving a large number of tasks incomplete. For a model aiming for generalization, the ability to transfer novel actions from past experience is a worthwhile starting point for continued investment; the next step is to pinpoint exactly where the failures occur.

We asked Ye: if the model predicts that a cup can be picked up but the corresponding action fails to execute it, should the issue be attributed to future prediction, action generation, or the connection between the two?

His observations somewhat defied the premise of the question. The team conducted extensive rollouts—having the model physically execute tasks on the robot—and found that the alignment between video and action was remarkably strong, with jointly learning this correspondence proving less difficult than anticipated. However, when video prediction failed, the action typically failed as well; they almost never observed cases where the action succeeded but the video prediction failed.

"Once video prediction fails, the action fails as well." Ye concluded from this that the primary bottleneck in DreamZero is future prediction, and that improving the foundation model's ability to predict physical changes should further refine the policy. This experimental observation from the team's work on the system also explains why he remains bullish on world modeling: the model's understanding of the future is now inextricably linked to actual execution outcomes.

Learning the future doesn't require generating every frame

On March 20 this year, Ye introduced DreamZero on the RoboPapers podcast. That discussion centered on a novel question: how can a 14-billion-parameter video model achieve generalization for real-world robots? In this interview, we revisited that conversation to ask which of his assessments remain unchanged months later, and which have evolved.

Ye admitted he doesn't entirely recall the context of his answers at the time, but one point remains clear: "I still believe world modeling is crucial. However, it doesn't necessarily have to mean raw pixel generation."

DreamZero leverages the pre-trained capabilities of off-the-shelf video models while taking on the computational burden of video generation. It must process objects, positions, motions, and extensive visual details in future frames. For video products, photorealism, naturalism, and aesthetic appeal are all goals; for robot control, the paramount requirement is accurately understanding physical changes. While these objectives overlap, they do not necessarily demand the same training targets or computational allocation.

Synchronizing video generation with robot actions is a formidable task in itself. The DreamZero paper notes that through system, implementation, and model-level optimizations, the team reduced inference latency on a GB200 setup from approximately 5.7 seconds to roughly 150 milliseconds, enabling the large model to execute closed-loop control at approximately 7 Hz. The iterative process from training to evaluation that Ye described also involved integrating a video model directly into the robot's control loop.

"Manipulation doesn't require highly polished or photorealistic videos; what we actually need is an accurate understanding of the world, such as physics," Ye noted. He believes that in the long run, the optimal foundation model should be purpose-built for robotic manipulation, rather than starting with a video generation model optimized for visual fidelity and merely fine-tuning it for robotics.

This is not to dismiss his earlier choices. Off-the-shelf video models allowed the team to quickly acquire usable priors, and DreamZero demonstrated that these priors can be translated into actionable capabilities. The more pressing question now is how this knowledge should be retained within the model to most effectively serve control.

Fast-WAM, released in March, already broke this problem down in a series of experiments. It retained joint video-and-action learning during training but skipped future video generation at inference. The researchers found that while the model remained competitive when video generation was omitted at inference, removing video learning during training caused a much more significant performance drop. This makes a compelling case for a specific possibility: the primary benefits of future prediction may already be encoded into the model's representations during training.

DreamFree, a project Ye contributed to, pushes this focus on representations further. In this context, representations refer to the internal information the model extracts from observations to inform subsequent decision-making and control. During training, a frozen video teacher model has access to both current and future frames, while a separate encoder learns to extract representations containing future information that closely align with the teacher's, using only current observations. The action module then learns control policies from these representations.

This training approach leverages future information, but deployment no longer requires the model to generate the future frame by frame. In the configuration reported by the project, DreamFree runs a 14-billion-parameter video foundation model on a single H200 GPU, achieving an inference speed of approximately 11 Hz. The goal is to retain the knowledge gained from future prediction while eliminating the time cost of generating video before every action.

This diagram consists of two sub-module processes, (a) and (b), corresponding to the technical implementation logic of DreamFree introduced in the text. Sub-figure (a) is the future representation distillation process. The training mask on the left corresponds to multi-frame inputs from f₀ to fₕ. After the video DiT encoder extracts features, knowledge is distilled into the VDM, which serves as the teacher model, through a projection alignment operation, thereby embedding knowledge related to predicting the future into the model. Sub-figure (b) is the Action DiT training process, which requires only the initial frame f₀ as input. After processing through a readout head, feature fusion is completed, and the data is finally fed into the Action DiT module for training. The entire architecture aligns with the design concept mentioned in the text of retaining only future prediction knowledge without the need for frame-by-frame generation.

Ye uses the term "representation alignment" to summarize the connection between DreamZero and DreamFree. DreamZero applies explicit supervision to the next state, training the model to learn what the world should look like next; DreamFree, through distillation, compresses this relevant knowledge into the model's internal representations. While the two projects employ different methods, both require the internal information used for manipulation to be linked to future state changes.

"People will debate whether the model should generate future states at test time. But I think what matters more is that somewhere during training, the representations must be aligned so they can model the future," he said.

Based on this perspective, simply checking whether a new architecture outputs video is no longer sufficient to evaluate its capabilities. A model that directly outputs actions may have already learned future dynamics during training, while a model capable of generating full videos still needs to prove that those computations actually improve manipulation. The research focus therefore shifts to finer details: which future information should be retained, how it should be trained, and whether the action module can utilize it effectively.

The latest GR00T 2 illustrates a different trade-off. To reduce computational load, the Turbo version shrinks the spatial grid of the predicted video to one-quarter of its original size. However, across a suite of post-trained dexterous manipulation tasks, the success rate dropped from 68.6% in the full version to 34.3%. The technical report does not attribute this drop to a single factor, but instead raises a point worth further investigation: the amount of visual detail that must be preserved likely depends on the specific task and the robot's physical embodiment. Determining which future information can be compressed cannot rely on inference speed alone.

This also blurs the boundary between WAM and VLA. The DreamFree project page describes its approach as "WAM-style training, VLA-style inference." Training objectives, internal representations, and deployment formats can be combined in various ways; the model's name alone cannot capture these distinctions.

Ye remains convinced that WAM offers superior generalization and faster adaptation, but he does not treat DreamZero's specific architecture as the final word. In fact, having built this system, he has begun to ask a more precise question: what kind of future should the optimal foundation model for manipulation learn to predict?

After a million hours, still missing a layer of information

Human data is another rapidly expanding answer.

When asked what else DreamZero should clearly focus on next, Ye quickly pointed to UMI and human data. UMI, or the Universal Manipulation Interface, involves a human operator holding a gripper to complete tasks and recording the manipulation for the robot to learn from. Human data, meanwhile, allows researchers to capture a broader range of how people work in real-world environments, reducing the reliance on the robot itself during data collection.

He recalls that DreamZero was primarily trained on robot data collected via teleoperation. The paper also included transfer validation using a small amount of human video, but scaling up the integration of other data sources into training remains a direction worth pursuing. In his view, GR00T 2 has already moved forward along this path.

Public data shows that this shift in scale is quite significant. The pre-training for GR00T 2 mixed approximately 403,000 hours of first-person human video, about 10,000 hours of handheld UMI demonstrations, and 23,500 hours of robot teleoperation data. Human video now accounts for the largest share. NVIDIA reports that in tests across 72 downstream tasks, progressively increasing the amount of human video improved real-robot task completion from 50% to 70%.

This chart illustrates the changes in task performance in real-robot and video simulation scenarios after scaling up the proportion of human videos in pre-training. The horizontal axis represents the proportion of human videos in pre-training (logarithmic scale), gradually increasing from 0 to 100%. In the real-robot scenario, the success rate increases from 18% to 41%, and task progress improves from 50% to 70%. In the video simulation scenario, the success rate increases from 30% to 51%, and task progress improves from 64% to 77%. All four metrics show a significant upward trend as the proportion of human videos expands, presented as upward-trending line charts, all with error bars.

Released in August, Dyna-2 scaled its human video pre-training from 1,000 hours to 1 million hours in stages, followed by real-robot validation using a consistent post-training pipeline. Its report shows that the average normalized score across 14 tasks improved from 20% to 53%. The tests included language following as well as tasks such as tying knots, turning keys, and manipulating bottle caps; the benefits are already evident in real-world execution.

The data mix has gone beyond simple blending. The GR00T 2 technical report outlines three training stages: first, using human and robot videos to learn interaction dynamics; then, jointly learning from multi-view videos and actions; and finally, increasing the weight of robot data to further adapt to real-world control. Human experience is responsible for expanding coverage, while robot demonstrations help the model learn to execute on a specific physical body. However, the report also acknowledges that there is no clear input-output relationship between the different data sources: it remains unclear exactly how much capability is gained by adding one more hour of human, UMI, or teleoperation data, respectively.

These results provide a rationale for continued investment in human data. Ye also acknowledges that scaling up first-person human data can improve transfer to robots. However, he remains cautious about how deeply this benefit extends to actual manipulation capabilities.

"This is just my personal view," he was careful to qualify. In his assessment, the benefits of current data primarily manifest in language following and high-level visual and semantic understanding; when it comes to actual manipulation-level motion and generalization to new tasks, many problems remain unresolved.

His reasoning comes down to the captured signals: much of the first-person human data primarily records vision, supplemented by pose estimation, without directly capturing contact or force. While video preserves the visible changes during a manipulation task, the critical information required when a robot physically interacts with an object may not be fully retained.

GR00T 2 has not entirely solved this problem either. Its DexUMI data collection device can already record normal and shear forces at the fingertips, but the model does not yet use tactile feedback as an input. The technical report lists how to integrate force and tactile data, how to preserve existing visual capabilities, and what control frequency to adopt as open questions. Capturing a signal and enabling the model to use it effectively are two distinct steps that must be solved separately.

This diagram intuitively presents the step-by-step execution process of a robot completing three sequential manipulation tasks, with each task broken down into six clear sub-steps marked with numerical sequence numbers. The first row is the "Make Banana Yogurt" task, sequentially completing picking up a banana, getting a butter knife, peeling the banana, removing the yogurt seal, pouring yogurt into a bowl, and moving the bowl to the center. The second row is the "Scoop Granola" task, sequentially completing grabbing the granola jar, opening the jar lid, getting a spoon, scooping granola into the bowl, switching hands to hand over the bowl, and placing the bowl on the table. The third row is the "Strike a Match" task, sequentially completing taking the matchbox from the shelf, grabbing the matchbox, striking the match on the side of the box, holding the match, lighting the lamp stand, and extinguishing the used match in water. Each step synchronously displays a comparative view of human demonstration and robot replication, intuitively reflecting the robot's reproduction of manipulation details.

"This is a missing layer that is worth investigating," Ye said. While tactile sensors are not yet stable enough, he believes both tactile and force data are crucial. Simply continuing to collect more visual data in the current manner will not automatically turn this missing layer of information into a direct measurement signal, no matter how many additional hours are logged.

This diagram consists of three sub-figures, (a), (b), and (c), all of which are 3D schematic diagrams of mechanical structures with green as the main color tone, intuitively presenting different local details of mechanical hand components related to tactile and force sensing. Sub-figure (a) shows the cross-sectional structure of a certain sensing module, presenting the assembly relationship of internal parts. Sub-figure (b) shows the overall structure of a complete anthropomorphic mechanical palm, presenting the layout of multiple humanoid fingers and the linked wrist. Sub-figure (c) shows the local installation details of a certain sensing component on the mechanical hand. This diagram echoes the hardware design content of tactile and force-related sensors mentioned in the text, supporting the relevant discussion on supplementing contact and force modality information.

Existing public results have already shown that the benefits of human video can translate to real-world manipulation, though Ye’s assessment of its capability boundaries still requires further validation. The contact and force data he proposes addresses another question that these scaling experiments have yet to answer: for more fine-grained manipulations involving complex physical interactions, can introducing new modalities yield benefits that simply adding more video data cannot easily achieve?

This also dictates where the next data investment should be directed. Increasing data hours, expanding task variety, and adding sensor modalities solve different problems. The former expands the coverage of existing experience, while new modalities may expose the model to previously unrecorded information for the first time. Determining which approach is more effective requires empirical comparison on target tasks using real robots.

This reflects Ye’s perspective on the value of academic research. He believes that before scaling up investment, it is essential to first conduct small-scale proof-of-concept studies to determine which information can be effectively transferred. Unlike language models, robotics has not yet settled on a relatively stable architecture, training methodology, or data scaling paradigm, meaning researchers still have the opportunity to redirect future investments.

We followed up by asking which one or two sensing modalities he would add if he could only choose that many. Instead of picking a single answer, he first distinguished what specific information needed to be extracted from tactile feedback.

For a tactile glove, he considers resistive sensors one possibility; if the focus is more on normal and shear forces, magnetic tactile sensors or other solutions might be better. Others are exploring wrist sensors, IMUs, and human wearables to estimate force-related information. For him, the sensor category is just the starting point—what ultimately matters is whether the captured signals are actually useful.

His evaluation criteria come down to just two factors: whether the data can be transferred directly and effectively to a robot, and whether the data collection can be scaled. Simply strapping sensors onto a human and recording continuously for long durations does not automatically satisfy the first criterion, nor does capturing high-quality signals in a lab automatically satisfy the second.

UMI has already provided an excellent demonstration of this transfer process. A human operator holds a gripper to perform a task, and the robot executes it using the same gripper, creating a relatively direct correspondence between data collection and control. However, this interface simultaneously restricts human action: the collector must grip the device and use the gripper to complete the task, which means the full freedom of natural human hand manipulation cannot be preserved.

Ye believes that if data collection scalability is pushed to its absolute limit—allowing people to move naturally using their own hands—that is likely the ultimate state we hope to reach. But once the gripper is removed, figuring out how to extract useful manipulation signals becomes a new challenge. The closer the data collection gets to everyday life, the more we need to reconsider exactly what interface should be used to transfer human experience to the robot.

Data sources are expanding, ranging from DreamZero to millions of hours of human video. What Ye wants to investigate further is which manipulation information is preserved in this process and what is still missing. This will influence both sensor selection and the learning objectives the model should target.

Beyond Generalization: Where Do Precision and Reliability Come From

Looking ahead to the next phase of WAM development, Ye outlined a key requirement when discussing GR00T 2: maintaining generalization during more fine-grained manipulation. Both of these investments must ultimately pass this test—can a better foundation and more data give the robot a higher starting point for new tasks, while ensuring precise movements and stable outcomes?

The GR00T 2 report breaks down these benefits more specifically: pre-training provides broad capabilities, corrective demonstrations and reinforcement learning address shortcomings in precision and physical contact, and upper-level agents supply memory, verification, and recovery for long-horizon tasks. Attributing the final success rate entirely to the world action model architecture would overlook the distinct contributions of each of these components.

Ye emphasizes data diversity to ensure the model does not lose the broad capabilities gained from pre-training when adapting to specific robots and tasks. While repetitive training can improve precision, whether the model maintains the same level of adaptability when transferred to new tasks and environments requires separate validation. For him, true progress means both fine manipulation and generalization must hold.

To achieve this, the DreamZero team also modified how the model utilizes historical information. While pre-trained video models typically employ a bidirectional architecture, they switched to an autoregressive structure. This allows the robot to use its previous execution history as context for subsequent predictions, which Ye refers to as "visual memory".

This enables predictions to build upon the execution that has already taken place. The environment a robot must navigate changes continuously in response to its own actions; therefore, what it has previously observed and executed should serve as the basis for its next decisions. Ye groups this capability with a strong foundation model and diverse data as critical factors for improving real-world execution.

However, even after retaining the process in memory, the model may still yield varying action outcomes. Ye recalls a specific observation from the team: by changing the random seed and having the robot execute the same task multiple times, they achieved success in about one or two out of eight attempts. A single failed attempt does not mean the model is entirely incapable of finding a viable solution for the task.

This observation sparked his interest in scaling compute at test time. If the model can propose multiple candidates and a sufficiently capable critic model can evaluate which is better prior to execution, it could reliably select the successes that otherwise occur only occasionally. He believes this approach has the potential to evolve into a form of test-time planning for the WAM.

However, the team has not pursued this direction in depth. The observation of one or two successes out of eight executions is merely an experimental phenomenon he recalled during the interview and cannot be directly equated with a validated planning algorithm. To translate this potential into consistent gains, the challenge of how to accurately evaluate candidate quality must still be resolved.

A specific example has already emerged from the long-horizon task experiments in GR00T 2. During 300 consecutive assemblies of three types of toys, the system succeeded 292 times. Of the eight failures, three involved screws that were not fully tightened, yet both the action model and the upper-level agent judged the task as completed. A physical state that appears close to success can still bypass the verification checks of the entire system.

This diagram presents a step-by-step demonstration of two practical tasks executed by a robot. The top is the "Hot Wheels Assembly" task, which is marked as having a total of 24 operation steps. The diagram shows four representative steps: 1 installing the wheels, 2 tightening with a power drill, 3 placing the toy car's engine hood, and 4 tightening the engine hood bolts. The bottom is the "Make Shipping Label" task, which also shows four representative steps: 1 handing over the scissors, 2 cutting the label, 3 pasting the cut label onto the package, and 4 scanning the barcode on the package. This diagram intuitively presents the typical multi-step operation process in long-horizon tasks, echoing the relevant content of the GR00T 2 long-horizon assembly experiment mentioned earlier in the text.

This brings us back to future prediction. A World Action Model (WAM) can generate a predicted sequence of events, giving researchers the opportunity to evaluate outcomes before execution. Yet, the utility of this evaluation depends on how closely the prediction aligns with the real world. A task completed in a predicted video does not automatically guarantee the robot will complete it physically. Predictive capability, candidate selection, and real-world execution must be validated together.

Therefore, bypassing future video generation and utilizing the future for planning can address different requirements. For operations requiring rapid responses, DreamFree attempts to keep future knowledge within the representations to reduce online computation. For decisions worth spending more time comparing, Ye envisions multi-candidate selection, which might use additional computation to improve performance. The specific approach depends on how much latency the task can tolerate, the cost of failure, and whether the model can reliably evaluate the outcomes.

This is also why he did not provide a unified answer when discussing applications. Factories prioritize precision and reliability, while homes require both safety and reliability, along with the ability to handle a much wider variety of tasks. Ye believes that world action models could help improve safety, but his team has not yet deeply explored this direction, so he does not consider it a proven capability of DreamZero.

He is personally very interested in domestic robots. Household chores require an extremely broad range of knowledge, and everyday objects and tasks are constantly changing, making it a difficult yet attractive problem. When it comes to timelines, his intuition is that in two to three years, the technology may reach a level where it can perform more daily tasks; however, it will take much longer for consumers to use it reliably in their homes over the long term.

Safety, maintenance, and deployment all require this extra time. The fact that a model can already perform a task is not enough to determine when a product can be shipped. Users are receiving a system that must work continuously; problems that researchers can address individually in the lab will all be brought into the home.

Ye has not changed his views on world modeling despite these challenges. What has changed is his description of the next research priorities: off-the-shelf video models provide an effective starting point, but in the long term, a foundation model better suited for manipulation still needs to be developed. Human videos offer a much larger data source, and the next step is to determine which physical information is worth recording and how to capture it.

On his personal homepage, in addition to general robot policies and world action models, his recent research interests now include sensor-augmented human data, and how it can help robots perform contact-rich manipulation and compliant control. After developing DreamZero, his focus has also expanded to the stage before data enters the model.

As human data begins to be measured in millions of hours, Ye wants to first determine which unrecorded signals are worth collecting at scale. His advice remains highly specific: "Before scaling up massively, small-scale proof-of-concept studies are even more critical."