Robots Spend 97% of Their Time Waiting for Foundation Models: Galaxea's Hang Zhao on Solving Real-Time Constraints at CoRL 2026

At the CoRL 2026 Conference on Robot Learning in Austin, Texas, on November 9, Hang Zhao, Assistant Professor at the Institute for Interdisciplinary Information Sciences at Tsinghua University and Co-founder and Chief Scientist at Galaxea, delivered the first early-career keynote titled "Organizing Physical AI Across Timescales".

November 9. The CoRL 2026 Conference on Robot Learning was held in Austin, Texas. From 9:30 to 10:00 a.m. local time, the conference hosted its first early-career keynote address.

Hang Zhao, Assistant Professor at the Institute for Interdisciplinary Information Sciences, Tsinghua University, and Co-founder and Chief Scientist at Galaxea, delivered a talk titled "Organizing Physical AI Across Timescales". He began by discussing the temporal constraints involved when models control robots, proposing a hierarchical layering of robotic intelligence based on execution clocks.

Under specific action interfaces and prompt configurations, GPT-6 Astra can already directly output robot actions in simulation tests without robot-specific fine-tuning.

But Hang Zhao pressed further: Why is it that while the model has the capability to output actions, real robots still struggle to utilize it effectively?

He said the answer lies in timing. When generating text responses, users can wait for the model to finish thinking; but when controlling a robot, the environment will not stop and wait for it—the robot must react in real time.

Therefore, a robotic system must not only know what to do but also keep pace with the real-world environment. Starting from the waiting problem observed in simulation tests, Hang Zhao further discussed why robots need to layer their intelligence according to execution speed, and what information should be passed between these different layers.

Models can output actions, but robots cannot afford to wait

Before discussing the timing issue, Hang Zhao first presented control tests of GPT-6 Astra on two simulation benchmarks: under specified action interfaces and prompt configurations, the model can output control actions without requiring task-specific fine-tuning for robotics. This, however, does not mean that physical robots can already perform tasks stably and in real time.

On the NVIDIA RoboLab-120 benchmark, using GPT-6 Astra as a "direct policy"—where the model directly outputs robot actions during decision-making—yields a 66% success rate. When used as an agentic policy capable of calling tools, the success rate reaches 75%. In contrast, the specifically trained robot foundation model π0.5 achieves a success rate of only 28%.

On another simulation benchmark, RoboDojo, paper metrics show that GPT-6 Astra achieves an average success rate of approximately 22.5%, while π0.5 reaches about 6.9%—making the former roughly 3.3 times higher than the latter.

"This is truly unexpected; I believe many in our community are not yet prepared to face this reality," Hang Zhao said.

However, he immediately pointed out: "These benchmarks do not account for time." In NVIDIA's test, the simulation pauses while the model thinks. The virtual environment waits for as long as the model takes to process. Hang Zhao displayed an episode timeline showing that GPT-6 Astra, running under a direct policy, takes nearly four minutes to complete a single episode.

In the chart, gray represents the model's inference time, while orange represents the robot arm's movement. The arm's actual movement totals only 5.2 seconds; according to the presentation slides, each model invocation takes approximately 18.6 seconds, meaning the robot spends 97% of its time waiting. This means that even if the model can output control actions, inference latency may prevent it from keeping pace with a real robot's execution speed.

Foundation models are highly capable, but their latency is a major bottleneck—so how exactly should they be deployed?

Hang Zhao outlined an approach using a set of controlled experiments on RoboCasa365. The experiments featured five unseen kitchen scenes and five tasks, each with a 50-second time limit: turning on the microwave, moving food from a plate to a pot, navigating to a kettle or sink, sorting bottles by category, and placing desserts on a tray.

The experiments used the same frontier foundation model, with the only variable being the frequency of model invocations during execution. Hang Zhao compared five approaches, ranging from generating the entire program at once before the task begins to continuously querying the model during runtime:

• Coding policy: The large model understands the task and writes a piece of code, after which the program runs autonomously without further model queries.
• Coding + Ask policy: The model is asked to rewrite the code only if execution fails.
• Orchestrated policy: The large model is responsible for invoking different skills, which are themselves small neural networks; the presentation listed approximately 4 calls during execution.
• Direct policy: At each policy decision step, the model directly specifies where the hand should go; the slides indicated an action interval of 2 seconds.
• Agentic policy: At each step, in addition to outputting actions, the model can also invoke tools and perform computations.

During his presentation, Hang Zhao showed the number of model calls during task execution for each policy: 0 for the coding policy, approximately 1 for the coding + ask policy, approximately 4 for the orchestrated policy, and approximately 18 for both the direct and agentic policies. The 0 count means the model is not queried during the task execution phase; however, the model still generates the program once before the task starts.

The difference between these policies lies in how many decisions are delegated to the model during execution, and how many times the robot must wait for the model's output.

The experiment was conducted in two rounds: in the first round, the clock was paused, temporarily excluding the time spent on model inference.

The results were intuitive: the coding policy, which queries the model only once, had the lowest success rate at 45%; policies that queried the model performed better, with the orchestrated policy achieving the highest at 75%; the direct and agentic policies, which move the robotic arm step by step, both achieved 60%.

In the second round, the clock was allowed to run, and exceeding the time limit counted as a failure.

The results showed a clear reversal. The direct policy dropped from 60% to 15%, the agentic policy dropped from 60% to 20%, and the coding + ask policy dropped from 65% to 45%; the orchestrated policy dropped from 75% to 65%, but remained the highest. The coding policy, which queries the model only once, was unaffected, remaining at 45%.

When inference time was factored in, the success rates of the direct and agentic policies, which frequently call the large model, dropped significantly. The coding policy, which calls the model only once at the beginning, achieved 45%; the orchestrated policy still achieved 65%, making it the best-performing approach in this comparison: the large model makes a decision at intervals, while high-frequency actions are handled by small neural network skills.

Hang Zhao argues that small models can also be highly effective, provided they are integrated with large models.

Robot intelligence should be layered by clock speed

If large and small models are to be combined, the next question is where to draw the boundaries between these layers.

Hang Zhao notes that the traditional approach divides functions by role: perception in one layer, planning in another, and control in a third. Instead, he advocates partitioning by operational clock speed: "The clock dictates the hierarchy. Draw the boundary where the clock speed changes."

Hang Zhao categorizes robot intelligence into three tiers: System 2 operates on the scale of seconds, represented by frontier models like GPT-6 Astra; System 1 runs at tens of hertz, represented by vision-language-action (VLA) models or world-action models (WAM); and System 0 operates at hundreds of hertz, represented by whole-body motion trackers.

He specifically clarified that his use of System 0, System 1, and System 2 refers to time budgets, entirely distinct from the "fast and slow thinking" concepts in psychology. To determine which layer a given function belongs to, one should first ask what it needs to react to. The target of the reaction dictates the required execution speed, which in turn determines its layer.

System 2 reacts to changes in task state, such as completing a step and initiating the next, which occur on a timescale of seconds to minutes. System 1 responds to the relative motion between objects, such as the changing distance between a robotic hand and a cup. System 0 tracks physical contact and forces, requiring it to run hundreds of times per second.

Under this framework, a single function can be distributed across multiple layers: Perception is no longer confined to a single tier; instead, each layer has its own perception and control loops, operating at different frequencies.

Hang Zhao's research background has repeatedly brought him face-to-face with the same problem. He earned his Ph.D. from the Massachusetts Institute of Technology, where he was advised by Antonio Torralba, and later worked as a research scientist at Waymo before now leading the MARS Lab at Tsinghua University. In his talk, he summarized his research trajectory in three stages: first, wheeled robots; then, legged robots, "teaching quadrupeds and humanoids to run, jump, and traverse various terrains"; and now, robots with arms, "because at the end of the day, we want robots to do real work for humans."

Discussing why he shifted from autonomous driving to humanoid robots, Zhao recalled seeing Waymo vehicles and Cybercabs on the streets of Austin. Autonomous driving has already entered daily life and moved toward mass production—can humanoid robots achieve the same? His answer relates to time.

The first lesson comes from autonomous driving. Zhao said the team integrated a vision-language model (VLM) into a car and found its ability to understand scenes surprisingly strong; for instance, it could recognize school zones and slow down for children crossing the street. However, it could not respond in real time to driving tasks. The team then deployed the slower VLM and the fast driving system in the same vehicle.

During the talk, Zhao cited the DriveVLM research conducted jointly by the MARS Lab and Li Auto as an example. DriveVLM-Dual deploys a slower-running VLM alongside a fast driving system in the same vehicle: the two systems work in parallel, with the slower system stepping in only when encountering long-tail scenarios.

Zhao used a 5%/95% split to illustrate the division of labor: the slow system handles the few difficult cases, while the fast driving system manages the vast majority of situations. He summarized: "Let the fast layer do the driving, and let the slow layer run at its own pace."

The second lesson comes from humanoid parkour. Zhao said the team first implemented climbing, jumping, and crawling on smaller robots before transferring these capabilities to full-size humanoids. This year, the robots have been able to vault and climb using their hands for support, interacting directly with the environment. These movements are executed by learned controllers that react autonomously within the fast layer, making the entire system function as System 0.

The takeaway from this is that "reflexes must be placed in the fast layer," and this layer must handle everything from visual perception to action entirely on its own. Combined, these two experiences point to the same principle: assign fast tasks to the fast layer, and do not let the slow layer interfere with the real-time reactions that the fast layer must execute.

Which layers are required for real-world tasks depends on the specific job.

Hang Zhao outlined four scenarios. Industrial assembly lines involve fixed, repetitive motions and require only System 0. Logistics sorting follows a fixed workflow, but every parcel is unique, demanding robust System 1 capabilities. Restaurant service requires conversing with customers, taking orders, and carrying trays, necessitating some capacity across all three systems. Household chores involve a much more complex task definition than restaurant service and require strong System 2; however, execution speed can be slower, allowing System 1 to be relatively weaker.

Once the system is layered, two main challenges emerge: how to ensure each layer runs fast enough, and what information should be passed between them.

Keeping System 1 on its own clock

Hang Zhao focused on the first challenge: how to enable each layer to operate at its own pace.

System 1 must respond to human-object interactions at frequencies of tens of hertz, but existing models are often too slow to keep up.

He identified three sources of latency: waiting for the model to generate future video, waiting for the model to complete inference, and waiting for the entire action sequence to finish decoding.

The first source of latency comes from world-action models. Built on video generation models, they first predict future frames and then output actions based on those predictions.

Hang Zhao noted that video generation itself is inherently slow: "Generating a 5-second video usually takes two or three minutes," making it impractical for direct real-world deployment.

The team then asked: during inference, does the model really need to generate future frames first? In world-action models, video prediction and action prediction share the same attention mechanism.

The team's proposed Fast-WAM retains video prediction during training, allowing the model to learn how the future unfolds, but skips future frame generation during inference.

The public release of Fast-WAM comprises a 5-billion-parameter video backbone and a 1-billion-parameter action expert, totaling approximately 6 billion parameters. Its inference latency is 190 milliseconds.

By comparison, the baseline version that generates video before outputting actions takes 810 milliseconds, making Fast-WAM more than four times faster. The research demonstrates that video prediction primarily helps the model learn world representations during training, meaning future frame generation can be safely skipped during execution.

During his talk, Hang Zhao also introduced a larger pre-trained version. This 16-billion-parameter model requires approximately 60 milliseconds per action inference when running in FP8 on an RTX 5090D. He emphasized that this is a distinct version from the public Fast-WAM, and their parameter scales and inference latencies should not be conflated.

The second source of latency stems from vision-language model inference. Vision-language-action (VLA) models, which evolved from vision-language models, inherently involve time-consuming reasoning processes. If the model must reason step-by-step while controlling the robot's body, the latency costs become prohibitive. Hang Zhao's solution is to supervise the model with chain-of-thought during training, but suppress the generation of the reasoning process during execution.

Using the Galaxea G0.5 as an example, he explained that the model integrates visual and text inputs, along with reasoning and action outputs, into a single autoregressive sequence. Researchers can disable the reasoning output on the same model checkpoint to observe how execution performance changes.

As shown in the figure, visual inputs, language, the robot's proprioceptive state, and its body identifier are all fed into the same model.

The model can first output subtasks, then provide target bounding boxes and action codes, which are finally decoded into continuous actions. Disabling reasoning saves the wait time for generating intermediate reasoning during execution, but it does not mean the model did not learn this information during training. Researchers can also toggle the reasoning settings on the same model checkpoint to compare the benefits of intermediate reasoning on execution performance against the time cost it incurs. Related research on G0.5 also explores the role of reasoning in long-horizon tasks.

The third source of delay comes from action chunking. Hang Zhao notes that nearly all papers—whether on Vision-Language-Action (VLA) models, diffusion policies, or world-action models—require generating a complete chunk of actions before execution. "You are always waiting for the entire action chunk to be generated, which costs a lot of time," he says.

He illustrates this issue with a demonstration of a robotic arm chasing a moving box. A chunking policy must wait for the entire action chunk to be decoded before it can start executing, meaning it can only re-observe the box after each action chunk finishes. During the execution of an action chunk, the policy has no timely awareness of how the box moves.

The team proposes ActionCodec2, a streaming action tokenizer where each action token represents a small segment of motion, allowing the "robot to execute it as soon as it receives it." The accompanying closed-loop policy can incorporate new observations at every control step. The streaming tokenizer makes it possible for the robot to start executing earlier; however, whether it can correct its trajectory between steps depends on the closed-loop control, not the tokenizer itself. At the end of the animation, the chunking policy loses track of the box, while the streaming policy catches up.

The common thread among these three approaches is freeing System 1 from waiting: no need to wait for future frames to be generated, for reasoning to be completed, or for the entire action chunk to be decoded.

Between layers, it is not just about passing motion

With System 1 running faster, another problem remains to be solved: what information should be passed between different layers?

Hang Zhao categorizes the information flowing between the three-layer system into two types. Passed downward is intent: System 2 can pass language, goals, or programs to System 1, which then passes motion commands to System 0. Passed upward is feedback: System 0 informs System 1 about contact status, and System 1 reports task progress and execution confidence to System 2.

Specifically, System 0 can feed back contact, force, and tracking states upward, while System 1 can report progress and confidence, or even explicitly state, "I'm not sure."

With this information, the upper layers can determine how far the lower layers have progressed in execution and whether replanning is necessary, rather than simply issuing a command and assuming the task is complete.

Hang Zhao notes that current research focuses predominantly on passing intentions downward, while cross-layer upward feedback remains weak.

For the downward interfaces, multiple modalities already exist between System 2 and System 1. Hang Zhao gives the example of a motion generator that can accept text, music, or even human pose descriptions to understand the intended actions and emotions. System 1 then translates this into executable trajectories for System 0 to track.

A more significant problem lies between System 1 and System 0, where the information passed is often limited to motion commands.

But is motion information sufficient? Hang Zhao illustrates its limitations using two everyday actions: when wiping a table, the key is applying force, even if the hand's displacement is minimal; when sitting down, the motion trajectory might be similar across different chairs, but the chair must still support a person's weight. Transmitting only motion trajectories fails to convey the actual forces required for these tasks.

Hang Zhao says researchers have begun moving beyond pure motion commands—progressing from joint targets to whole-body motion and finally to contact—to incorporate both motion and force. Some approaches preserve the contact points and geometric relationships between the robot and objects when transferring motions to a new body. However, this only preserves contact geometry and does not yet describe the applied forces.

He then introduced Ctrl-F from the University of California, Berkeley. In this system, a tracker receives a reference motion and a force plan, while a higher-level, lightweight planner determines the applied forces. According to the authors' project page, the planner reads the object state and the robot's proprioception, and the physical robot tracks the object pose using AprilTags.

In the box-pushing comparison demonstrated by Hang Zhao, an additional 20 kg load was placed inside the box. The SONIC baseline from NVIDIA's GEAR-SONIC, which only tracks motion, failed to push the box in this trial, whereas Ctrl-F successfully moved it. This example illustrates that beyond motion trajectories, the system must also convey commands on how to apply force.

Why is upward feedback so important? Hang Zhao used another failure demonstration to show that systems often break down at handoffs: the low-level controller continues executing without realizing the task has already failed, while the high-level planner fails to detect the issue in time and adjust its strategy.

In another comparison using Ctrl-F, a 2 kg load was added to the box. While the SONIC baseline tracked the box-lifting motion, the gripper slipped off the box, leaving it unlifted. Executing motion commands does not equate to task success; if the low level does not report the failure upward, the high level cannot adjust its strategy in time. This demonstration highlights that cross-layer feedback requires deliberate design.

Looking to the future, Hang Zhao outlined two possible approaches.

The first is to continue using a hierarchical system, allowing each layer to run at its own pace, with intent passed downward and feedback passed upward. A slower-running upper layer is ill-suited for issuing precise contact force commands; the challenge with this approach lies in ensuring the lower layer can accurately perceive the situation and proactively report back upward.

The second is to adopt a unified model with a shared architecture and context, while configuring dedicated experts for different operating clocks to run at varying frequencies. A shared context is the primary advantage of this approach, but the challenge lies in training System 0, System 1, and System 2 jointly. Currently, System 1 and System 2 are mostly trained via behavior cloning, while System 0 is primarily trained in simulation; integrating these distinct training mechanisms into a single unified environment remains an open problem.

"A chatbot can wait for the model to finish thinking, but a robot cannot," Hang Zhao concluded. He also proposed that simulation evaluations should report task completion success rates within a specified time limit, factoring in inference time.