Robot-use agents have sequentially overtaken world and vision-language action models on the RoboDojo leaderboard over a 33-day period, prompting the creators of the prominent embodied AI benchmark to announce structural adjustments to maintain evaluation fairness.

33 Days of Relentless Succession: Agents 'Camped on Mount Everest' and RoboDojo Entering an Adjustment Period

RoboDojo is one of the representative evaluation benchmarks in embodied AI. Comprising 42 simulation tasks and 18 real-robot tasks, it serves as a comprehensive aptitude test for robots. Building upon simulation environments such as CALVIN, LIBERO, RoboTwin, and RMBench, as well as real-robot benchmarks like ManipulationNet and RoboArena, RoboDojo fills the gaps left by previous benchmarks. It achieves this by providing unified, structured evaluation, sharing policy interfaces and evaluation pipelines, and combining capability-focused simulation diagnostics with reproducible real-robot validation.

Upon its release, the core team behind RoboDojo, including Tianxing and his colleagues, hoped the benchmark would accelerate the development of general robot manipulation policies. Even to outsiders unfamiliar with the field, its symbolic weight is clear: it represents the "Mount Everest" of embodied AI.

Just as mountaineers in the real world aspire to summit the actual Mount Everest, countless eager challengers are gathering at the foot of this embodied peak. The contenders include not only research from scholars and student teams at top universities like Peking University, Tsinghua University, UC Berkeley, and NUS, but also cutting-edge tech companies such as Physical Intelligence, OpenAI, AgiBot, Xiaomi Robotics, Meituan, RobotEra, and Manifold AI. During this period, diverse technical approaches—including world action models (WAM), vision-language-action (VLA) models, and agents—flourished. From the base to the midway point of the mountain, the surge of emergent intelligence and fierce technological competition invites a compelling question: who will ultimately plant the flag at the summit?

However, over the past month, this landscape has changed dramatically.

33 Days of Relentless Succession

Since OpenAI released GPT-6 Astra in early September 2026, subtle but profound changes have begun to unfold at the level of embodied agents (robot-use agents).

On September 28, PhysicalRSI, developed by MMLab at The University of Hong Kong and Kinetix AI, propelled an agent to the top of the leaderboard for the first time.

On October 14, EmbodiedEvo from Einsia.AI claimed the top spot with a score of 45.08, overtaking PhysicalRSI.

Three days later, the WorldScape-R²SI agent developed by Manifold AI claimed the top spot with a score of 46.58, overtaking Einsia.AI.

Just three days after that, HexaShell from Hexa Future claimed the top spot with a score of 53.18, overtaking its predecessor.

On October 24, PyRUA-Lean from Peking University's DAGroup once again overtook HexaShell to take a temporary lead with a score of 55.26.

On October 27, GO-3 Nano Preview from AgiBot took first place on the leaderboard, replacing Peking University's PyRUA-Lean, as the average score surpassed 60 for the first time.

On October 30, Q0-Preview from Qoo Tech reclaimed the top spot with a score of 74.20, overtaking AgiBot.

Public results show that Q0 ranked first in four of the five major dimensions—memory, open vocabulary, precision, long-horizon, and generalization—scoring 93.67, 75.0, 58.5, 58.5, and 58.67, respectively. Because RoboDojo's evaluation policy allows teams to participate without publicly releasing their project papers or code, we have yet to determine exactly why this work delivered such a massive performance boost for the agent.

In just 33 days, successive breakthroughs from various teams have propelled the agent track far ahead of WAM and VLA approaches. Why is this happening? Does this mean that embodied agents are poised to end the technological race for general-purpose robots?

Why the gap widens

What is the common thread across the recently open-sourced RUA studies?

The common thread across multiple studies is leveraging the programming capabilities of large models for robot policy generation. Upon processing the camera feed and understanding the task, the agent writes a control program to actuate the robotic arm and gripper, then adjusts its approach based on the updated visual feedback and execution results. Conditional logic, loops, and variables within the program handle tasks that the model would otherwise need to learn through training. This breaks down complex tasks into step-by-step verifiable operations, making it easier to trace failures to specific execution steps. Researchers can simply modify a segment of code and retry, eliminating the need to retrain the model from scratch for every improvement.

However, this shared paradigm alone does not explain the performance gaps between different methods. Peking University’s DAGroup introduced PyRUA-Lean, which modifies how the agent interfaces with the robot by generating a single Python script. This script handles continuous operations and local retries internally, fetching images and state data only when necessary. This approach reduces the overhead of repeatedly querying the model while ensuring that well-defined operations execute seamlessly.

HexaShell focuses more on bridging the gap between vision and action, while decoupling task execution from tool repair. It employs two agents: one operates the robot, while the other reads failure logs, inspects the tool code, modifies it, and retests. Successful modifications are saved as code and notes, which are then frozen before formal evaluation. This allows subsequent attempts to reuse the repaired tools without having to rediscover solutions for the same errors.

EmbodiedEvo follows a similar philosophy but places greater emphasis on transferring experience to subsequent executors. The system first explores in practice environments, distilling reusable operations into coded skills. It encodes applicability conditions, failure scenarios, and remedial actions into its decision logic before passing these skills to a new agent for iterative use in previously unseen layouts.

PhysicalRSI expands the modifiable scope to the entire control pipeline. The large model responsible for comprehension and diagnostics rewrites skills, tool calls, and control code based on execution outcomes, generating multiple candidate versions and selecting the best one through testing. It also allows the invocation of VLA skills. Rather than merely fixing an isolated action, this approach aims to improve how the robot selects and orchestrates its next steps when encountering a specific situation.

Manifold AI takes a significantly different approach to designing the model's external agent. Its WorldScape-R²SI framework further integrates the action model into the improvement process. An internal Robot Agent decides when to use control tools and when to invoke the WorldScape Policy for execution, while an external Recursion Agent analyzes the execution logs. The agent modifies both the skills and the execution pipeline, and also filters experience to train the action model. The updated version is then tested and validated before being deployed in the next execution cycle.

These differences in agent self-iteration directly impact performance across various tasks. Key takeaways include: the execution sequences generated by these agents are generally more compact, which helps reduce redundant operations; localization is more reliable, making grasping and placing less prone to failure; and the retained experiences preserve their applicability conditions, making them more likely to remain effective when the layout changes. The integration of action models, meanwhile, offers an alternative approach to executing complex operations.

Although recent top-ranking entries such as GO-3 Nano Preview and Q0-Preview remain unpublished, the boom in the agent track at least demonstrates that, as foundation model capabilities improve, the competition over agent harnesses is converging toward engineering compactness—specifically, whether the entire system can respond quickly, correct errors, and reuse past experiences.

Why, then, has this current boom not extended to WAM and VLA?

The evolution of technical approaches has objective boundaries. From a broader perspective, the WAM and VLA routes are not as tightly coupled with agents as one might assume, and they differ significantly in how they leverage existing model capabilities. VLA maps vision and language directly to actions, while WAM tends to model actions in conjunction with subsequent environmental changes; both require training to establish these connections. Agents, on the other hand, can directly leverage the semantic understanding, coding, and reasoning capabilities already possessed by general foundation models, and then interface with the robot using tools.

Consequently, for tasks such as interpreting ad-hoc instructions, recalling prior context, and orchestrating multi-step operations, agents can quickly capitalize on the accumulated capabilities of general foundation models. When faced with requirements they have not been specifically trained for, they can dynamically combine existing tools on the fly. This allows them to handle task variations that are difficult to cover with standard robot training data through reasoning and programmatic composition. This is likely a key reason why agents excel in tasks involving memory, open-ended instructions, and long-horizon execution.

Another factor may be the stark difference in how iteration is implemented. The difficulty of designing the brain from scratch is incomparable to simply modifying the scaffolding. For systems that rely primarily on learning to generate actions, a recurring issue typically requires collecting more data, adjusting the training process, or modifying the model itself. For models like WAM, adjusting the training demands massive computational resources and time. In contrast, many issues with agents can be addressed directly in the code. For instance, adding a verification step after a failed grasp, or breaking down and re-executing an error-prone action, can be validated quickly. During phases of frequent leaderboard updates, this speed of iteration translates into a distinct advantage.

Nevertheless, it is overly simplistic to attribute the back-and-forth between these approaches entirely to the inherent superiority of one technical route over another. This is not only because several studies have consistently highlighted the general difficulties agents face in fine manipulation, but also because the different paradigms are increasingly borrowing from and integrating with one another. For example, PhysicalRSI actually incorporates the invocation of VLA, and WorldScape-R²SI does not completely abandon WAM. Other studies have also emerged that blend agent and WAM methodologies.

No one denies that the release of GPT-6 has shown more people the potential of RUA, prompting the realization that "agent capabilities have changed; the era is indeed shifting." Perhaps the future division of labor will see agents handling task comprehension, step planning, and exception management, while action models take charge of more continuous and proficient execution. The current lead held by agents on the leaderboards also reflects the progress brought by this combined approach. Yet, beyond these technical trends, one must ask: does the current boom in the agent track also contain anomalous elements?

Anomalies in the "agent boom"

The release of GPT-6 Astra has drawn more researchers to applying foundation models to embodied agents (Robot-Use Agents). Meanwhile, systems like HexaShell and Embodied Evo continue to set new records on benchmarks such as RoboDojo, LIBERO PRO, and RoboTwin 2.0. Yet, it is worth examining whether the current "agent boom" on embodied AI evaluation leaderboards hides any anomalies.

Indeed, it does.

In the paper "What Are We Actually Benchmarking in Robot Manipulation?", researchers have identified failure modes in benchmarks such as LIBERO, CALVIN, SimplerEnv, RoboCasa, and RoboTwin 2.0 across certain scenarios. These include solutions that exploit shortcuts, statistically insignificant results, overfitting, and training on data closer to the test distribution to artificially match the actual test performance of larger, more general models.

Furthermore, some dimensions within these benchmarks are no longer sufficiently effective, preventing the evaluations from objectively reflecting a model's true capabilities in certain scenarios. Naturally, any ranking competition involves "drawing the target before throwing the dart," and seeking external attention to drive growth is understandable. However, intentionally exploiting benchmark blind spots—equating "closeness to the tested metrics" with a model's actual overall capabilities and using benchmarks for exaggerated marketing—is unacceptable. This practice does not contribute to the progress or real-world deployment of cutting-edge AI technologies.

Coincidentally, the OpenRUA team previously noted in their paper "Behind the RoboDojo Boom" that since early 2026, a large number of embodied agents have been breaking records on these leaderboards. The study identifies three underlying loopholes behind this "abnormal boom" on evaluation leaderboards.

First, to maintain fairness, RoboDojo imposes a native step limit of 200 to 1,900 steps for each task. Initially, this limit was specifically designed for policies that output a single action or a short action sequence per step. In World Action Models (WAM) or Vision-Language-Action (VLA) models, this is suitable for testing the efficiency of control policies that output one action or a short sequence per step; beyond the step limit, the evaluation should assume a decrease in the model's capability and success rate. However, due to differences in technical approaches, an agent harness can replace this limit with other formats during actual testing without detection, such as allowing a maximum of N decisions or N seconds of runtime.

OpenRUA's testing on RoboDawn revealed that out of 99 successes for certain agents, 97 actually occurred outside the native step limit. If evaluated strictly under the native step limit, the actual success rate would plummet from 47% to 0.1%. As shown in the figure, because different studies adopted different replacement methods, the results on the same leaderboard actually derive from varying computational budgets, leading to a lack of standardization in compute allocation.

Another issue lies with the verifier. A real robot must rely on its sensors to determine whether an action is complete and then stop, a judgment that is extremely difficult for the robot to make. However, many harnesses bypass this by feeding the simulator's judgment directly to the Agent. For example, they might leak privileged information during tool returns, or simply terminate the episode on the Agent's behalf the moment success is detected.

Ideally, tool return values should not contain privileged information such as task completion status or progress, nor should they expose real-time scores to the Agent, which would be equivalent to revealing the exam answers in advance. Once the Agent terminates an action by calling done or give_up, that first call should be treated as the final decision.

OpenRUA testing also revealed that some systems on the leaderboard engage in repeated trial and error directly on the test tasks, use the benchmark's own verifier to generate labels, freeze the resulting skills, and then report scores on other layouts from the same batch of tasks. This is hardly fair.

In fact, patches in RoboDojo such as seed control, hidden layout masking, and checkpoint auditing were mostly designed for standard policies, not for Agents capable of writing code, retaining memory, and learning through trial and error. The capabilities of embodied agents are indeed growing increasingly powerful as foundation models scale up. Although the RoboDojo team has made every effort to filter out the anomalies behind this apparent prosperity, the original unified evaluation benchmark covering WAM, VLA, Agent, and Code As Policy is no longer suited to increasingly divergent technical approaches. This 'Mount Everest' of embodied AI is now facing an adjustment of its own.

Recently, Tianxing Chen, the first author and a core maintainer of the benchmark, publicly stated on social media that it might be time to 'divide and conquer.' Because RoboDojo was created before the emergence of Astra and existing agentic solutions iterate rapidly, the team has been actively listening to community feedback to maintain fairness and plans to adjust the leaderboard in the future. The adjustment plan includes, effective December 1, 2026, tracking data hacking, maintaining additional closed-set tests, and establishing an independent Agent sub-leaderboard. Scores will be displayed on both the overall and Agent sub-leaderboards but will not count toward the Open dimension score. The team also plans to introduce stricter open-source requirements—meaning all solutions must be rapidly open-sourced on XPolicyLab—continuously iterate on evaluation tasks, and develop assessments better suited for agentic solutions.

Additionally, industry insiders have offered several suggestions: 'Given that the code-as-policy approaches in some submissions are excessively exaggerated and refuse to be open-sourced, we recommend introducing more randomization into the evaluation, or incorporating the actual execution time of the agent into the assessment.'

Regarding potential future adjustments to RoboDojo, 42HOW Robotics reached out to relevant team members for comment. As of press time, no public response has been received.

The robotics industry remains somewhat restless, largely because AI technology is still evolving at an exceptionally rapid pace. Just a year ago, VLA was still a novel concept. In such a fast-moving sector, frontline professionals fear making the wrong move, falling behind, or failing to survive until the next quarter. Amid the hype surrounding RSI, AGI, and physical AI, and the constant chorus of 'torch-relay-style' claims of reaching new peaks, the ability to stay grounded and resist being consumed by anxiety is testing the resolve of the vast majority of industry practitioners.

Meanwhile, the rapid advancement of frontier technologies is challenging the validity of evaluation benchmarks. As a reference testing standard, a benchmark should not be modified too frequently. The disparity between the speed of technological iteration and the pace of benchmark updates has also created a growing gap between the two.

A review of representative benchmarks such as RoboCasa, RoboDojo, LIBERO, and RoboArena reveals that many have made timely adjustments to dimensions and methods affecting evaluation fairness. These adjustments include raising the difficulty of assessments in areas such as domain randomization, action precision, disturbance generalization, episode length, data annotation, and real-robot deployment, or imposing stricter requirements on the thoroughness of submitted work. Currently, embodied agent (i.e., RUA) solutions iterate on a weekly basis, whereas industry-wide evaluation protocols are updated on a quarterly or yearly basis. This creates an asymmetry in time costs between the two. Furthermore, since 2026, the volume of submissions across various research tracks has continued to rise, making it a genuine issue that frontier models or agents over-exploit omissions in evaluation dimensions.

For an evaluation benchmark to maintain objectivity and ensure fairness, transparency, and scientific rigor, technical adjustments to certain dimensions are entirely normal. At the same time, the industry must calmly consider a fundamental question: what is the original purpose of an evaluation benchmark?

The original intention behind RoboDojo was to establish a unified evaluation loop to broadly assess existing robot manipulation strategies, create an open and transparent platform for technical exchange among researchers, and point toward key breakthroughs in the general-purpose robot technologies that will determine the future of the entire embodied AI industry. However, as the domestic industry enters its actual race, the rising number of team submissions has led to benchmark rankings being increasingly used in brand promotion and fundraising materials. Consequently, evaluation benchmarks have naturally garnered attention and gained the influence to sway short-term commercial interests. Against this backdrop, collective industry supervision and the joint maintenance of the long-term scientific rigor and objectivity of these benchmarks have become crucial.

What changes with technology, and what must not

To some, establishing a benchmark might seem like a cumbersome task. While certain standards must naturally evolve alongside technological progress, some principles must remain unchanged.

For instance, once a benchmark is established, the organizing team has a responsibility to maintain its objectivity and fairness. They must scrutinize submitted papers, projects, and datasets, evaluate reproducibility, and present to the public only those achievements that genuinely reflect the technological progress of our time—while filtering out the dark, utilitarian, cunning, and hollow aspects of frontier research. Submitting teams, in turn, should curb the habit of looking for shortcuts and avoid trying to game the system with work that is difficult to reproduce or fails to meet objective standards. Only then can the academic community and industry ecosystem, which are built upon the peer-review system, be safeguarded. It also allows the outside world—the general public, pre-university students, and technology enthusiasts—to witness how frontier technology is genuinely evolving through exploration.

Approximately 65 million years ago, the Indian subcontinent and the Eurasian plate collided with immense force, compressing, folding, and uplifting the Earth's crust to form what is now Mount Everest. Today, multiple technological tracks—such as WAM, VLA, and Agents—are advancing in parallel and colliding with one another. RoboDojo, the evaluation benchmark symbolizing the Everest of embodied AI, is likewise enduring the howling winds at the very forefront of technological exploration.

Top university research teams and technology companies in China continue to race against the clock in the ascent toward physical AI. Yet it is worth remembering that the objective advancement of frontier technologies and decisive breakthroughs cannot be rushed by impatience; only the steady and relentless will see the climb through to the summit.