Tsinghua and UC Berkeley release uni-robot-agent framework for open-source robot control

By Ethan

Nov 3, 2026

A joint research team from Tsinghua University and UC Berkeley has released uni-robot-agent, a unified framework for Robot-Use Agents. The open-source model Qwen3.8-Flash-Next achieved a score of 31.39 on the RoboDojo benchmark by directly controlling a robot, surpassing the previously published score of 28.97 by GPT-6 Astra.

Open-Source Models Can Now Directly Control Robots! Tsinghua University and UC Berkeley Release uni-robot-agent, Advancing RUA Training and Local Deployment

On the RoboDojo benchmark, Qwen3.8-Flash-Next scored 31.39 by directly controlling the robot. This score surpasses the previously published 28.97 achieved by GPT-6 Astra on the same leaderboard. Furthermore, Qwen outperformed Astra's published results at approximately 6.6% of the cost.

An open-source model with downloadable weights and a per-episode inference cost of just over USD 1 has outperformed the results delivered by leading closed-source models.

Behind these results, the team is primarily focused on unlocking new possibilities for the future training of open-source Robot-Use Agents (RUAs)—that is, embodied agents.

On November 3, a joint research team from Tsinghua University and the University of California, Berkeley, released uni-robot-agent, a unified framework for embodied agents. The landmark study, titled "Democratizing Robot-Use Agents: A Unified Harness with Open Models," lists Jiarui Yuan as the first author, with Yuxin Zuo and Xiaoyu Huang serving as project leads and core contributors. The work also includes several other core contributors. Notable scholars in the field of embodied AI among the 28 co-authors include Dr. Ning Ding, Assistant Professor in the Department of Electronic Engineering at Tsinghua University; Pieter Abbeel, Professor in the Department of Electrical Engineering and Computer Sciences and Co-Director of the Berkeley Artificial Intelligence Research (BAIR) Lab at UC Berkeley; Zhiyuan Liu, Associate Professor in the Department of Computer Science at Tsinghua University; and Mengdi Xu, Assistant Professor at the Institute for Interdisciplinary Information Sciences at Tsinghua University.

The core contribution of this research lies in the design of a modular, pluggable, and cross-environment unified Harness foundation. Results show that uni-robot-agent not only achieves excellent performance on open-source models, but its modular design also allows researchers to conduct ablation studies on various RUA components, laying the engineering foundation for future multi-environment RUA training.

Team homepage: https://openrua.ai/

uni-robot-agent originates from OpenRUA. OpenRUA is a cutting-edge research initiative launched by researchers from top global universities and frontier laboratories, focusing on embodied AI and foundation models. The project aims to create an open-source, general-purpose agent foundation capable of entering the physical world and interacting with it. To date, uni-robot-agent is the first open-source release from this project, marking a significant milestone.

Beyond Astra: How Can Embodied Agents Take the Next Step?

The traditional paradigm in robotics research involves building dedicated models for specific tasks and robot bodies. Following the release of GPT-6, its capabilities in robot manipulation and navigation have profoundly impressed researchers. On the RoboDojo benchmark, which comprises 42 tasks, GPT-6 Astra has demonstrated zero-shot manipulation capabilities across generalization, precise manipulation, long-horizon execution, memory, and open-vocabulary instruction following. When facing unfamiliar tasks, Astra can now observe the scene, write code, call robot tools, and adjust its next actions based on execution feedback. Capabilities originally designed to solve problems in the digital world are now extending to real-world grasping, moving, and manipulation.

However, advancing this approach from demonstrations to sustainable research still faces three major hurdles.

Robot manipulation requires continuous observation and error correction, with multi-view images and interaction histories repeatedly fed into the model; a single task may involve dozens or even hundreds of API calls. While this is manageable for a few demonstrations, scaling up to large-scale experiments and data collection makes cost a very real constraint. Previous research has shown that although Astra's closed-loop coordination is indeed effective, it is highly expensive.

In terms of autonomy, because the model weights are not open-source, researchers can only access existing capabilities via APIs. This makes it difficult to deploy the models on their own hardware or optimize the model, caching, and inference processes specifically for robot interaction. The speed and scale of experiments largely depend on external services. More importantly, every attempt by the robot generates experience, but researchers cannot directly use these trajectories to update the weights of a closed-source model to form an effective closed loop. The most critical challenge at present is translating benchmark successes into reliable, efficient, and scalable real-world deployment capabilities.

Therefore, the capabilities demonstrated by Astra raise the next question: Can open-weight models be made to use robots just as effectively, allowing researchers to both run and train them, turning today's interactions into tomorrow's capabilities?

uni-robot-agent is a step in exactly this direction.

Design Choices

Today, embodied agents (Robot-Use Agents) have demonstrated promising zero-shot control capabilities. Their basic workflow is straightforward: read the task, view the image, invoke tools, observe the results, and decide on the next step. The model determines what to do, while the Harness translates these decisions into the physical world. Behind this seemingly simple interaction flow lies a complex set of challenges, including capability differences across robots, tool interfaces, execution feedback, and exception handling. The ultimate goal of the uni-robot-agent team is to build a unified, accessible, and generalizable Harness that enables different models to interact with various robots in a consistent manner.

The defining feature of uni-robot-agent is its cross-environment unification and modularity. From the outset, it was designed to serve as a foundation for agentic training. In both simulation and real-robot environments, four categories of tools—perception, motion, skills, and episode control—share a single unified definition, with environment-specific differences handled by dedicated adapters. This design ensures that capabilities learned in one environment can be seamlessly transferred to another, requiring engineers to maintain only a single set of tools.

Viewed as a complete Harness framework, uni-robot-agent functions like a set of modular building blocks. During Agent execution, every component—from the context policy and perception backend to the action model and environment adapter—can be individually toggled or replaced. This design allows researchers to facilitate secondary development and conduct rigorous ablation studies to verify the importance of each component within the embodied Agent.

The team strictly limits the information available to the Agent to what can be obtained on a physical robot: proprioception, robot description files, RGB images and calibration parameters from fixed cameras, and task instructions. Precise depth data from the simulator, object CAD models, and hidden annotations are entirely withheld. Scores from task execution are not fed back to the model, and the skills remain general-purpose rather than customized for specific benchmark tasks. There are no resource hacks, nor are there any information hacks.

The parallel acceleration enabled by uni-robot-agent is equally crucial. When a physical action is highly time-consuming, the tool can immediately return a task handle, allowing the model to proceed with preparing the next step. Meanwhile, actions that share the same resources are planned sequentially. As the environment changes, the plan is updated dynamically, and the Agent can also revoke relevant actions. While the robot arm is in motion, the model is already computing the next step. This approach allows the model to maintain and track overall progress across multiple subtasks, thereby maximizing the advantages of uni-robot-agent.

To bridge visual understanding and physical action, uni-robot-agent provides the model with camera intrinsics and extrinsics to establish the relationship between image coordinates and robot coordinates. This information can be obtained by calibrating the physical camera and robot, eliminating the need to read depth data from a simulator or rely on task-specific localization shortcuts, and it is readily available in real-world deployments. The model can use this information to establish spatial coordinates through triangulation; however, unlike direct depth calibration, the model must infer positions by combining multiple viewpoints and geometric relationships.

"Our reasoning was: if localization could be more direct, how much better would the performance be? Conversely, if the model lacked even calibration information, to what extent could it still perform?"

The team therefore compared three experimental settings: providing both depth and camera calibration, providing only camera calibration, and providing only RGB images.

When depth data is available, the tools can use the camera's intrinsic and extrinsic parameters to directly convert target pixels into 3D coordinates. Without depth, the model must instead find corresponding points across different views and use triangulation to recover the target's position. Test results show that Astra's score dropped only marginally from 55.83 to 55.04, remaining virtually unaffected, while Qwen's score fell from 35.89 to 31.39. This discrepancy indicates that Astra is better at leveraging known camera geometry for spatial localization, whereas Qwen still lags in cross-view correspondence and geometric inference, relying more heavily on directly provided depth information.

Further removing the camera's intrinsic and extrinsic parameters shifts the challenge from leveraging known geometry for localization to independently inferring the spatial relationship between visual observations and physical actions. The model must not only determine where objects are but also understand what the directions and distances in the image imply for the robot's movements. Under this setting, Qwen's score dropped to 9.13, while Astra still achieved 31.29. This more directly reflects the model's inherent spatial reasoning capabilities and clearly demonstrates that calibration information can bridge vision and action for open-source models, enabling those with insufficient spatial abilities to perform effective manipulation.

Simulation and real-robot results

To verify the generalization of the Harness, the uni-robot-agent team conducted evaluations in both simulation and on real robots.

In the simulation environment, the combination of Qwen3.8-Flash-Next and uni-robot-agent scored 31.39, significantly outperforming both GAP and the combination of Astra with Roboprobe, which validates the ability of open-source models to complete embodied tasks. Notably, DeepSeek-V4.1-Flash also scored 27.11 in this test, while GPT-6 Astra achieved 51.90. This demonstrates that uni-robot-agent is not specifically optimized for Qwen, but is rather a Harness that can generalize to any model.

uni-robot-agent delivered even more impressive results in real-robot testing. The team set up a real-robot evaluation, Real Eval, on a dual-arm parallel-jaw platform, covering 19 tasks across five dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks, with 20 points per dimension. Without having seen any real-robot data for the evaluation tasks, Qwen scored 53.50, surpassing the fine-tuned π0.5. Meanwhile, Astra achieved an even higher score of 80.50 under this Harness.

Of course, the applicability of this Harness extends beyond parallel-jaw grippers; it can also be applied to dexterous hands and other platforms involving fine, dexterous manipulation. In a demonstration, the team connected it to a Franka robot arm and a Sharpa dexterous hand. With only simple extensions to the dexterous hand tools, uni-robot-agent paired with Astra successfully completed dexterous tasks such as picking up a small bird plush toy, pressing a green button, and placing an apple onto a plate.

But uni-robot-agent goes even further. While skill libraries in typical Harness frameworks can reuse accumulated manipulation experience, it also raises the question of whether the system might have simply prepared a solution for every task in advance.

To investigate this, the team conducted another ablation study. The experiment disabled both Qwen's access to the robot skill library and its ability to write new skills, leaving only the task instructions, environment descriptions, and basic robot tools. With the skills removed, the system's overall score dropped only slightly from 31.39 to 30.31**. The results indicate that uni-robot-agent's overall performance does not overly rely on the task knowledge provided by the skill library. Of course, this does not mean the skill module itself lacks value; it still provides an interface for experience reuse and future expansion.**

Both sets of experiments point to the effectiveness of uni-robot-agent's design philosophy. The Harness framework needs to provide information that aids reasoning, enable tools to reliably execute actions and return valid feedback, and leave the specific problem-solving to the model. Camera calibration and robot body descriptions help the model establish spatial relationships, while reliable motion tools and execution feedback support it in executing actions, verifying results, and correcting errors. This is precisely the foundation uni-robot-agent aims to build for open-source models.

After achieving impressive scores with open-source models, the team did not stop there. Instead, they began to explore what capabilities open-source models truly need to excel at embodied tasks.

Beyond the scores: what capabilities open-source models actually need

To investigate which capabilities models need to complete embodied tasks, the team also evaluated models of various series, generations, and scales—including Qwen, GPT, and DeepSeek—on RoboDojo, uncovering an unexpected contrast. Test results showed that Qwen3.8-27B could already complete certain tasks, whereas the larger Qwen3.7 Plus failed to succeed in the evaluation tasks.

Similar contrasts also emerged across model generations. GPT-5.6 Sol has already demonstrated a degree of robot manipulation capability, while some newer models still struggle to complete the tasks. Clearly, neither a model's parameter count nor the recency of its version updates can independently explain its performance differences in robotic tasks.

Why does this occur?

During task execution, the robot breaks down a seemingly simple instruction into a sequence of continuous challenges: first clearly perceiving the scene, locating the target, and calculating the action, then judging whether that action is feasible in the physical world. Any subtle deviation from expectations at the beginning or end of a step can cause errors to accumulate, leading subsequent operations to deviate from the target. The team stated that after further examining the models' interaction trajectories, they found that failures primarily concentrated in four areas:

Understanding the scene. Misreading a "0" as a "9" or mistaking an upside-down cup for an upright one can send the operation in the wrong direction from the start. In earlier Qwen Flash and Plus models, such uncorrected visual errors occurred in approximately 12.2%–23.8% of trials.
Locating the target. The model must identify the same physical feature across different viewpoints to recover spatial position using camera geometry. Even for GPT-6.1 Sol, which performed best in this category, 31.5% of relevant calls failed calibration due to inconsistent corresponding points or excessive reprojection error.
Calculating geometry correctly. Translating images into robot actions requires coordinate transformation, pose calculation, and code execution. Earlier Qwen Flash and Plus models made errors in 47.3%–81.8% of geometric calculations, whereas this ratio dropped to 8.2% for Qwen3.8-Flash and 1.5% for DeepSeek.
Understanding the physical feasibility of operations. Whether the wrist orientation is appropriate, if the gripper can approach the target, and how dual arms should coordinate all require the model to understand the physical meaning of the actions.

Beyond analyzing trajectories, can we correlate general model evaluations with embodied performance to uncover underlying patterns? With this in mind, the team compared the RoboDojo performance of six foundation models against benchmarks for embodied reasoning, multimodal understanding, visual grounding, and code generation. They found that all four metrics were positively correlated, with the model's embodied reasoning and multimodal understanding showing a stronger association with embodied performance.

Therefore, the performance of foundation models does not necessarily improve as parameter counts scale up. To enhance robotic capabilities more precisely and effectively, beyond just learning the actions themselves, it is necessary to specifically train models on how to observe, locate, calculate, and judge. This provides clear guidance for the subsequent embodied training of foundation models.

Why uni-robot-agent matters

In the past, discussions about RUA often centered on the most capable closed-source frontier models. However, uni-robot-agent now demonstrates that a smaller, lower-cost open-source model that can be deployed locally can also achieve strong performance on many challenging manipulation tasks.

Moreover, research costs have plummeted at a freefall rate, which is particularly beneficial for small and medium-sized research teams. According to the team's estimates, the API cost per episode for Qwen is USD 1.09, compared to USD 21.29 for Astra. Completing a comprehensive evaluation like the one used in this study costs less than USD 300 for the former and over USD 5,000 for the latter. This means RUA research no longer needs to be confined to a few closed-source models; more developers can build their own systems, run experiments, and explore new directions based on open-source models.

Even more promising is the model's ability to continuously accumulate experience through real-world interaction. With open weights, researchers can not only deploy the model locally to reduce reliance on external APIs but also use the trajectories accumulated during robot interactions to continue training the model. Every attempt, success, and failure by the robot can become data for the next round of training, thereby improving the model's capabilities in return. This allows the agent to form a continuous improvement loop: interacting with the robot, accumulating experience, training the model, and then deploying the trained model back onto the robot for further attempts. This represents a more long-term research horizon enabled by open weights; with the right methodology, using open-source models for effective robot manipulation is no longer just an empty promise.

"We found a reference point for this system: a mini-swe-agent for the coding agent domain. Its goal is to be a simple, universal harness that anyone can use for secondary development, featuring unified interfaces and structured logging so that interaction experiences across different environments can be collected in the same format. Currently, we have implemented the export of Codex interaction trajectories, which serve as the raw material for supervised fine-tuning (SFT) and reinforcement learning."

uni-robot-agent brings together top research teams from the embodied AI communities in China and the United States in a rare collaboration, and its proposed Harness significantly lowers the training barrier for applying foundation models to embodied AI. The encouraging performance of Qwen, which briefly outperformed GPT-6 in certain scenarios, highlights the potential for technological breakthroughs in agentic systems. Building on this research, robotic experience could continuously feed back through training until a true learning loop is established. True to the open-source spirit, uni-robot-agent offers vast room for further exploration and holds great promise.