Guanya Shi: Agents Expected to Unify Model-Based and Model-Free Approaches | CoRL 2026

By Yu Yue

At the Agentic Robotics workshop at CoRL 2026 on November 12, Carnegie Mellon University's Guanya Shi presented research on agentic self-improvement, demonstrating how agents can unify model-based and model-free approaches through real-world and simulation-based auto-research.

On the afternoon of November 12, 2026, the Conference on Robot Learning (CoRL) hosted the "Agentic Robotics" workshop in Rooms 203–204 at the JW Marriott in Austin. Guanya Shi, an assistant professor at the Robotics Institute at Carnegie Mellon University, delivered the second invited talk, titled "Robot Learning from Real-World Experience: Model-Based, Model-Free, and Agent Self-Improvement." His presentation focused on agent-based robot self-improvement and the results his team had already achieved on physical robots. When he first saw these real-robot results, they seemed almost like science fiction.

He began with a question the robot learning community has been asking for years: when will robotics have its ChatGPT moment?

On September 4, GPT-6 Astra was released. On September 8, @cdngdev shared on X that they used Astra to control an inexpensive YAM robotic arm with the prompt "draw this picture of the Golden Gate Bridge onto the canvas." After just five attempts, the results were excellent. The robot continued to improve the drawing iteratively. He considered this a form of recursive self-improvement.

But for those working in robotics, the reaction was mixed. While the community is still waiting for a GPT moment in robotics, GPT's moment in robotics has already arrived. He believes this is actually not a good thing for the community.

On September 8, @cdngdev used GPT-6 Astra to control a robotic arm, painting the Golden Gate Bridge on a canvas

His own earlier realization occurred on March 7, 2026. Andrej Karpathy released the automated research repository autoresearch, using Claude Code to perform hill-climbing optimization on the language model training for nanochat. The progress chart indicated 83 experiments, yielding 15 improvements.

Having trained smaller models himself, Shi knows exactly how difficult it is to tune architectures and loss functions; yet the coding agent was already capable of proposing numerous modifications to lower the validation loss. This led him to ask himself: what is the fundamental difference between improving robot policies and conducting automated research on nanochat?

Over a weekend in April 2026, shortly after the release of Claude Opus 4.7, he tested automated research in the Push-T simulation. The setup was "code as policy," entirely without neural network policies; the agent directly wrote code to control and execute the task. Push-T is a highly challenging multimodal dexterous manipulation problem.

The hill-climbing process revealed that, toward the end, the agent devised a rather clever approach, incorporating planning and various control strategies. This excited him, prompting him to tell his students: why not test it on a physical robot? This was also in April 2026, still before the release of Astra.

They quickly realized there is a fundamental difference between running code-writing agents in a digital environment and deploying them to improve real-world robots. Both nanochat and Push-T simulations act as sandboxes where environments are easily reset, reward signals are readily available, and exploration can deviate significantly from existing baselines. The real world is not a sandbox.

The challenge then became how to close the loop: diagnosing failures, modifying code or training pipelines, and rerunning experiments. He presented two approaches to close this loop. The first, compared to the published work ENPIRE, involves experimenting directly on physical robots. Given a suitable execution framework, this allows for code-level hill climbing on the real hardware. The second, compared to the published work SimEX, involves first training the agent in simulation and then transferring its capabilities to the physical robot for fine-tuning.

ENPIRE involves conducting automated research directly in the real world. He first demonstrated the learning process for a pin insertion task. This did not start from scratch: each code-writing agent was provided with 50 teleoperation demonstrations as a starting point, though the codebase was initially completely empty. The initial success rate was zero.

The agent first attempted to download a vision-language-action (VLA) model for zero-shot inference, which failed. It then tried pure behavior cloning, which also yielded poor results. Next, it realized it needed to perform online reinforcement learning (RL), and subsequently discovered that a behavior cloning term had to be reintroduced into the RL objective. Combining online RL with demonstration data increased the team's average peak success rate by 3.8 percentage points. Adding behavior cloning regularization then boosted the success rate by an additional 10.8 percentage points.

Even after performance neared saturation, extensive local hill climbing continued: fine-tuning the weight of the behavior cloning term added 0.4 percentage points; reducing the batch size from 1,024 to 512 added 0.9 percentage points; and compensating for the underlying proportional-derivative (PD) controller added 1.3 percentage points. This progression was plotted against research wall-clock time, spanning up to 3 hours. Ultimately, the system successfully picked up all six pins on the table and returned them to the pin box.

ENPIRE hill-climbing timeline on the peg insertion task: starting from an empty codebase and 50 demonstrations, the team's average best success rate across 8 robots increases with research wall-clock time

The specific experimental setup was as follows. They collaborated with the NVIDIA GEAR team, utilizing eight robots. Each task began with 50 demonstrations, and each generated idea was given only 20 minutes for implementation. All agents started from an empty codebase.

This operates as a team setup: eight agents work concurrently, sharing a single GitHub repository. Each agent has its own branch, and every idea must be committed to this branch. Other agents can read these ideas, copy them, or even repurpose them for their own use. For instance, Agent 8 extracted ideas from Agent 1 and Agent 2, synthesized its own solution, and achieved a breakthrough in the success rate.

The most difficult part of this project was actually the very first step: building a Gym-like environment for the agent on the physical robot, enabling it to make calls just as it would in a digital world. The approach involved the agent, under human guidance, writing a fixed env.py for each task. This script had to include functions for resetting the environment, retrieving rewards, fetching observations, and executing step commands. Only after this environment was coded could the automated research begin. This initial step could not be fully automated at the time; it was designed collaboratively by humans and the agent.

He cited the reward function as an example: humans and the agent collaboratively devised a highly robust approach using SAM 3 for segmentation, combined with top-down and side-view camera feeds. The system awards a reward of 1 only if both camera views confirm the zip-tie cutting task is successful; otherwise, it assigns 0. This yields a highly robust reward signal. Once the environment is set up, the automated research begins, which actually becomes a rather tedious phase for the human researchers. However, prior to this, the environment was co-developed by humans and the agent.

A fixed env.py for each task; the reward is given only when both the top-view and side-view cameras simultaneously judge the task as successful

The zip-tie cutting task is exceptionally difficult. After four hours of real-robot hill-climbing optimization, the system succeeded and learned several interesting retry behaviors.

It failed on the first attempt but, after a few seconds, tried again and successfully cut the zip tie. The first attempt was somewhat clumsy; the second succeeded on the first try; the third also succeeded on the first try and was even faster. This remains a vision-based policy, starting from an empty codebase and utilizing only 50 teleoperation demonstrations.

A glaring issue with ENPIRE is its high cost: it requires eight agents, a massive number of tokens, eight GPUs, and four hours of real-robot hill-climbing optimization. The biggest hurdle during this optimization process is the motors hitting their physical limits, a step that still requires human intervention. Could this be improved? SimEX proposes a different approach: since ENPIRE barely uses simulation, what if the agent is first trained in simulation and the resulting skills are then transferred to the real robot?

SimEX adopts an even stricter setup. It provides zero demonstrations—not a single one—and forbids any neural policies, enforcing a strict code-as-policy paradigm. The agent cannot train a policy, nor can it download pre-existing ones. Instead, it leverages intuitive physics within a simulation that need not be perfectly accurate to build a reusable toolbox, subsequently transferring these tools, knowledge, and procedures from simulation to the real robot.

He first presented several real-robot results. SimEX performed exceptionally well on barcode scanning: using only 10 minutes of real-robot data per task and limited to five real-world trials, the system completed the task autonomously using a vision-based policy. It also handled towel folding, a task that is often inherently difficult to simulate. Additionally, it successfully placed plates into trays.

All policies demonstrated in these videos are code-as-policy, with no neural policies involved. The real-robot movements appear somewhat slow because safety constraints were added; for safety reasons, the agent is restricted from exceeding a certain speed on the physical hardware.

The specific approach is divided into two stages, with the agent handling the sim-to-real transfer. The first stage involves two agents. One, called the probe agent, is responsible for generating new tasks for the other agent to solve. In the first round, the probe agent proposes a very simple task, such as interacting with just a single bottle.

The other agent solves these tasks using a code policy. In subsequent rounds, the tasks become progressively harder: changing lighting, gravity, and friction, adding every conceivable perturbation, until the final round features a cluttered scene. In one of the intermediate rounds, the success rate barely improved. He joked about this, calling it "agentic domain randomization"—anyone working on sim-to-real transfer would immediately know what he meant.

After this phase of mutual difficulty escalation, they transfer the learned code and operational knowledge to the real robot, entering the second stage. Real-robot trials are limited in number, but there is a key point: the agent can return to the simulation to modify the simulation itself. This is more like the agent performing real-to-sim, and then back to real.

The contrast is clear: learning from scratch on the real robot without simulation, the success rate stays near zero across 5 trials; pre-training the code in simulation first and then transferring to the real robot causes the curve to rise significantly after the second trial, approaching a high level by the fifth. The intuition is: even if the intuitive physics in the simulation is not perfectly accurate, it is sufficient for the agent to learn a reusable toolbox.

SimEX conducts five rounds of increasingly difficult probing in simulation before transferring the toolbox to the real world; the baseline without simulation shows almost no progress across five real-world trials

Shi Guanya wanted to emphasize that this code policy is not hard-coded. When organizing a tray, if a blue plate is flush with the tray's edge, the code policy grasps it from the right; if the plate is further from the tray, it retrieves it from the left. It has scene awareness, and these behaviors are not hard-coded into the policy. Grasping also involves recovery and retrying: if it fails to grasp the object on the first attempt in the first round, it immediately tries again.

He believes that precisely because they adopted an agentic sim-to-real approach, the policy is highly robust and can adapt to various situations. In the barcode scanning task, all three objects were eventually successfully processed. He emphasized again: zero demonstrations, code as policy, no neural policies; and the simulation does not need to be perfectly accurate. They ran simulations for towel folding, where the object simulation quality was very poor and far from the real robot, but it was still sufficient because intuitive physics was enough for the agent to learn a reusable toolbox.

During tray organization, the code-based policy chooses to grasp from the right or the left depending on the plate's position

In the towel folding task, the rough simulation differs significantly in appearance from the real world, but intuitive physics is still sufficient for the agent to learn a reusable toolbox

After finishing SimEX, he used the remaining time to discuss the third piece of work, which serves as a counterpart to the publicly released Embodied First Exam (EFE). This is one of their newer projects: a comprehensive, agent-native evaluation framework designed for code-writing agents performing robotic tasks. The design goal is to be as simple as possible, closely aligning with the evaluation methods used in the language model agent community.

The entire evaluation suite comprises 1,156 tasks, 2,288 tests, 35 suites, and 47 robot types. The underlying infrastructure uses Harbor, allowing a single terminal command to evaluate code-writing agents across thousands of tasks simultaneously. Users can specify their own execution frameworks, such as Codex, Claude Code, Antigravity CLI, or dsh, as well as the model, such as GPT-6 Astra. The task scope is broad, going far beyond pick-and-place to include single-arm, dual-arm, dexterous hand, humanoid, mobile manipulation, quadruped, aerial robots, and musculoskeletal systems.

Embodied First Exam: 1,156 tasks, 2,288 tests, 35 suites, and 47 robot types, evaluable with a single command

The evaluation features three settings to emphasize the complexity of the real world. In the privileged setting, the agent and the simulation run in the same container, allowing it to directly read the full state of the simulation and reset it on demand; this setting includes 391 tests in the core task. In the standard setting, the agent and the simulation run in separate containers and can only communicate through predefined interfaces, receiving sensory inputs like RGB and RGB-D, along with proprioception, but unable to read object poses; this setting corresponds to 456 core tests.

What he found most interesting was the real-time setting: 54 dynamic environment tests in the core task suite ran under this mode. The rules were identical to the standard setting, but the simulation did not pause while the agent thought; it had to reason while the world continued to run. Claude Fable 5.2 had just been released, so the students immediately ran supplementary experiments and included Fable in the results. Unsurprisingly to him, Fable outperformed Astra.

The total number of tasks was 1,156. Because running all of them was too costly, they used manual judgment and then ran GPT-6 Luna as a first pass to filter out overly simple tasks that Luna could solve, leaving 456 core tasks.

Under the standard setting, the agent only received raw sensory inputs. Claude Fable 5.2 achieved 50% (226/456), GPT-6 Astra achieved 34% (157/456), Claude Opus 5.5 achieved 25% (114/456), and Gemini 4 Argon achieved 24% (110/456). They also tested other models: Kimi 3.1 Pro scored 10% (46/456), Gemini 3.8 Flash scored 6% (29/456), Qwen3.8 Max scored 4% (17/456), and DeepSeek V4.1 Flash scored 3% (14/456). The real-time setting was harder because the simulation did not pause. There were only 54 tests for this in the core set, and the numbers were lower: Fable 5.2 reached 41% (22/54), Astra reached 31% (17/54), and Gemini 4 Argon reached 9% (5/54), with the rest scoring in the single digits.

Embodied First Exam standard setting leaderboard: across 456 core tests, Claude Fable 5.2 achieves 50% and GPT-6 Astra achieves 34%

In the privileged setting, the agent could already read information such as geometry, friction, and contacts. Fable 5.2 achieved 80% (311/391), Astra achieved 79% (307/391), Claude Opus 5.5 achieved 60% (233/391), Gemini 4 Argon achieved 43% (167/391), and Kimi 3.1 Pro achieved 38% (148/391). Open-source and smaller models performed noticeably lower: DeepSeek V4.1 Flash scored 17%, GLM 5.3 Flash scored 10%, and Qwen3.8 Max scored 9%.

The numbers were significantly higher than in the standard setting, but still far from achieving 100% completion. He believes the robotics problem remains hard because the world is inherently partially observable.

Embodied First Exam privileged setting leaderboard: across 391 tests, Claude Fable 5.2 achieves 80% and GPT-6 Astra achieves 79%

The task distribution was carefully designed. They created 9 original suites comprising 411 tasks in total: RoboARC has 106 tasks involving untying knots, threading ropes, and assembling IKEA furniture; DexHandExam has 30 tasks requiring dexterous hands to flip pages and open cans; RoboPaint has 43 tasks where a robotic arm dips a pen to draw and write; RoboDynamicExam has 25 tasks involving catching, hitting, and handing over moving objects; Embodied Olympics - Fun has 79 tasks for humanoid robots to perform athletic skills; Embodied Olympics - Match has 14 tasks consisting of full sparring bouts; iSpatialGym has 12 tasks where goals are provided only as images; Embodied Safety Exam has 72 tasks requiring the execution of safe requests and the refusal of harmful ones; and RoboEngineer has 30 tasks that involve tuning a controller first and then deploying it to previously unseen dynamics.

They also incorporated 26 existing community suites with 745 tasks, such as RoboDojo, RoboCasa, BEHAVIOR-1K, LIBERO-MAX, HumanoidBench, and RoboTwin 2.0. In terms of morphology, there were 417 single-arm tasks, 256 dual-arm tasks, 221 mobile manipulation tasks, 134 humanoid tasks, 62 dexterous hand tasks, 22 musculoskeletal tasks, 19 aerial manipulation tasks, and 17 quadruped tasks. In terms of capabilities, there were 482 object manipulation tasks, 209 articulated mechanism tasks, 204 precision and contact tasks, 251 multi-object rearrangement tasks, 112 navigation and search tasks, 89 dynamic interaction tasks, 83 deformable and fluid tasks, 75 tool use tasks, and 51 locomotion and body movement tasks.

Task distribution in Embodied First Exam: 9 original suites with 411 tasks, plus 26 existing benchmark suites with 745 tasks

He specifically emphasized that they did not mandate which method the agent must use to control the robot. There are generally four approaches. The agent can directly output control signals, meaning "agent-as-policy," such as directly outputting the end-effector pose in RoboTwin 2.0; it can write code to control the robot, as in RoboLab; it can train a policy in simulation and then deploy it for control, as in HumanoidBench; or it can mix these approaches, as in Embodied Olympics - Fun.

They believe the agent itself should determine which approach to use, rather than having the method predetermined. In some toolkits, they equipped the agent with a GPU to train policies. For instance, in the fencing task at Embodied Olympics, the target was a humanoid robot, and the agent used the GPU to train a whole-body control policy.

As for the results, the key takeaway he wanted to share is: it is not a winner-takes-all scenario. While there is extensive analysis and data in the publicly available work for comparison, he focused only on the point he values most. He presented a chart to illustrate this: it features various agents, with each bar representing a task. The color indicates which agent successfully completed the task, and the number shows how many agents were able to accomplish it.

This chart evaluates 456 core tests under standard settings. GPT-6 Astra successfully completed 153 of them and failed the remaining 303. The first observation is that it remains very expensive. At list prices, the cost per attempt ranges from USD 0.30 to USD 30.

The second point is that no single model monopolizes the performance. While Astra succeeded in certain tasks, other models—including smaller ones—occasionally managed to complete tasks that Astra failed. He noted that capabilities are distributed across different models, leaving significant room to develop meta-agents: a unified framework that can invoke various types of agents.

Across 456 core tests in the standard setting, the tasks completed by different agents do not overlap; the list price cost per trial ranges from USD 0.3 to USD 30

He also shared a personal anecdote. He had his PhD student incorporate his own doctoral research into this benchmark. One of the tasks is Neuralfly Vector Wind: it requires completing six timed laps along a randomized figure-eight trajectory. The flight plane can be vertical, horizontal, or tilted, and the drone must fly in both directions. The semi-axis lengths range from 0.9 to 1.5 meters and 0.45 to 0.85 meters, respectively, with an angular velocity of 0.8 to 1.1 radians per second. The phase, center, and initial offset all vary randomly, and the agent must track position, velocity, and acceleration commands provided in the world coordinate system. At least in simulation, the agent successfully completed these tasks—thanks to a highly effective simulator he developed back in the day.

Finally, he stated, he genuinely believes that agents will reunite the model-based and model-free approaches to policy learning. He referenced a blog post by Ken Goldberg: traditionally, proponents of model-based and model-free methods have looked down on each other. However, from the perspective of an agent, both vision-language-action (VLA) models and direct control can serve as skills and tools at different levels of abstraction.

He believes there are still many research questions worth exploring in this area. Currently, the tools invoked by agents are mostly legacy robot skills. How can we learn new skills and tools that agents truly need—skills and tools that are native to the agent? What should the interface between agent-native and robot-native systems look like? Should it be based on tracking, latent space, or token space, or is there another, more suitable form?

Furthermore, for high-dimensional systems like humanoids, how can skills be composed spatially and temporally, and how can upper-body and lower-body skills be integrated? He considers these to be open questions that the community should work together to clarify.

After the presentation, an audience member asked about using agents for contact-rich motor control. Shi Guanya believes that such tasks still require a low-level System 0 for now, but agents can design how System 0 is trained and can also follow a real-to-sim-to-real pipeline. He feels that System 0 is still necessary.

Relying purely on the "agent-as-policy" approach is not yet practical. If simulation is good enough, he believes it is entirely possible to write model predictive control to govern everything, but he has not yet seen this capability emerge.


Workshop on Agentic Robotics (CoRL 2026 workshop website, November 12, 2026) https://agentic-robotics-workshop.github.io/

ENPIRE: Agentic Robot Policy Self-Improvement in the Real World (arXiv:2606.19980, CoRL 2026) https://arxiv.org/abs/2606.19980

SimEX: Simulation-Integrated Robotics AutoResearch (arXiv:2609.38982) https://arxiv.org/abs/2609.38982

Embodied First Exam (arXiv:2611.12177) https://arxiv.org/abs/2611.12177

Embodied First Exam website https://www.embodied-first-exam.ai/

This article is based on live reports from the workshop on November 12, 2026.