Tsinghua University Releases HandsOn Simulation Benchmark for Dexterous Manipulation with 18,000 Demonstrations

Most mainstream robot manipulation benchmarks focus on two-finger grippers, leaving five-finger dexterous hands under-evaluated with inconsistent metrics. To address this, researchers from Tsinghua University, Beijing University of Posts and Telecommunications, Natural Will, and Peking University have introduced HandsOn, a large-scale simulation benchmark featuring 66 tasks and 18,000 human teleoperation demonstrations.

Mainstream benchmarks in robot manipulation are predominantly built around two-finger grippers. Evaluations for five-finger dexterous hands remain scarce and limited in scale, with inconsistent task setups and scoring criteria, making it difficult to compare experimental results across different research teams.

On October 19, Tsinghua University, Beijing University of Posts and Telecommunications, Natural Will, and Peking University jointly released the paper HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation. The paper lists 17 authors, with Tianwei Luo, Yiming He, Hongjin Wu, Ting Pan, Debin Ren, and Haorui Guo serving as co-first authors. The corresponding author, Ding Ning, is an assistant professor and doctoral supervisor in the Department of Electronic Engineering at Tsinghua University, and is also affiliated with Natural Will. Ding has conducted extensive research in artificial intelligence, with publications in top-tier journals and conferences including Nature Machine Intelligence, ICLR, and NeurIPS.

The team simultaneously open-sourced the HandsOn dexterous manipulation simulation benchmark, which includes 66 tasks and 18,000 human teleoperation demonstrations, with both the code and datasets made fully publicly available.

Using this benchmark, the research team tested several mainstream robot models. The results indicate significant room for improvement in dexterous manipulation, while also validating that the benchmark can effectively differentiate the performance of various approaches.

Dexterous hand evaluation lacks a common standard

Over the past few years, robot manipulation datasets and evaluation frameworks have been predominantly designed for grippers, and existing large-scale simulation and real-world manipulation datasets remain gripper-centric. While grippers excel at simple pick-and-place tasks, many everyday actions require the coordinated use of multiple fingers: turning knobs, pressing buttons, gripping a hammer to strike, or aligning holes to insert pins. It is difficult to design appropriate tasks for these actions within a gripper-based evaluation framework.

Dexterous hand evaluation is not entirely unexplored territory. Early evaluations of dexterous manipulation largely relied on reinforcement learning or utilized very small demonstration sample sizes. This year, however, a series of benchmarks based on learning from demonstration has emerged.

DexJoCo includes 11 tasks and 1,100 human demonstrations; DexVerse comprises 100 tasks and 3,180 demonstrations; and Bench2Dex features 26 tasks and approximately 1,300 demonstrations. The task lists and evaluation boundaries across these benchmarks are not standardized. Related work also mentions LabDex, a hierarchical benchmark designed for laboratory settings.

Clear progress has been made on industry hardware standards. According to the National Standard Library registry, the group standard T/CIE 387-2026, General Technical Requirements for Dexterous Hands of Humanoid Robots, administered by the Chinese Institute of Electronics, was released on March 25 and took effect on April 1. Drafting organizations include the China Industrial Internet Research Institute, among others. This standard governs the dexterous hand hardware itself and applies to hardware testing and evaluation. However, the industry still lacks a widely recognized benchmark for evaluating the task completion capabilities of dexterous hand control models.

This is precisely the gap HandsOn aims to fill by providing a unified evaluation framework for dexterous hand control models, featuring a diverse range of tasks, abundant demonstrations, and support for standardized cross-model comparisons.

What HandsOn does

HandsOn is a composable simulation platform built on NVIDIA Isaac Sim 5.1, supporting five robot arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable objects. The platform includes 18,000 human-curated teleoperation demonstrations across 66 tasks. According to the paper's metrics, this demonstration volume is 5.7 to 16.4 times larger than the number of human trajectories in representative dexterous manipulation benchmarks. The authors also reused and adapted select task designs and object assets from DexJoCo and DexVerse.

On the open-source front, the code repository is released under the MIT license, providing task environments, success criteria, evaluation configurations, and integration code for the nine models tested in the paper. Demonstration data collected via the PhysX backend is publicly available on Hugging Face. The dataset for the Newton backend is marked as "coming soon," and its evaluator currently covers only two tasks: "insert pin" and "place fruit in bucket." The model weights used for evaluation in the paper have not been released, so fully reproducing the reported scores requires training the models from scratch using the paper's methodology.

The 66 tasks are divided into two main sets.

Core-50 is "practice before test": All 50 tasks include demonstration data, evaluating whether a model can learn from these demonstrations and execute them reliably. Based on finger dexterity requirements, these are divided into 17 basic manipulation tasks, 18 functional manipulation tasks, and 15 contact-rich manipulation tasks, covering tool use, numerical and spatial reasoning, articulated object manipulation, and bimanual coordination.

Transfer-16 is "test without practice": These 16 tasks have no corresponding training demonstrations. Seven tasks introduce unseen objects or novel object relationships, six require chaining learned skills in sequence, and three involve deformable object manipulation, despite the complete absence of deformable object demonstrations in the dataset.

The task scenarios are highly concrete.

Core-50 includes tasks such as "stirring a pot," "setting a timer," "pouring seasoning," "squeezing scissors," and "photographing a sign." Some tasks require twisting two knobs simultaneously with both hands, while others involve passing a block from one hand to the other.

Transfer-16 features composite tasks like "scrubbing a bottle and then filling it with water," alongside deformable object tasks such as "folding a towel in half twice." Each task explicitly defines its completion criteria. For instance, a mouse must first be moved 8 centimeters toward a mousepad before a click is registered, and two knobs must simultaneously stop within 5 degrees of the target position and maintain that alignment.

The time limit for each Core-50 task is set to three times the 75th percentile (P75) of the expert demonstration duration, providing a margin for the model. Since Transfer-16 lacks corresponding demonstrations, its time limits are based on estimated durations, with the three deformable object tasks assigned specific limits of either 240 seconds or 180 seconds.

The entire task suite follows a standardized design and review process. Each task begins with a "task card" that defines the scenario, success criteria, and failure conditions. Before a new task is launched, at least 20 trial demonstrations are collected, requiring an agreement rate of over 98% between the labels and objective evidence. The roles of data collectors, reviewers, and arbitrators are strictly separated so that no one reviews their own data. Approximately 20% of each data batch undergoes random blind review, and 5% to 10% of the approved data is subject to a secondary review. Prior to release, collectors manually verified all 66 tasks in the release environment to ensure their physical feasibility.

The demonstration data is collected through human teleoperation. Operators wear MANUS Prime I data gloves to capture finger movements and use HTC VIVE trackers to record wrist positions, driving the robotic arm and dexterous hand in real time within the PhysX physics engine. The paper reports that at least 10 operators participated in over 1,050 collection sessions, yielding 18,000 trajectories (15,178 for the training set and 2,822 for the validation set). The total recording time is approximately 56 hours, with each trajectory featuring three synchronized camera views and a control frequency of 30 Hz.

To prevent the model from merely memorizing fixed visual inputs, the team applied high-fidelity re-rendering to the collected trajectories. This involved swapping four tabletop materials, three backgrounds, and three ceiling textures, along with introducing minor perturbations of ±2 millimeters and ±0.3 degrees to the camera poses, while keeping the action labels unchanged. The language instructions were also augmented: each of the 91 base instructions was paired with 32 synonymous rewrites for training and eight additional rewrites reserved for testing.

Results: Difficulty and discriminability

The paper evaluates 10 methods: five vision-language-action (VLA) models (π0.5, DM0.5, LingBot-VLA, HY-Embodied, and Spatial Forcing), three world-action models (WAM) (OpenWAM, FastWAM, and Motus), a task-specific diffusion policy (DP), and GPT-6 Astra (hereafter GPT-6), which is prompted directly without any fine-tuning. VLA models map visual inputs and language instructions directly to actions, whereas WAMs predict the subsequent visual frame alongside the action. All eight multi-task models were trained on the Core-50 demonstrations. Across all trained models, evaluation on Core-50 comprised approximately 45,000 episodes in total, including the diffusion policy.

The eight multi-task models exhibited a clear performance gap. OpenWAM achieved the highest success rate at 29.1%, followed by FastWAM and π0.5 at 26.4% and 24.9%, respectively. DM0.5, LingBot-VLA, HY-Embodied, and Motus scored 21.4%, 19.7%, 18.9%, and 18.1%, respectively, while Spatial Forcing reached 12.4%. The diffusion policy scored 9.7% (trained on a per-task basis, with an evaluation protocol not entirely consistent with that of the multi-task models). If the best-performing model is selected for each individual task, the overall success rate reaches 35.4%. OpenWAM achieved or exceeded a 50% success rate on 11 tasks. Across the 50 tasks, at least one model successfully completed 47 of them at least once. When restricting the statistics to episodes completed within one times the expert execution time, π0.5 ranks first at 14.6%, while OpenWAM drops to 8.3% and ranks sixth.

Task difficulty scales progressively with the complexity of finger manipulation. The researchers grouped tasks by their specific finger manipulation requirements and aggregated the episodes across all eight models. Tasks requiring only pushing or flipping achieved a 51% success rate. Opening and closing hinges, drawers, levers, and lids reached 43%, while pick-and-place tasks scored 35%. Using fingers to pull triggers, press buttons, and turn knobs yielded 14%, pouring water 13%, and in-hand regrasping or flipping objects 11%. Grasping a tool to manipulate another object succeeded in 137 out of 4,800 episodes (2.9%). Precision alignment tasks, such as inserting pins, stacking, and aligning, succeeded in 107 out of 4,800 episodes (2.2%). After controlling for the number of demonstrations and expert execution time, the four categories of finger actuation, in-hand manipulation, tool use, and precision alignment remained 20 to 25 percentage points lower than pick-and-place tasks. The number of demonstrations for precision alignment tasks was comparable to that of pick-and-place tasks (medians of 413 versus 392, respectively).

In the Transfer-16 zero-shot transfer tasks, the success rates of all trained models did not exceed 0.9%. Out of 6,232 valid trials, there were only 13 successes, and the success rate for flexible object tasks was zero across the board. Because the success rates were too low for meaningful ranking, the paper adopted an 'ordered progress' metric, which scores based on how many sequential stages a model completes. Under this metric, OpenWAM ranked first at 12.9%, outperforming HY-Embodied, LingBot-VLA, and Spatial Forcing by 6.0, 3.7, and 2.6 percentage points, respectively, with paired statistical confidence intervals excluding zero. The remaining four models trailed OpenWAM by no more than 1.7 percentage points. Even on tasks that were nearly impossible to complete, this evaluation framework effectively differentiated model performance.

Pinpointing the failure step

HandsOn embeds stage checkpoints within each task to identify exactly where a model fails. Averaged across the eight models, 21.4% of episodes succeeded, 13.9% terminated early, and 64.8% timed out. Most timed-out episodes devolved into stagnation: in 65% to 83% of these cases, the model made no further progress after exceeding the expert execution time, even though the hands continued to make small, repetitive movements. The paper found that the object-pickup stage incurred relatively few failures. Instead, the majority of failures were concentrated in three specific phases of multi-finger bimanual control, with composite tasks introducing an additional failure category.

Premature triggering upon grasping. The fingers used to grasp a tool often rest directly on buttons or triggers, meaning that simply closing the hand can inadvertently activate the function. The 'click mouse' task requires moving the mouse onto a pad before clicking; out of 800 episodes, 443 terminated immediately due to premature clicking, with OpenWAM failing 16 times and Spatial Forcing failing 78 times. In the 'take photo of a marker' task, 774 out of 778 episodes involved the first shutter press occurring before the camera was properly aimed.

Hands reach their targets individually but fail to coordinate. The 'dual knobs' task requires both knobs to reach the target angle simultaneously. Across 800 episodes, the left and right knobs individually reached their target angles 411 and 416 times, respectively. However, they were in position simultaneously only 38 times, resulting in just 5 ultimate successes. After the first knob reached its target, it rotated out of the acceptable range in 570 out of 608 episodes, with 505 of those instances caused by over-rotation.

Reaching the target, but failing to hold it. The "open laptop" task requires opening the screen to a specified angle. Spatial Forcing and OpenWAM reached the target position at similar rates (73 and 71 times out of 100 episodes, respectively), but by the end of the episode, the screen remained in position only 23 and 67 times, respectively. In almost all failures, the hand was still pressing on the screen, pushing it past the target. In the "pass the block" task, out of 367 episodes where both hands grasped the block, the receiving hand successfully took it only 188 times; in the 179 unsuccessful attempts, the block dropped 175 times.

Repeating or anticipating steps. The composite task "cover the kettle and the box" requires covering the kettle first, yet in 400 episodes, the box lid was moved first 340 times. In "place the fruit and take a photo," the shutter was pressed before the fruit entered the bin in 311 out of 399 episodes.

Language understanding also has flaws. The paper conducted offline tests on 314 initial frames: when the name of the target ball in the instruction was changed, HY-Embodied and LingBot-VLA still reached for the original ball in 57% and 71% of cases, respectively, while DM0.5 did so in 10% of cases.

Can more data or a larger model bridge the gap?

Can these shortcomings be overcome with more data? The team first conducted a data scaling experiment using π0.5, training on 10, 50, 100, 200, and all demonstrations across two tasks. For both tasks, more demonstrations led to higher success rates: "put the ball in the bowl" increased from 7% to 83%, and "pinch the scissors" from 1% to 44%. Increasing the data from 200 demonstrations to the full dataset further improved performance on the two tasks by 18 and 26 percentage points, respectively.

However, when evaluated across all 50 tasks, the number of demonstrations had limited explanatory power for success rates; the time an expert takes to complete a task is a better predictor of difficulty. With 500 demonstrations each, the average success rate was 2.8% for precision manipulation tasks and 37.8% for pick-and-place tasks. The paper emphasizes that these two curves cannot be used to infer cross-task scaling laws.

The team also tested an alternative technical approach: without using HandsOn data or fine-tuning, they fed three camera views to GPT-6 to directly output hand poses and up to 40 finger joint targets, controlling the hands step by step. It had no pre-built grasping action library, no collision avoidance planner, and no access to object poses or tactile feedback.

The results showed 26 successes out of 500 trials (5.2%) on Core-50, and 3 successes out of 160 trials (1.9%) on Transfer-16. It succeeded in 12 out of 66 tasks, with the best performance on "spin the propeller," achieving 7 successes in 10 trials. By category, the success rates were 7.6% for basic manipulation, 1.1% for functional manipulation, and 7.3% for contact-rich manipulation; the latter's success came entirely from "spin the propeller" and "lift the cardboard box," and the paper notes this does not indicate proficiency in precision manipulation. Out of 660 trials, 260 (39.4%) ended due to exhausting the maximum number of API calls, and 150 (22.7%) hit the physical time limit.

The 'operation notes' generated by the model demonstrate a degree of understanding of task logic. For the towel-folding task, it plans to lift the fabric slightly first, 'confirming the edge moves along before executing the first forward fold.' In the composite task of 'opening the laptop and then clicking the mouse,' it instructs itself, 'neither object has moved yet, so do not raise the hand or click.' Both attempts failed. The authors suggest these notes merit further investigation, specifically using large models for high-level planning combined with object state estimation and reusable low-level skills. However, the current experiments cannot isolate how much of the failure originates from the interface itself.

Boundaries, Comparative Analysis, and Significance

We first outline the boundaries of this research.

First, all quantitative results are derived from simulation. The appendix includes real-robot execution videos to illustrate sim-to-real transfer, but these serve only as qualitative demonstrations and do not report real-robot success rates.

Second, although the platform supports various hardware configurations, all experiments in the paper utilized a single setup: a ROKAE AR5-08 robotic arm (7 degrees of freedom, 5 kg rated payload, 683 mm reach) paired with the first-generation Wuji dexterous hand, featuring 20 active joints per hand. The physics backend is PhysX across the board, and no comparisons between different dexterous hands have been reported yet.

Third, the model relies solely on vision and proprioception to detect contact, as it lacks tactile sensing. The paper also notes that the wrist-mounted camera is easily occluded once an object is grasped. Fourth, GPT-6 was evaluated with only 10 trials per task. The authors use it as a baseline reference, explicitly noting that its input and call budgets differ from those of the trained models.

The authors' future plans include integrating tactile sensing, conducting more quantitative physical evaluations, tackling longer-horizon tasks, and developing more capable large-model agent frameworks.

In the context of related work, the paper's comparison table evaluates HandsOn alongside DexJoCo, DexVerse, Bench2Dex, DexMimicGen, and SPD, noting that each features a distinct task catalog and evaluation scope. According to the table, HandsOn includes the largest number of human demonstrations at 18,000; DexMimicGen has only 60 human demonstrations, supplemented by auto-generated data to reach 21,000. In terms of task coverage, DexVerse's 100 tasks exceed the 66 in HandsOn. HandsOn is the only benchmark in the table that reports zero-shot tasks (16 tasks), whereas the other five do not. It also evaluates a wider variety of methods: in addition to diffusion policies, it tests five VLAs, three world-action models, and GPT-6.

Other benchmarks also possess capabilities that HandsOn currently lacks. According to their respective paper titles, Bench2Dex evaluates bimanual manipulation combining vision and tactile sensing across multiple dexterous hands, while DexVerse is a modular, multi-task, and multi-embodiment benchmark. HandsOn currently lacks tactile feedback, and its paper experiments utilized only a single robot arm and dexterous hand combination. Regarding real-robot evaluation, the table shows that DexJoCo, DexVerse, and Bench2Dex did not report real-robot evaluations; SPD performed fine-tuning on real-world tasks, whereas HandsOn only provided qualitative videos.

The AI news aggregator AI Weekly noted in its summary that for those betting on general-purpose dexterous robots, this serves as a sobering report card. As of press time, no public commentary from academic peers or industry stakeholders regarding this work has been observed.

The impact of this work can be evaluated from two perspectives: research and industry.

For researchers, HandsOn provides a directly comparable set of tasks, open-source demonstration data, and evaluation code. Its evaluation checkpoints can break down a "low success rate" to identify the exact step where a policy fails, making it suitable for verifying whether a new method actually improves contact, coordination, or goal maintenance. The paper's data also points to specific directions: tool use, precise alignment, bimanual handover, and transfer to novel tasks are the weakest links in current methods. Simply scaling up the volume of demonstrations may not bridge these gaps.

For companies developing dexterous hands and embodied models, this platform supports multiple robot arms and dexterous hands, providing the conditions to compare different hardware and models on the same set of tasks in the future. Based on the paper's current data, success rates for tasks such as pushing, opening and closing, and pick-and-place are significantly higher than those for tool use and precise assembly. Under the testing conditions of the paper, "general-purpose dexterous manipulation" remains some distance from stable, real-world usability.

Synthesizing these limitations and comparisons, if more teams submit results on HandsOn in the future and supplement the benchmark with quantitative evaluations on real robots and across different hardware, it has the potential to become a standard public evaluation suite in the field of dexterous manipulation.