Peking University Team Succeeds in 27 of 30 Real-Robot Dexterous Task Tests by Optimizing Hand-Object Configurations

A joint team from Peking University, PrimeBot, and the Beijing Institute for General AI has introduced DexHOT, a framework that optimizes hand-object configurations rather than just object poses, achieving a 90% success rate in real-robot dexterous manipulation tasks.

The most intractable challenge in dexterous manipulation lies in determining the exact posture in which the hand should contact an object. Even when manipulating the same cup, grasping the rim, holding the handle, or supporting the body can lead to vastly different outcomes in operational stability and the ability to seamlessly transition to subsequent actions, despite the final pose of the object remaining identical.

To address this challenge, on November 3, a joint team from the Frontier Computing Research Center at the School of Computer Science and the National Key Laboratory of Multimedia Information Processing at Peking University, alongside PrimeBot and the Beijing Academy for General Artificial Intelligence, published the paper "DexHOT: Learning Hand–Object Configuration Transfer for Dexterous Manipulation." The study introduces DexHOT, a dexterous manipulation framework that incorporates the relative relationship between the hand and the object into its optimization objective, shifting the learning target from a single object state to a paired hand-object configuration.

The paper has seven authors. Zhewei Gui and Anthony Bolton are co-first authors, and Jiyao Zhang is the project lead. Dong Hao, a tenured associate professor at the Frontier Computing Research Center of Peking University's School of Computer Science and Chief Scientist at PrimeBot, serves as the corresponding author.

The team's research approach breaks away from the traditional paradigm: rather than optimizing the object pose in isolation, it starts directly from the hand-object configuration itself. The method attempts to learn reusable local transfer capabilities using only synthetic configuration pairs, which are then stitched together to execute long-horizon operations.

The implementation of DexHOT is divided into four core modules:

• Configuration representation and distance metric: The hand-object configuration is defined as the finger joint angles plus the object pose relative to the wrist. The distance metric uses the average displacement of 28 hand keypoints in the object coordinate system, unifying two heterogeneous types of information into a millimeter scale.
• Shared policy training: Adjacent targets are selected from a feasible configuration pool comprising 40 objects to train a universal recurrent policy across objects. The action space is expanded through exploration across 64 populations.
• Long-distance greedy planning: When executing long-distance manipulations, a greedy planner selects intermediate transition configurations. Subgoals are updated at 1 Hz, while the hand policy tracks and executes them at 30 Hz.
• Demonstration-guided integration: When integrating human demonstration scenarios, the configuration aligned with the demonstration is used directly as the subgoal.

On the self-built DexHOTBench benchmark, the policy achieved an average success rate of 80.5% across 40 training objects and 69.4% on 80 unseen held-out objects. In real-robot tests, the configuration transfer success rate reached 61.5% over 200 trials, and the system completed 27 out of 30 tests across six demonstration-guided tasks.

Whether this solution can reliably support longer operational sequences remains subject to two key questions that need to be verified:

First, the effective upper limit of the learned short-range transfer capability;

Second, how to properly divide responsibilities among the hand policy, external planner, and human demonstrations to prevent the modules from constraining one another.

Hand placement directly integrated into the objective

DexHOT redefines the optimization objective for dexterous manipulation through a four-step process: first, formally representing the relative placement state between the hand and the object; second, defining a distance metric between configurations; third, incorporating this distance into the policy observations; and finally, stitching short-range local transfers into long-range operations via a sequence of subgoals.

Step 1: Define the hand-object configuration. Let W be the wrist coordinate frame and O be the object coordinate frame. A single configuration is defined by the 22-dimensional finger joint angles combined with the 6D pose of the object relative to the wrist frame. The target configuration simultaneously constrains the finger posture along with the position and orientation of the object relative to the wrist.

Step 2: Design the configuration distance metric. The team maps each configuration to 28 hand keypoints in the object coordinate frame, comprising a palm reference point, 22 joint origins, and 5 fingertip points. The distance between two configurations is calculated as the average displacement of these keypoints, measured in millimeters. This metric captures only the relative relationship between the hand and the object, remaining invariant to the translation and rotation of the entire hand-object system in the world coordinate frame.

This is the core reason the method works: finger joint angles and the 6D object pose are heterogeneous quantities that cannot be directly compared. By mapping both to the same 3D space via the 28 keypoints, the policy gains a unified metric to measure the progress of short-range adjustments.

These keypoints serve purely as kinematic references and do not strictly correspond to actual contact points. A configuration is deemed feasible only if it simultaneously satisfies three constraints: it can stably support the object, it avoids interpenetration, and it lies within a reachable workspace that facilitates subsequent manipulation.

Step 3: Build a conditional policy for configuration tracking. The policy takes the current configuration, proprioceptive state, a coarse description of the object shape, and the subgoal as inputs, and outputs a 22-dimensional hand action. It also reserves six wrist control channels, which were not enabled in the experiments. Actions are added to the joint position commands at a fixed step size and clipped to the joint limits. A single transfer is deemed successful if the configuration distance converges to within 20 mm and is maintained for 10 consecutive control steps, with no object dropouts throughout the process.

Step 4: Chain multiple short transfers. Given a distant goal, a greedy planner continuously selects candidate configurations from the configuration pool. The candidates must be within 50 mm of the current configuration and closer to the final goal. The selected configuration serves as the subgoal, updated at 1 Hz, while the hand policy tracks it in a closed loop at 30 Hz. If the target cannot be reached directly, the planner expands its search range to find new transition nodes.

This hierarchical scheme only guarantees that the initial and final transition configurations are feasible; it does not assume that every intermediate path is dynamically smooth. Mid-manipulation finger adjustments, finger swaps, and regraspings are left to the policy to generate autonomously in the closed loop. The planner is responsible for decomposing long-horizon tasks, while the policy handles local motion execution. By dividing these responsibilities, the system avoids forcing a single policy to directly learn highly complex, long-horizon manipulation.

Once the goal paradigm is established, the training phase presents the real challenge: a single policy must simultaneously learn configuration transfer capabilities across 40 objects and hundreds of thousands of initial state–goal pairs.

Configuration pool construction and population exploration training

The entire training pipeline is also divided into four steps: first, collect all feasible configurations to build the configuration pool; second, select nearby goals from the pool to learn short-distance configuration transfers; third, integrate the observation, reward, and action interfaces; and finally, leverage a population mechanism to broaden the policy's behavioral coverage.

Step 1: Build the synthetic configuration pool. The team selected one object from each of the 40 categories in Omni6DPose. Using BODex, they sampled initial seeds on the object surfaces and simultaneously optimized whole-hand and fingertip grasps for various object orientations, generating three types of poses: pre-grasp, grasp, and squeeze. All candidate configurations were fed into Isaac Gym and validated against the physical parameters and collision geometry used in training. Samples exhibiting interpenetration or an inability to stably support the object were discarded. This process ultimately yielded 825,000 configurations, which were partitioned into independent training pools for each object.

Step 2: Sample local training goals. A neighborhood is precomputed for each configuration anchor, retaining only goals within a 20 to 40 mm distance range. Samples closer than 20 mm are discarded outright, as this distance already falls within the evaluation success tolerance. A training round begins by randomly selecting an anchor configuration from the pool and sampling a goal within its precomputed neighborhood. After completing a transfer, this goal automatically becomes the new anchor, while the system avoids repeatedly selecting recent goals. If the object drops or a timeout occurs, the episode resets. The entire neighborhood is precomputed offline, eliminating the need for real-time retrieval during training.

Step 3: Aligning the observation and action interfaces. The target is encoded using joint angles combined with the residuals of 28 keypoints. The object shape is represented by a 64-dimensional base point set, supplemented by four bounding box keypoints expressed in both the current and target wrist coordinate frames. The reward function comprises three components: a shaping reward that increases as the keypoint distance decreases, a one-time completion reward of 100 points upon reaching the target, and an action magnitude penalty with a coefficient of 0.02 based on the sum of the absolute values of the actions. In the later stages of training, an additional penalty for excessive bending of the distal joints is introduced to constrain the finger posture toward the target configuration.

Step 4: Population-based exploration. A standalone PPO policy tends to converge to a limited number of grasp transition modes when faced with a massive number of initial state-target pairs. To address this, the team employs EPO: 64 exploration units are set up to share the same network weights, with each unit carrying a 32-dimensional learnable latent variable. During the genetic update, the best-performing elite units are retained, while the latent codes of the remaining units are generated through elite crossover and Gaussian mutation. Fitness is evaluated based on the number of transitions completed per control step, which is independent of the return used for the policy gradient.

The training configuration is scaled as follows: the policy network consists of a single 1024-dimensional LSTM layer stacked with four MLP layers. The value network adopts an asymmetric architecture, granting it access to noise-free joint, fingertip, and object velocities, as well as contact information. Both the policy and value networks receive the latent variables from the exploration units. The total number of parallel environments reaches 524,000, with a physics simulation frequency of 60 Hz and a control frequency of 30 Hz. The domain randomization scope includes object mass, size, friction, and restitution coefficients, as well as hand friction, link mass, joint stiffness, and damping. Additionally, random external forces, observation noise, and 0–2 step control delays are introduced, while the gravity coefficient is progressively annealed from 0.2 to 1.

The core of this configuration lies in population-based exploration. Even with hundreds of thousands of parallel environments enabled, a single PPO policy still tends to be confined to a few grasp transition strategies. Only by relying on 64 exploration units coupled with genetic updates can the behavioral space be sufficiently expanded. During deployment, only a single unit is retained: the team selects the unit with the highest empirical success rate among all 64, and simply freezes the shared weights along with its corresponding latent variable.

After the policy training is complete, it still needs to be verified that it has truly learned general hand-object configuration rules and can be reused on unfamiliar objects.

120 objects to validate generalization capabilities

The team's validation is divided into three parts: building an evaluation benchmark, testing the variation of success rates with respect to distance and objects in simulation, and then conducting ablation studies to verify the necessity of each training component one by one.

The benchmark is constructed by combining the category division of Omni6DPose with the novelty rules of UniDexGrasp, comprising a total of 120 objects: 40 are used for training, 40 are novel instances unseen within the training categories, and the remaining 40 come from entirely unfamiliar new categories. The reserved 80 objects are completely excluded from training throughout the process; each is independently generated and its configuration pool is verified following the same pipeline. The 120 configuration pools collectively contain 1.18 million configurations, of which the reserved objects account for 359,000. Object sizes are constrained: the diameter of the convex hull bounding sphere must not exceed 300 mm, and the minimum convex hull thickness must be no less than 10 mm, to prevent overly thin meshes from causing mesh penetration and slowing down contact computations.

The evaluation samples are divided into three tiers based on the transition distance: the adjacent tier of 20–40 mm, which corresponds to the sampling interval during training; the close tier of 20–70 mm, which exceeds the training distribution; and the far tier of over 70 mm. For each object and each tier, 1,000 hand-object configuration pairs are sampled, with each pair tested exactly once. The evaluation uniformly employs a greedy planner, allocating a quota of 20 target updates and replanning on schedule; network configurations, latent variables, and planning parameters remain constant across all objects.

The model performs exceptionally well on the 40 training objects: the success rates are 90.9% for the adjacent tier, 85.5% for the close tier, and 75.6% for the far tier, with an overall average of 80.5%. For novel instances within the same categories, the success rates across the three tiers are 88.3%, 81.4%, and 67.3%, respectively, averaging 74.4%. When facing entirely novel categories, the rates are 81.5%, 72.2%, and 56.7%, averaging 64.4%.

The most critical takeaway from these results is the magnitude of performance degradation: within the same distance tier, switching from training objects to entirely novel categories results in a drop of only 9.4 percentage points in the adjacent tier, compared to an 18.9 percentage point drop in the far tier. The most challenging scenario is precisely the combination of novel categories and large-scale configuration shifts.

The ablation study is conducted on the 40 training objects using the same evaluation protocol. When EPO is removed and replaced with standard PPO, the success rate drops to just 11.5% in the adjacent tier and 2.5% in the close tier, with the far-tier tasks failing completely, yielding an overall average of only 1.3%. If neighborhood sampling is abandoned in favor of uniformly random target selection across the entire configuration pool, performance declines across all tiers, and the average success rate falls to 68.2%, indicating that merely observing distant targets does not mean the model has acquired long-distance transition capabilities. Replacing the fixed configurations in the library with real-time current configurations as the sampling anchors maintains an 86.0% success rate in the adjacent tier, but the far-tier rate drops to 52.8%, with an average of 64.0%.

The distance-tiered evaluation design disentangles two questions: first, whether the model can stably complete configuration transitions within the training distribution, and second, whether the learned local transition capabilities can generalize outward. Currently, there are no readily available baseline methods to compare against this benchmark; the core supporting evidence comes from the cross-object generalization results and the individual ablation studies.

The simulation results demonstrate the feasibility of this paradigm in virtual environments, but how much performance is retained on physical hardware requires further validation through real-robot experiments.

Two hundred transitions, six demonstration tasks

The team conducted validation on the physical robot in two rounds: the first round tested configuration transitions in isolation with a fixed wrist; the second round used the same policy, driven by subgoals derived from human demonstrations, to complete a full suite of household manipulation tasks.

The hardware platform uses a 22-DOF Sharpa Wave dexterous hand mounted on a 7-DOF Franka Emika Panda robotic arm. The current hand-object configuration is reconstructed using joint encoders combined with FoundationPose, with visual input from a third-person Intel RealSense D435i camera. The policy runs at 30 Hz without any real-robot fine-tuning throughout the process. For the transfer experiment, the arm is fixed in a standard palm-down posture, allowing only the fingers to adjust the configuration, thereby isolating and evaluating the manipulation capability of the dexterous hand itself.

The transfer test selects five seen and five unseen objects. For each object, 20 sets of random initial states and targets are sampled from the configuration pool, totaling 200 tests with a single attempt per set. The experimental procedure first reproduces the initial configuration and then transfers to the target configuration. The success criteria follow the simulation standard: the configuration distance must drop within 20 mm and be maintained continuously for 10 control steps without the object dropping. Additionally, a progress metric is tracked to measure the proportion of the expected distance reduction that is ultimately and stably achieved.

The overall success rate across the 200 real-robot transfers is 61.5%, with an average progress of 0.735. Performance on seen and unseen objects shows little difference, at 63% and 60%, respectively. Broken down by object: the tennis ball achieves 95%, the pill bottle 90%, the tape measure and banana both reach 80%, and the mouse and flashlight hit 70%. The shampoo bottle and face cream fall to 45%, while the hammer and screwdriver achieve only 20%.

Failures are concentrated on elongated tools, which the team attributes to four causes:

• Inability to recover from slipping: Once a low-friction object slips and begins to spin, the closed-loop controller cannot correct it in time.
• Initial configuration reproduction deviation: Grasps achieved through interpolation and squeezing are inherently marginally stable, leading to deviations right at the real-robot starting point.
• Mid-motion blockage: This stems from mismatched friction characteristics between simulation and reality, lateral finger motion deviations, or inconsistent collision geometry.
• Loss of object tracking: FoundationPose occasionally loses its lock, preventing the acquisition of a valid current configuration estimate.

The second round focuses on long-horizon tasks. Six tasks reuse the same frozen policy, with each task paired with one redirected demonstration and five fixed initial states. During execution, the system first follows the demonstration trajectory to reach a designated pre-grasp frame. Hand control is then handed over to the policy, while the wrist continues to follow the demonstration trajectory via an independent controller. From that point on, configuration subgoals are extracted from the demonstration sequence at 1 Hz, and the hand policy maintains closed-loop tracking at 30 Hz.

The six tasks are barcode scanning, shelving, block alignment, salt shaker shaking, water pouring, and screw driving. Barcode scanning requires rotating the bottle 180°, while ensuring the barcode is exposed and aligned with the scanner. Shelving also requires a 180° rotation, along with switching from a pinch grasp to a power grasp. Water pouring requires righting an inverted cup. Screw driving requires pointing the drill bit downward for alignment.

Final results: Out of 30 test runs, 27 successfully completed all three stages, yielding a 90% task completion rate. Shelving, block alignment, salt shaker shaking, and water pouring succeeded in all five attempts each. Barcode scanning succeeded in 4 out of 5 runs, and screw driving in 3 out of 5. As a baseline for comparison, a method using only kinematic redirection and simple trajectory playback managed to grasp the object in 24 out of 30 runs, but the subsequent hand configuration sequence completely failed to keep up, resulting in a 0 out of 30 full-stage pass rate. A version of SimToolReal modified for traditional object targets achieved a grasp success rate of 21 out of 30 and an object pose target achievement rate of 9 out of 30. It satisfied the hand configuration constraints simultaneously in only two block alignment attempts, resulting in a full-stage pass rate of just 2 out of 30.

The most critical design of this evaluation is its three-stage scoring system, which assesses grasping, object pose accuracy, and hand-object configuration accuracy as separate metrics. Even if the object pose is correct, the trial is marked as a failure if the palm occludes the barcode. This explicitly quantifies the fundamental distinction between an "object pose objective" and a "hand-object configuration objective."

DexHOT shifts the optimization objective of dexterous manipulation from isolated object states to explicit hand-object pairings: it synthesizes feasible configurations to learn reusable short-horizon transitions, then stitches these into long-horizon, multi-step operations using subgoals provided by a planner or human demonstrations. Generalization success rates of 80.5% and 69.4% in simulation, a 61.5% success rate in real-robot transfer, and a 90% completion rate in demonstration tasks collectively demonstrate that explicit hand-object configuration is a practical, decomposable representation paradigm that can incorporate external guidance.

The team's core conclusion centers on the configuration interface layer: it simultaneously constrains object placement and hand pose, enabling regrasping and contact transitions in long-horizon tasks to be decomposed into sequential subgoals. The remaining performance bottlenecks on the physical robot involve slender tools, low-friction contact surfaces, and visual tracking loss, highlighting the need to enhance dynamics modeling and perception capabilities in future work.