OmniDex Dataset Scales Dexterous Grasping to 2.6 Million Cluttered Scenes

By pairing a reusable seed-grasping library with collision filtering, OmniDex expands the scale of dexterous-hand grasping datasets in cluttered scenes from thousands to 2.6 million scenarios and approximately 400 million grasp annotations. The accompanying generative model, which combines soft-winner-takes-all learning with physical constraints, eliminates the need for iterative pose optimization during inference.

On October 8, a team comprising ACE Robotics, CUHK Multimedia Laboratory, InnoHK Centre for Perceptual and Interactive Intelligence, Tongji University, Shanghai Jiao Tong University, Zhejiang University, and Nanyang Technological University published the paper "OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes." The work features 10 authors; Xiaogang Wang, professor at The Chinese University of Hong Kong and co-founder of SenseTime, is listed as an author. Naiyu Fang and Zhongjin Luo are co-first authors, while Zhongjin Luo and Siyuan Huang serve as project leads. Hongsheng Li of CUHK Multimedia Laboratory is the corresponding author.

Dexterous-hand grasping is a foundational action in embodied AI, with all complex manipulations relying on stable grasps. Collecting high-degree-of-freedom grasp poses on real robots is costly and difficult to scale for mass production, leading the industry to widely adopt simulated synthetic data.

However, cluttered scenes most closely resemble real-world deployment environments, yet existing solutions have long been constrained by three bottlenecks.

First, inputs often rely on depth point clouds, causing models to focus solely on object geometry while losing semantic information. This makes it difficult to distinguish the target object when facing different items with similar appearances.

Second, traditional grasp annotation requires scene-by-scene optimization. Optimizing just 500 initial solutions for a single object takes approximately 15 minutes on an A800 GPU, which has kept public benchmarks for cluttered scenes limited to a few thousand scenarios.

Third, previous benchmarks rarely cover support surfaces or placement configurations beyond tabletops, excluding real-world operational scenarios such as shelves, cardboard boxes, and sorting stations.

OmniDex breaks through these limitations from both data and algorithmic perspectives.

On the data side, the team abandoned per-scene optimization in favor of offline precomputation: they first built a reusable seed-grasp library for each object in empty environments, then placed objects into cluttered scenes to perform collision filtering, ultimately constructing a dataset of 2.6 million scenes with approximately 400 million grasp annotations.

On the algorithm side, the team trained an end-to-end grasp generation model that requires no post-optimization. By employing a soft-winner-take-all learning objective, it retains multiple feasible grasp solutions, while introducing physical constraints during training to achieve millimeter-level positioning accuracy. Inference uses parallel sampling combined with physical scoring. The entire approach attempts to balance grasp candidate diversity with millimeter-level operational precision in cluttered environments.

01 Gathering Seeds in Empty Spaces, Filtering Through Clutter — Data Module Details

The core breakthrough of OmniDex lies first in its data system construction.

In real-world operational scenarios, desktop objects are densely stacked and closely adjacent. Dexterous hand grasping must simultaneously avoid surrounding items and support structures, demanding extremely high spatial adaptability. However, labeling costs for traditional cluttered-scene grasp data are prohibitively high: because object arrangements differ completely across scenes, per-scene optimization and solving are required to generate valid grasp poses. The paper provides explicit computational cost metrics: optimizing just 500 initial solution sets for a single object requires running on an A800 GPU for approximately 15 minutes; since a single scene often contains more than ten objects, the overall computational cost is entirely unviable for large-scale deployment. This is also the core reason why public cluttered-scene grasp benchmarks have remained stuck at small scales of only a few thousand scenes.

The team’s insistence on building large-scale data stems from the need to align with the spatial interaction logic of cluttered environments.

Single-object aerial grasping requires attention only to the object’s own morphological features, whereas in dense, cluttered scenes, objects occlude and support one another. The robotic hand must precisely learn spatial strategies for avoidance and reaching. Such complex spatial relationships cannot be covered by a small number of samples; massive scene data is essential for model generalization learning. This is the key reason OmniDex targets data scaling as its core breakthrough point.

To break through the computational bottleneck of per-scene optimization, the team innovatively adopted an 'offline pre-optimization + scene reuse' strategy.

The researchers first optimized and generated 4,000 candidate grasp poses for each object in unobstructed, open scenes. After preliminary screening via force closure, they conducted lift tests in a simulation environment, retaining only stable grasps to build a globally reusable seed grasp library for objects. This process ensured that each object possessed multiple directional, highly adaptable standard grasp poses, which could be directly reused in any cluttered scene, completely abandoning the inefficient traditional mode of per-scene repetitive optimization. This is the core prerequisite enabling the model to support a scale of 2.6 million large scenes.

After establishing the seed library, the team proceeded with large-scale scene generation and rigorous screening.

Using the Isaac Gym simulation platform, the team dropped various objects from random heights onto different support surfaces. Once the object poses stabilized, the pre-stored seed grasp poses were mapped and adapted to the objects' real-time positions to generate scene-specific grasp candidates. The team applied strict triple-screening criteria: the robotic hand must precisely contact the target object, avoid touching the support surface, and prevent collision interference with surrounding objects. The screening approach was conservative, retaining only direct-grasp valid poses and excluding complex schemes requiring pushing or clearing surrounding objects, ensuring clean and effective samples.

Thanks to prior offline optimization, this stage required only coordinate transformation and collision detection, resulting in significantly lower computational overhead compared to traditional per-scene optimization workflows. Additionally, the team added a cross-modal consistency verification step. To address issues such as mesh discrepancies between physical and visual models and simulation convergence errors, they replayed all generated scenes using the visual model to eliminate invalid samples where geometric labels misaligned with visual frames, ensuring complete alignment of spatial logic and modal information across all data.

To cover diverse real-world operational environments, the team customized differentiated generation rules for different application scenarios, constructing five distinct scene subsets.

• The standard desktop and mixed subsets both use random placement on desktops, with objects landing randomly, to represent everyday tabletop operations;

• Box subset: Simulated parcel sorting scenarios, achieving dense stacking of objects within confined containers;

• The grid subset adapts to warehouse sorting scenarios, with similar objects scattered uniformly on a table and only stable, upright object poses retained;

• The shelf subset simulates retail warehousing scenarios, with objects neatly stacked by level.

The unified asset library, combined with differentiated generation rules, creates cluttered samples varying in form, scenario, and difficulty, enabling precise, scenario-specific quantification of model performance during subsequent evaluations.

The research team performed fine-grained selection and optimization of the dataset's objects and support assets. Object materials for grasping were sourced from three public datasets: OmniObject3D, ShapeNetCore v2, and Google Scanned Objects. After three rounds of filtering—semantic category screening, sample balancing to remove long-tail distributions, and removal of incomplete meshes and textures—more than 6,000 high-quality object instances suitable for dexterous hand grasping were retained. These cover diverse daily items such as mobile phones, toys, household goods, and shoes, all manually verified for adaptability.

Meanwhile, each object is equipped with two high-precision aligned models: a physics simulation model built based on manifold normalization and convex decomposition, and a high-fidelity visual rendering model. Both models share a unified normalization standard, completely avoiding spatial misalignment issues. Support surface assets also break through the limitation of a single tabletop, encompassing 244 tables, 79 open-top boxes, and 21 sets of shelves, comprehensively covering mainstream operational carriers such as tables, sorting bins, and warehouse shelves, thereby significantly expanding the boundaries of scenario adaptability.

Compared to previous industry benchmarks, OmniDex achieves a magnitude-level breakthrough.

Traditional single-object grasping datasets have sufficient samples but lack cluttered interaction scenarios; mainstream cluttered scenario benchmarks are all in the thousands scale, with DDGC having only about 400 scenarios, ClutterDexGrasp about 1,000 scenarios, and DexGraspNet 2.0 only 8,000 scenarios, all supporting only a single tabletop scenario. OmniDex scales up the scenario size to 2.6 million, expands effective object samples to over 6,000, covers support surfaces including tables, boxes, and shelves, upgrades observation dimensions from single-depth to RGB-D multimodal, and achieves comprehensive upgrades in data scale, scenario richness, and modality completeness.

After scene construction, the team established a standardized multimodal input observation protocol.

Using Blender to replay all geometric information from cluttered scenes, they unified rendering outputs of RGB images, depth maps, semantic segmentation maps, and camera parameters. Combined with scene semantic labels to generate target object masks, these served as instruction bases for the model's "specified grasp target." The camera setup employed a "1 primary + 4 auxiliary" multi-view layout, dynamically adapted to scene types: desk scenes used conventional oblique angles to avoid large-scale occlusion; box scenes tightened the field of view to accommodate densely stacked boundaries; shelf scenes adopted layered frontal views to eliminate vertical inter-layer occlusion. This standardized multi-view layout completely resolved the shortcomings of traditional solutions that relied solely on depth point clouds lacking semantic information, making semantic recognition and target localization core capabilities of the data baseline.

This multimodal observation system redefined the input standards for cluttered grasping tasks. RGB images provided semantic features and object identity; depth maps precisely depicted geometric forms; target masks clarified the grasping subject; and camera parameters anchored global spatial coordinates, ensuring complementary alignment of multimodal information. Particularly for stacked objects with highly similar appearances, semantic information enabled precise distinction of grasp targets, addressing the core pain point of pure geometric recognition.

The overall dataset was divided into five subsets by scene type, covering scales from hundreds of thousands to over a million scenarios, with clear division of labor adapting to different evaluation dimensions. The standard desk subset contained 1.14 million scenarios, featuring fixed support surfaces and interchangeable target objects, used to test the model's cross-object transfer capability. The mixed subset included 940,000 scenarios, simultaneously replacing objects and support environments to simulate irregular complex operational scenes. The box subset (140,000 scenarios), grid subset (280,000 scenarios), and shelf subset (140,000 scenarios) respectively targeted three industrial application scenarios: package sorting, warehouse sorting, and shelf retail.

All subsets were divided into dedicated training sets and two types of test sets, corresponding to spatial reasoning capabilities for "seen objects in new layouts" and zero-shot generalization capabilities for "unknown objects in new layouts." The evaluation system set dual-dimension core metrics: grasp success rate was categorized into three tiers based on gravity direction, using real-scenario gravity success rate as the primary metric, supplemented by dual standards for stable grasping in all directions and usable grasping in single directions; collision avoidance rate was used to verify the compliance of the model's grasping strategy, quantifying the dexterous hand's ability to avoid support surfaces and non-target obstacles, comprehensively measuring dexterous grasping performance in cluttered scenes.

02 Retaining Multiple Possibilities for Grasping

With data expanded, the model layer needed to handle two mutually balancing objectives simultaneously.

Under observation of the same scene, a single object typically has multiple valid grasp poses, which is the inherent one-to-many characteristic of grasping tasks; however, if the dexterous hand's position deviates by just 1 mm, the contact state may fail, imposing strict requirements on positioning accuracy. Traditional solutions either relied on intermediate representations like affordance maps, compressing multiple feasible schemes into a single heatmap and breaking the end-to-end link; or iteratively optimized during inference to supplement precision, bringing significant latency. Both approaches had shortcomings.

OmniDex uses a diffusion model as its generative backbone, progressively denoising from noise to recover grasp poses conditioned on scene observations.

The model first constructs condition inputs by fusing multimodal information: RGB images extract semantic features via DINOv2, depth maps extract geometric features through a lightweight adapter, and target masks are encoded separately. These three feature streams are fused to form the scene representation. To unify the global coordinate system for grasp generation, the team encodes camera parameters and table height as parameter tokens, concatenating them to the scene representation afterward. Depth maps are not used directly as input; instead, they are back-projected to obtain 3D coordinate maps, ensuring alignment between geometric information and ground-truth grasps. The team chose depth reprojection because depth carries rigorous spatial geometry, whereas texture information can be interpolated but geometric relationships cannot be approximated.

Multi-view inputs employ a dual-branch fusion strategy: RGB performs early-stage semantic fusion; depth is reprojected to the main view based on camera parameters, with point cloud alignment achieved via hardware-accelerated Z-buffer scattering. Camera pose is encoded separately as a view-relation token. The final condition sequence includes image, depth, mask patch features, and parameter tokens, all fed into the denoising network. Parameter tokens compress the camera projection matrix and table height into vectors, fixing the generation coordinate system so the model perceives the relative positions of the camera and table, allowing output poses to map to real physical space.

To preserve multiple feasible grasp options, the team adopts a soft-winner-take-all learning objective. For a set of valid grasp poses for the same object, identical noise is injected. The training process guides the network to fit this batch of samples, selecting only the best-matching pose rather than averaging across multiple grasping methods. This prevents the generation of physically impossible intermediate fake poses that would result from averaging diverse postures. The soft-winner-take-all objective preserves the one-to-many distribution of grasp tasks, eliminating invalid fake poses caused by average interpolation. A temperature parameter regulates the retention range of candidate modes: low temperatures favor retaining only the single optimal mode, while high temperatures retain multiple approximately feasible solutions simultaneously, allowing the team to balance diversity among multiple candidates with grasp fidelity.

03 Improving Accuracy in Training, Scoring Only in Inference

Accuracy constraints are embedded in the training phase, leveraging a differentiable dexterous hand simulator to add physical constraint losses composed of four loss terms.

• The thumb contact term encourages the thumb to stably adhere to the object;

• The opposing-finger term computes contact between the closest set of fingers; stable grasping typically requires only some of the fingers to participate.

• The closure term constrains the thumb and opposing fingers to approach from opposite directions, forming an antipodal grasp.

• The penetration term uses signed distance fields to limit finger positions, preventing the model from outputting poses where hands penetrate objects.

The computation flow filters depth maps by mask and back-projects them to obtain local point clouds of target objects. Predicted grasps are sent to a simulator to acquire hand-surface contact points, calculating the minimum distance to object point cloud points for each. To prioritize learning grasp morphology before physical rules, the team employs a delayed warm-up strategy: all physical terms are disabled for the first 10 epochs, with weights linearly increasing to 100 from epoch 10 to 20. The four loss terms are assigned differentiated weights—thumb, opposition, penetration, and closure at 5:1:0.2:2—with the thumb receiving the highest weight, reflecting its irreplaceable role in stable grasping.

Inference no longer executes iterative optimization.

The model samples 64 groups of noise in parallel based on the same conditions, decoding 64 candidate grasp poses which are then sent to the simulator for physical scoring. The scoring formula integrates three metrics: kinematic closure stability as the primary term, multi-finger contact rewards encouraging more fingers to participate, and safety penalties that linearly deduct scores for minor interpenetrations and heavily penalize severe collisions or missing thumb opposition, directly filtering physically infeasible candidates to select the highest-scoring solution. By front-loading physical precision constraints into training, inference retains only parallel decoding and candidate scoring, completely eliminating post-optimization steps. Embedding physical constraints into training objectives reshapes the model itself. Inference-side optimization performs local tuning on already generated poses, which can diverge when initial pose deviations are large; integrating physical rules during training biases the network toward outputting physically feasible poses from the start, allowing inference to simply generate candidates in batch and select the best.

The computational cost of the entire scheme concentrates on data generation and model training.

Generating geometric and visual databases using A800 GPUs consumes approximately 5,000 and 17,000 GPU hours, respectively. The model uses the AdamW optimizer with an initial learning rate and weight decay both set to 1e-4, trained for 25 epochs. Validation was performed on the Shadow dexterous hand with 24 degrees of freedom; this device has a motion space far larger than parallel grippers, used to test the model's generation capability in high-dimensional pose spaces. Training and inference uniformly use 64 parallel grasp hypotheses, serving as both the sample set for fitting grasp distributions during training and the candidate pool size during inference, reusing the same parameter for both purposes.

04 Stable Grasping of Unseen Objects

The team retrained all baseline models on the OmniDex training set, avoiding direct reuse of original baseline weights for comparison.

Under standard desktop subset conditions with first-person-view input, OmniDex achieved a grasp success rate of 55.80% and a collision avoidance rate of 84.12%, outperforming DexGraspVLA by 14.81 and 2.68 percentage points, respectively. This success rate includes fingertip force-adjustment techniques; excluding these techniques, the rate drops to 33.60%. All baseline comparisons maintain this consistent statistical scope. When facing novel objects never seen before, OmniDex still achieved a grasp success rate of 59.74%, higher than DexGraspVLA’s 54.97%.

Performance remained stable even on unfamiliar geometries, demonstrating that the model’s capabilities are not limited solely to objects present in the training set. In the same data group, with fingertip force adjustment enabled, DexGraspNet 2.0 achieved success rates of 43.64% for seen objects and 47.60% for unknown objects. Meanwhile, Grasp-as-You-Say, which targets language instructions, recorded only 1.68% and 2.55%, respectively. Such language-driven grasping schemes nearly fail in dense, cluttered scenes, further confirming that crowded environments rely heavily on training data specifically constructed for cluttered scenarios.

Across five subsets of the entire benchmark, two observation perspectives, and evaluations for both "seen" and "unseen" object categories, OmniDex generally maintained leading or near-optimal performance. This forms the basis for the paper’s claim that it is currently the optimal solution for dexterous hand grasping in cluttered scenes.

These conclusions hold true when changing layouts and support surfaces.

Among the five subsets, the grid subset performed best, with a first-person-view grasp success rate of 55.01% and a multi-view rate of 60.37%. The team attributes this stability to structured arrangements providing consistent spatial cues, allowing the model to infer grasp positions based on evenly spaced objects. The box subset proved the most difficult, with success rates just above 30%. Objects inside containers were highly stacked and severely occluded, leaving very limited space for the robotic hand to reach. Multi-view input improved performance across most subsets; for the standard desktop subset, it rose from 55.80% (first-person view) to 60.65% (multi-view). The box subset was a rare exception, where adding views caused a slight drop in metrics, indicating that multi-view inputs do not always yield gains in extremely dense scenarios.

The remaining subsets showed stable performance: under multi-view conditions, the success rate for seen objects was 56.02% and for unseen objects 60.34%; for the shelf subset, it was 44.63%. As vertical occlusion increased, the value of multi-view information became more prominent.

The team conducted ablation experiments to verify the benefits of scaling up data volume. Models were trained using 20%, 40%, 60%, 80%, and 100% of the complete training set. The grasp success rate gradually improved from 40.50% to 44.41%, while the collision avoidance rate rose from 81.54% to 83.66%, peaking at 84.15% with 80% of the data. Complex spatial relationships, such as object-to-object avoidance and obstacle avoidance, require sufficient scene samples for learning. This necessity is a core reason why the team expanded the dataset to 2.6 million scenes.

Researchers performed ablations by breaking down input conditions one by one. When only depth maps were input, the model learned only basic object shapes, maintaining a success rate in the 30% range. Adding RGB images introduced semantic and target localization information, significantly boosting performance. Supplementing camera parameters further improved geometric perception accuracy. Multi-view inputs yielded the largest improvement, achieving a success rate of 49.87% under full multimodal input. The desktop height parameter, though seemingly minor, is also critical; with the coordinate system origin centered on the desktop, it aligns the geometric information derived from depth inversion with the ground truth for grasps. Omitting this metric directly causes performance degradation.

The team also validated the value of various model components. Using a standard denoising loss tends to produce average solutions across multiple grasp poses; replacing it with a soft-winner-take-all objective significantly improved success rates. Physical constraints embedded during the training phase determine whether millimeter-level contact is viable. During inference, switching from random candidate selection to physical scoring raised the success rate from the 30% level to the 40% level.

The effects of optimizing parallel sampling were quantified in ablation experiments. Increasing the number of parallel candidates from 1 to 64 steadily raised the grasp success rate from 36.08% to 44.41% and the collision avoidance rate from 79.71% to 83.66%, with inference time increasing only from 219 milliseconds to 1,598 milliseconds. In contrast, if iterative optimization were used during inference with 200 iterations against the same physical objective, the time cost reached approximately 6,044 milliseconds, yet the success rate remained only 36.16%, and the collision avoidance rate plummeted to 2.56%. The team found that iterative optimization often deviates from effective grasp manifolds, leading to increasingly unstable results. This instability is precisely why they adopted the parallel candidate scoring approach.

The team further conducted failure analysis and reachability verification.

Failure causes were analyzed on the standard desktop subset: among grasp failure cases, 95.71% failed to maintain a stable grip after contact was established, while only 4.29% failed to touch the object at all. In collision failures, 61.26% involved striking the support surface, and 38.74% involved colliding with other non-target objects. These data clearly illustrate that the primary difficulties lie in maintaining stability after contact and avoiding obstacles in cluttered environments. Reachability testing utilized a Franka robotic arm, solving inverse kinematics with five base placement configurations set from a first-person perspective. Among qualified grasp plans, 83.75%–87.98% fell within the reachable space executable by the robotic arm, demonstrating that most grasp poses generated by the model can be implemented on a real robotic arm.

The two categories of failure cases point to directions for future improvement.

The majority of grasp failures occur after contact is established, indicating that the model can successfully position its fingers at the target location; the bottleneck lies in maintaining a stable force closure. Roughly sixty percent of collision failures involve touching the support surface, meaning the dexterous hand’s understanding of spatial constraints when reaching downward toward the table remains insufficient. This also explains why inputting the table height is indispensable.

From a qualitative perspective, the model operates stably across vastly different scene layouts. Even in scenarios with densely stacked objects or multiple visually similar items placed adjacent to each other, the model can still lock onto the target using the target mask and output feasible grasp poses.

Simulation grasps are effective, but can they be reproduced on real hardware?

The team completed a small-scale hardware validation. OmniDex was retrained on a small dataset adapted to the hardware, using a Tianji Marvin M6S robotic arm paired with a BrainCo Revo2 dexterous hand. Target masks were generated via SAM, and grasping experiments were conducted in cluttered scenes containing apples, mangosteens, water cups, and other objects. The real-robot camera setup replicated the simulation environment’s configuration, allowing the spatial priors learned in simulation to directly match real-world observations. This experiment focused on qualitative verification; the paper does not report real-robot grasp success rates, but it demonstrates that grasp poses trained in simulation can be transferred to real hardware.

This model and its accompanying benchmark provide a large-scale, high-complexity training foundation for dexterous-hand grasping in cluttered scenes, eliminating the need for iterative post-optimization during every inference step for high-DOF dexterous hands. To move tabletop dexterous grasping from demonstration to practical application, such underlying infrastructure is currently missing.