First! 30 Researchers from Peking University, NTU, and Others Collaborate to Clarify the Embodied Data Pyramid

On July 28, nearly 30 researchers from more than ten institutions, including Peking University, Nanyang Technological University, the Hong Kong University of Science and Technology, the National University of Singapore, the University of Hong Kong, and Shanghai Jiao Tong University, jointly published a 72-page comprehensive review titled "Data Pyramid for Embodied Manipulation." This review concretizes the data system through a five-layer data pyramid and explores the development history and future directions of embodied foundation models utilizing heterogeneous data.

The paper's author team includes Professor Zhang Shanghang from Peking University as the corresponding author. Collaborators include several well-known scholars in the field of embodied AI, such as Mu Yao, Liu Ziwei, Yang Jianfei, and Luo Ping.

As "data" becomes a hot topic of discussion within the embodied AI community, people have vaguely sensed the existence of an evaluation system to assess the value of various types of data. This highlights the contribution of this review: it organizes what was previously "known by everyone but never systematically written down" into a public language for the first time.

This means that future discussions on the data types and ratios used for embodied foundation models no longer require defining concepts individually; simply referring to "which layer of the pyramid" is sufficient.

The image above shows the Embodied Data Pyramid, with five layers of data arranged from top to bottom:

• Layer 1: Real robot data: Trajectories collected via teleoperation or scripting on physical robots. They are directly executable but also the most expensive.

• Layer 2: UMI-style data: Collected outdoors using handheld grippers. It does not rely on robots, retains end-effector supervision, but lacks joint-level proprioception.

• Layer 3, Self-/External-Centric Data: First- and third-person human videos capturing real-world physical interactions and dexterous manipulation, but requiring redirection to robots.

• Layer 4, Simulation Data: Interaction data generated within physics engines; it is parallelizable and low-cost with privileged labels, yet its physical fidelity remains questionable.

• Layer 5, General Data: Internet-scale text, image, video, and 3D data providing broad semantic and reasoning priors, but entirely lacking action grounding.

The core insight of the data pyramid is that as you go lower in the hierarchy, data becomes easier to acquire but more distant from robot execution; conversely, as you move higher, data becomes closer to executable actions but harder to scale.

This does not imply that because top-layer data quality is superior, one should exclusively use real-robot data. The value of this data pyramid lies in offering a systematic trade-off framework, enabling research institutions or robotics companies to make more informed choices across different layers of the pyramid based on task requirements, budgets, and model architectures.

Of course, the data pyramid represents only part of the review's value. More importantly, this review is the first to systematically address how to combine 'real-robot data, UMI (Universal Manipulation Interface), human videos, simulation, and internet data'—a challenge each player in the Embodied AI community has been exploring. It brings this issue into the open for systematic decomposition.

Although the optimal data recipe remains undetermined, it outlines the current 'data recipes' of existing models, highlights gaps, and provides a comparable, iterative public framework for the data strategies of next-generation Embodied Foundation Models.

The Data 'Arms Race' Has Begun, But Scaling Laws Have Not Yet Been Written Into Textbooks

The figure below clearly illustrates a trend: whether it is simulation data, UMI data, real robot data, egocentric data, or general data,

The scale of data across all categories is growing exponentially.

Real robot data, from the 50,000 grasping trials by Pinto and Gupta in 2015, to AgiBot World Beta's one million trajectories and nearly 3,000 hours of interaction in 2025, to Open X-Embodiment's two million four hundred thousand trajectories spanning twenty-two robot platforms. Over the decade, dataset scale has expanded fiftyfold.

Simulation data is equally staggering. DexGraspNet 2.0 directly generated 427 million grasping samples, and MolmoB0T's training data reportedly reached the billion level. Of course, in the simulation world, 'not enough data' is clearly not a problem; the key is whether the generated data is actually useful.

Self-centric data is also skyrocketing. From the 55 hours of kitchen videos in EPIC-KITCHENS in 2018, to Ego4D's 3,670 hours of global first-person videos, to EgoDex in 2025 collecting 829 hours of high-quality hand interaction data using Apple Vision Pro. "Watching humans work" is becoming a new standard for Embodied AI pre-training, representing a trade-off between the high cost of real robot data and the low quality of general-purpose data.

The scale of general data is already astronomical. SA-1B provides 11 million images and 1.1 billion segmentation masks, while the LLaVA series has built multi-modal instruction data at the millions level. It can be said that general data has no direct relationship with robots, but they are becoming the "common sense foundation" for robot foundation models.

The scale of data is skyrocketing, there's no doubt about that. But the question is, is there a Scaling Law?

The researchers were quite rigorous: although data volumes are growing, the sources, mixing ratios, and processing methods of data for different models vary significantly. Currently, there is no systematic ablation study that proves the "more data equals stronger performance" hypothesis holds true in the field of Embodied AI.

However, the researchers also provided some clues worth paying attention to.

EgoScale conducted a very well-designed experiment: they controlled the pre-training scale of ego-centric data, ranging from 1,000 hours to 20,000 hours. The results showed continuous performance improvement, with no ceiling observed within the 20,000-hour mark. This suggests that ego-centric data may indeed follow a certain Scaling Law.

Xiaomi-Robotics-1 first performed pre-training on 100,000 hours of real-world UMI trajectories, followed by post-training using 10,000 hours of cross-embodied data. The post-training dataset included 7,200 hours of proprietary robot trajectories and 1,000 hours of instruction-labeled UMI data.

The significance of Xiaomi-Robotics-1's pre-training and post-training data scale lies in its replication of the two-stage Scaling paradigm already validated in the LLM domain: "massive internet text pre-training + fine-tuning with a small amount of high-quality instruction data."

Of course, the researchers also cited counterexamples that do not conform to the Scaling Law. Models such as LingbotVLA and DreamZero achieved strong performance using only robot data, without falling behind due to a lack of heterogeneous data. At least based on current experimental evidence, mixing heterogeneous data does not always yield better results.

It is worth noting that π0.7 found that low-quality trajectories, when paired with sufficiently clear prompts, can still provide useful supervision signals. This suggests a potential paradigm shift: rather than aggressively filtering out and discarding data that is "not good enough," we should learn to use data of varying qualities in a "conditional" manner. That is, π0.7 enables the model to distinguish between "this is a standard answer" and "this is an incorrect answer and how to correct it."

It can be said that a directional Scaling Law has already emerged, but an precisely quantified Scaling Law still needs to wait. That is, more diverse data, broader task coverage, and richer perception modalities are helpful for performance in trend, however the contribution of different data sources has not been systematically isolated. Before then, perhaps the problem that needs to be solved first is the optimal data ratio.

Embodied Brain Models, VLA, and World Action Models Have Different "Appetites" for Data

First, let's understand the basic capability requirements of the three types of embodied foundational models, which will help us later comprehend the data focus of each model type and the scale requirements for various data.

The core task of the embodied brain (VLM) is to establish perception, reasoning, and planning capabilities for the physical world, including spatiotemporal perception, physical reasoning, spatial construction, and task decomposition; the starting point of VLA is not only to understand scenes and tasks but also to translate that understanding into executable actions; the goal of world models is to model the temporal evolution and prediction of the physical world, as well as the causal effects of actions.

The image above illustrates the development roadmap of VLA and world models, along with the types of data used. It can be seen that real robot data is indispensable, while human-perspective Ego data is increasingly playing a significant role. Therefore, this article focuses on categorizing three types of models' "data recipes" based on whether action data is included.

For the embodied brain, the key capabilities of its models lie in perception and planning; therefore, it is more important to establish "world knowledge" using data without actions. As shown in the figure below, this type of data includes general image-text data, video-text data, VQA, OCR data, spatial and temporal QA data, instruction-following data, and egocentric videos that have been re-annotated.

These data provide capabilities such as object recognition, language understanding, visual commonsense, and spatiotemporal reasoning. However, these capabilities cannot be directly used for a robot's specific action execution; therefore, RoboBrain and ShareRobot have also explored the possibility of incorporating action data as a core supervisory signal.

The VLA model establishes the core status of action data. Regarding action representation, VLA development follows a clear trajectory: from RT-2's discretized action tokens, to GR00T N1 and RDT-1B using diffusion policies to model high-dimensional continuous action distributions, and finally to π0.7 and InternVLA-A1 employing flow matching strategies to directly generate action trajectories.

The core requirement driving the development path of "action representation" in the above VLAs is to reduce action errors; even small errors in continuous control can ruin an operation.

Another interesting aspect is how VLA models utilize action-free data. The paper outlines three technical approaches.

The most direct method is the Latent Action Proxy, which learns implicit representations "related to actions" from unlabeled videos. The advantage of this approach is scalability; it can learn dynamic priors from massive amounts of video without precise annotations, although it still requires a small amount of robot demonstrations for practical implementation.

The second approach is geometric motion representation, such as VideoDex transferring human hand movements to robot policies. Compared to implicit methods, these geometric representations offer greater interpretability and cross-embodiment transferability, but they rely more heavily on reliable perception.

The third way VLA models leverage action-free data is through hierarchical intermediate supervision. Models like π0.5 and CoT-style VLAs use this data to guide downstream action generation, effectively encouraging the model to "think more clearly before executing."

These can explain the trend mentioned earlier that data types have converged towards "egocentric human videos." The paper points out that through hand pose reconstruction, retargeting, hand repair, or shared representation spaces, human dexterous manipulation data can empower robots' dexterous policy learning, significantly expanding the coverage of "action data."

Finally, unlike embodied brains that lean toward motionless data, the core of VLA is motion data. The World Action Model (WAM), to predict the state after an action occurs, naturally requires both types of data. That is, motion data establishes the causal chain of "action → consequence," while motionless data establishes the prior of "how the world typically changes."

Regarding the lack of action data, internet videos constitute the largest source of dynamic supervision. UniPi reframes decision-making as text-conditioned video generation, UniSim aggregates heterogeneous video and interaction data into an interactive real-world simulator, Cosmos pretrains a full-modal world model on billions of frames of internet videos, and V-JEPA 2 learns latent video prediction from over one million hours of internet videos.

Regarding action data, most WAMs are built on diffusion-based video generation models, treating action prediction as a continuous generation problem. Genie Envisioner mounts a flow-matching action expert, Motus separates the video and action experts, and Cosmos Policy converts actions into latent frames. The exception is WorldVLA, which is based on an autoregressive architecture that uniformly discretizes observations and actions into tokens for joint prediction.

Subsequently, a closed loop regarding world models will also be established, where trained world models can 'feed back' data. For example, DreamGen and Cosmos can generate synthetic interaction data, while WorldGym and WorldEval can partially replace physical environments in policy evaluation.

Simulation Data and Real Data Are Never an Either/Or Choice

Among all discussions on embodied data, the debate of "simulation vs. reality" may be the most emotional. On one side, some believe that simulated data can never replace real physics; on the other, others insist that real data is too expensive and slow to scale up.

In fact, the paper argues that the true value of simulation data lies in providing something the real world can never offer: perfect controllability and privileged labels.

First, let's discuss privileged labels. In simulations, researchers can precisely know the 6D pose of every object, the force distribution at every contact point, the success or failure of each task, and the exact state of every joint step. This is information that is almost impossible to acquire with equal precision from real-world data.

More importantly, controllability allows for 'counterfactual' data generation. Unlike the real world, where we can only observe 'what happened,' simulation data can controllably guide tasks toward 'what might happen.' It is this kind of 'structured exploration' that holds the true significance of simulation data for learning robust policies.

The table above summarizes large-scale synthetic and simulation data resources. Recent datasets show an increasing inclusion of diverse embodied morphologies and larger data volumes, validated through a wider variety of tasks.

Another interesting trend regarding simulation deserves mention. From systems like GenSim, GenSim2, and RoboGen mentioned in the paper, to NVIDIA's recent SimFoundry and ByteDance's TableVerse, the generation of simulation data is shifting from 'human-written expert policies' to 'LLM-driven automatic generation.' This means the true advantage of simulation's 'scalability' is being amplified once again.

In fact, the paper also discusses the gap in Sim-to-Real transfer, which mainly involves mismatches at two levels.

First, the most direct issue is observation mismatch. Although recent datasets have enhanced realism through 3D assets, lighting, camera models, and various sensors, differences between rendered textures, depth noise, occlusion patterns, tactile responses, and force signals and those in the real world remain difficult to completely eliminate.

Second, the interaction mismatch that is crucial for robots to work in the real world. This is subdivided into kinematic gaps related to joint limits, coordinate frames, and differences in control frequency, as well as dynamic gaps involving actuator delay, friction, contact, deformation, and object inertia, which are harder to eliminate and can only be approximately captured by simulation engines.

The paper points out that the core issue of simulation is no longer just data generation, but whether the generated data captures perceptual variations, kinematic consistency, and dynamic uncertainties that determine the success or failure of real-world deployment.

Notably, the paper discusses another emerging route for simulation: world models as "learned simulators." The paper categorizes the role of world models in robot learning into three aspects:

• As a virtual environment for policy training: methods such as World-Env, VLA-RFT, and DiWA enable policies to be optimized through "imagined interactions," reducing the repeated reliance on physical robots.

• As a policy evaluation tool: WorldGym and WorldEval investigate whether the rollouts generated by models can support policy evaluation.

• As a synthetic data engine: DreamGen generates robot videos and recovers pseudo-actions; GigaWorld-0 uses world models as a data engine for embodied learning.

Unlike traditional simulations based on physics engines, world models possess both the scalability potential of simulation and 'physical priors' learned from real-world data. However, the paper also points out certain limitations: prediction errors, model exploitation, action ambiguity, and distribution shift may lead to the generation of trajectories that 'appear reasonable but are physically invalid.'

Regarding the comparison between simulation and reality, the paper's stance is: there is no issue of "one replacing the other" between simulation data and real data; the key lies in how to systematically and complementarily utilize them.

Real-world robot data provides the most direct and credible physical supervision. It naturally incorporates real sensor noise, control latency, contact dynamics, and hardware constraints, while recorded actions are directly executable on their corresponding platforms.

A best practice for leveraging their complementary strengths might be: using simulation for large-scale pre-training and covering long-tail scenarios, while employing real-world data for fine-tuning and calibration; utilizing digital twins and domain randomization to bridge the visual gap; and adopting world models as an intermediate layer connecting real observations with synthetic generation.

Most Promising Directions for the Future

This is arguably the most instructive section of the article, reflecting the Embodied AI academic community's thoughts on future technological development. It proposes six directions centered around three questions: "what to collect," "how to collect it," and "how to use it."

• Large-scale tactile data needed to fill the missing contact layer. This represents the largest "structural gap" in the current data pyramid. Although recent datasets like RoboMIND 2.0 and Humanoid Everyday have begun incorporating tactile signals, tactile data remains limited by specialized sensor hardware, inconsistent signal formats, restricted task coverage, and weak integration with long-horizon operation trajectories.

• Failure and recovery data, shifting models from "success-centric learning" to "resilience training." This recalls corner case data in the autonomous driving field; it is precisely those rare but critical failure scenarios that define a system's safety boundaries. Works such as RoboFail have already begun exploring failure data from multiple perspectives.

• Establishing scalable data collection pipelines essential for robust data engineering. As illustrated above, egocentric data collection is evolving from simple RGB video toward multimodal wearable sensing. Devices such as head-mounted AR/MR equipment, wrist cameras, hand IMUs, instrumented gloves, tactile patches, and force sensors are gradually completing the multimodal data acquisition puzzle. However, issues like viewpoint occlusion, motion blur, drift, and time synchronization indicate that these devices are not yet fully mature.

• Cross-embodied state-action alignment. This is a unique challenge for embodied AI compared to LLMs. Different robots vary in joint count, action dimensions, coordinate frames, and control frequencies, meaning there is no public database that enables embodied foundation models to evolve from 'specialists' to 'generalists'.

• Self-centered data empowers dexterous hands, bridging the human-machine gap. The paper argues that the human-machine gap is not only visual (the appearance difference between human hands and robotic hands), but also kinematic, morphological, and physical, which may manifest on real hardware as "violating robot constraints" or "failing to produce stable contact."

• Data recipe. This is one of the most forward-looking discussions in the paper and may also be the most underestimated issue in the entire field. As mentioned earlier, the data used to train embodied foundation models is becoming increasingly diverse and large-scale, but existing reports vary greatly in terms of data source combinations, sampling ratios, and phased allocation strategies; no two models use exactly the same "recipe".

Returning to those three questions: the first two directions answer that tactile data and failure data should be collected; the third direction indicates the need for more suitable, more general-purpose, and lighter data acquisition equipment. The subsequent three questions systematically guide the use of data through methods such as alignment, hardware adaptation, and reasonable proportioning.

Final Thoughts

In fact, the review dedicates considerable space to discussing the advantages and disadvantages of various data types such as 'data scale' and UMI. However, within the community's prevailing narrative that 'bigger is better,' quality is often the first thing sacrificed, making it an area that deserves equal attention.

As the paper places 'quality' first among four supplementary evaluation dimensions in its introduction, diversity, reusability, and physical fidelity are the other three. Several trends are emerging in both industry and academia; while not yet dangerous, they still warrant vigilance.

First, the quality dilution of automated annotations. General data increasingly relies on multimodal VLM models to automatically generate annotations, but these automated annotations may contain hallucinated objects, incorrect relationships, and templated language expressions. The paper explicitly points out that automated annotations usually require confidence estimation, consistency checks, or manual verification before they can be used.

Second, the trade-off between diversity and quality. The figure above illustrates the significant differences in trajectory diversity across various datasets; some cover a broad 3D space, while others are highly concentrated. High-diversity data originates from uncontrolled environments outside the laboratory, whereas high-quality data comes from controlled laboratory settings. Finding a balance between these two is crucial.

Therefore, a data quality assessment system encompassing metrics such as information density, annotation reliability, and physical consistency may constitute more urgent infrastructure development than the Scaling Law itself.

As this review demonstrates, the data ecosystem for Embodied AI is at a critical juncture transitioning from a 'wild era' to an 'era of intensive cultivation.' Essentially, this 72-page review offers an important reminder: Now that we possess this five-layer data pyramid, do we truly understand the value of each layer? How do different layers complement each other? Have the data recipes selected for specific tasks and model architectures been systematically considered and validated?

These questions have no ready-made answers, yet they may be far more important than merely asking 'how many terabytes of data we used.'