Beyond the Model: Ego2Robot Brings Data Processing Pipelines Back to the Spotlight

A robot completing a task on a fixed desktop and succeeding after the tablecloth is changed, lighting is altered, and the camera position is moved represent two entirely different capabilities. The former indicates that the policy has converged; only the latter determines whether it can leave the laboratory.

It is well known that teleoperating robots is inherently slower and more expensive than human-centric video recording. Previous work had only validated the approach of 'redirecting hand poses, rendering robotic arms, and replacing human hands in the scene' on small-scale, single-task scenarios. Furthermore, there has been a lack of systematic answers to the question of where exactly the benefit of synthetic data for VLA model pre-training comes from.

On August 3, the Alibaba Qwen team, along with institutions such as Renmin University of China, released 'Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data.' This work processed approximately 1,940 hours of human-centric manipulation videos through three pipelines—action alignment, visual alignment, and multi-level quality filtering—to generate 18,561 hours of synthetic training data covering 15 types of robot morphologies. Decoupled evaluation proved that this synthetic data significantly enhances generalization capabilities.

Seizing the opportunity of this paper's release, we spoke with Xionghui Chen, co-first author of Ego2Robot, who is also one of the core authors of the recently popular work 'Qwen-RobotManip.' We mainly discussed the following topics:

• What bottlenecks typically occur in previous attempts to train robots using human first-person perspective videos? Does this paper successfully pave the way for 'converting human videos into robot data'?

• There are three very critical stages in the pipeline: action alignment, visual alignment, and multi-level quality filtering. Many people believe there is a gap between rendering and the gripper center. How should this gap be interpreted?

• Ego2Robot introduced an innovative 'decoupled evaluation benchmark.' Results showed that mixing synthetic data with real robot data for training led to significant improvements in generalization under visual, proprioceptive, and semantic perturbations. However, the improvement on the Franka robot was minimal. What does this phenomenon indicate behind the scenes? Does it mean that cross-embodiment transfer has boundaries?

In terms of the specific results demonstrated by the paper, 1:1 mixed training increased success rates from 58% to 66% under lighting variations and jumped unseen object instance handling from 29% to 40%. All five long-horizon tasks on real hardware showed clear gains. Based on discussions with Xiong Hui, as well as the research content and effects presented in the paper, we can briefly address these topics:

• First, Xiong Hui's own response was relatively restrained. He did not claim that it was 'fully working,' but rather used the phrase 'seeing hope for it to work.' He provided two pieces of support for this 'hope': first, the current scale of data is effective and promotes generalization out-of-distribution (OOD); second, after adopting this pipeline, ego-centric (human-centered) data and robot data can achieve knowledge sharing through cross-embodiment transfer via mixed training.

• Regarding the gap, he did not shy away from it, stating, 'It’s not just the gripper; every step has a gap.' His approach to solving this is twofold. At the data level, a filtering pipeline is coupled into every processing stage to filter out data with large gaps or poor quality. At the training level, weights are reduced for dimensions with larger errors, such as gripper opening/closing mapped from hand keypoints.

• First, the Franka single-arm vertical articulated structure differs most significantly from human kinematics. Second, ego data provides diversity in scenes, objects, and instructions, preventing the model from overfitting to the narrow distribution of teleoperation. Fine-grained action precision is still ensured by real-machine teleoperation data. Xiong Hui also mentioned that this could lead to another exploratory direction: an object-centric pipeline, attempting to bypass whole-hand kinematic bias, allowing non-human morphologies to benefit from ego data.

In fact, Ego2Robot not only demonstrates a robust data processing pipeline but also highlights its core emphasis on data governance. Ego2Robot invested significant effort in three-level quality filtering, training weight reduction, and cross-morphology alignment. This underscores that the key challenge in fusing large-scale heterogeneous data may lie not in 'generating data,' but in 'filtering data.'

For the Embodied AI industry, successfully paving this path means the bottleneck for robot training data is shifting from 'how much is collected' to 'how well it is filtered,' with the latter being something that can be continuously optimized and scaled up engineeringly.

One Pipeline, Three Steps to Success

Ego2Robot is far more than just a combination of algorithms within a synthetic data pipeline; it is a systematic engineering effort. Chen Xionghui, co-first author of this work, mentioned in discussions, 'We cannot ignore that it is actually a relatively systematic, engineering-oriented endeavor. We spent considerable effort on data governance; without this data governance, achieving final success would have been quite difficult.'

The entire pipeline is divided into three major modules: action alignment, visual alignment, and quality filtering. Meanwhile, input supports two paths. Path 1: datasets with existing hand pose annotations enter the pipeline directly. Path 2: ordinary videos without any annotations first use WiLoR to estimate hand frames by frame, DynHaMR for temporal optimization, and Qwen3.5 to automatically split long videos into subtasks with natural language descriptions, before entering a unified processing flow.

First, look at the action alignment module. Similar to QwenManip, it converts hand keypoints into gripper poses. The paper and appendix provide detailed explanations; researchers specifically defined virtual fingertips, including a weighted average of the index and middle fingertip positions, which, together with the thumb fingertip, determine the TCP position, opening width, and grasping orientation of the gripper.

Handedness symbols ensure that left and right hands map to the same gripper coordinate system. Additionally, per-frame detection suffers from high-frequency jitter, which is smoothed out using Savitzky-Golay filtering and Gaussian-weighted SLERP.

It is worth mentioning speed alignment. Since hand movement speeds in human first-person videos are far faster than robot teleoperation, researchers appropriately lowered the sampling rate. ViTRA was reduced to 25% of the original frame rate (equivalent to a 4x deceleration), EgoVerse to 45%, and EgoDex and ANT to 60%, allowing the action distributions from various data sources to align more closely with the rhythm of real robots.

Visual alignment is responsible for removing the human hand from the frame and replacing it with a rendered robotic arm. SAM 3 segments the arm, while ProPainter inpaints the background to fill areas occluded by the arm. In fact, the real challenge lies in determining where to place the robot base. Robot-to-robot transfer usually has a source base to reference, but ego trajectories have none.

The paper's approach selects a circle of candidate bases around the trajectory centroid, verifying the solvability rate of keyframes one by one using a MuJoCo inverse kinematics solver, and choosing the highest-scoring option as the base. This step runs independently for each of 15 robotic arm morphologies with different arm lengths and joint configurations. Once the base is fixed, IK is solved per frame, the robot is rendered from the original camera perspective, and perception synthesis is performed using SAM3 depth estimation, where gripper pixels are determined to be displayed or occluded by the scene based on depth relationships.

The final and also innovative quality filtering module implements a three-tier fallback mechanism. L1 filters out basic failures within the pipeline, such as IK failures, self-collisions, and anomalous actions. L2 involves post-processing statistics, including Q1/Q99 filtering and mutation detection; any episode with an invalid frame rate exceeding 60% is discarded entirely. L3 entrusts a VLM with final judgment, specifically using Qwen3.5 to sample video frames at 4fps to determine whether the rendered robot actions match the task descriptions.

Data filtering is not a standalone step that runs after the preceding stages; rather, it is tightly coupled with the processing pipeline. For instance, solutions with poor Inverse Kinematics (IK) results are discarded—a design philosophy reflected throughout subsequent processing stages. Finally, a Vision-Language Model (VLM) is used to verify whether the robot's actions align with the instructions. The overarching logic of this pipeline is to reduce data volume by filtering out unusable samples while retaining those that are viable.

Beyond filtering at the data level, the training phase incorporates corresponding fallback strategies. Xiong Hui revealed that the team cleverly reduced the weight assigned to certain high-error dimensions, such as gripper opening and closing, during training. "This prevents these dimensions from causing significant interference in the overall pretraining, thereby enhancing the robustness of the results."

Which Shortcomings Does Synthetic Data Fill?

With the pipeline explained, we return to the core question: With so much data invested, where exactly does the policy improve?

Xiong Hui made a key statement during the discussion: "Pretraining should be evaluated in Out-Of-Distribution (OOD) scenarios. Therefore, when validating these pipelines, we focus more on OOD evaluation." This prior judgment directly determines that the evaluation must target generalization scenarios.

First, existing generalization evaluations suffer from a 'catch-all' flaw. Backgrounds, lighting conditions, table heights, and distractors are all combined into a single OOD score, making it impossible to distinguish whether the model successfully resisted lighting changes or was merely insensitive to the background.

To address this, Ego2Robot decomposes generalization into four independent dimensions and 12 evaluation settings based on RoboTwin 2.0. These include visual appearance (background textures, lighting, robot color), scene layout (table height, distractors, camera offset), embodied morphology (zero-shot adaptation to UR5-WSG, ARX-X5, Franka Panda), and task semantics (unseen object instances, colloquially rewritten instructions).

The experiment included four control groups: pure robot data (DROID + AgibotWorld + InternData, totaling 6,565 hours), and mixed synthetic and robot data at ratios of 1:3, 3:1, and 1:1.

Looking first at the overall scores. Under the Clean setting, the 1:1 mix achieved 68.1%, surpassing the pure robot baseline of 62.2% by 5.9 percentage points. Under Randomized conditions, the 1:1 mix scored 53.5%, an improvement of 2.6 percentage points over the 50.9% baseline. The gains from the 1:3 low-ratio mix were negligible, with most dimensions fluctuating within ±1 percentage point. This indicates that a significant effect only emerges when the proportion of synthetic data is increased to 1:1 or even 3:1.

Overall scores merely indicate effectiveness; decoupled data reveals where that effectiveness lies. There are substantial differences across dimensions.

The visual dimension was the biggest beneficiary. Lighting variation improved from 58.2% to 65.8%, a gain of 7.6 percentage points; robot color improved from 59.4% to 65.8%, a rise of 6.4 percentage points. Egocentric videos naturally cover various lighting and backgrounds in settings like kitchens, living rooms, and factories, while randomized rendering of 15 different colors significantly increases the diversity of robot appearances. The combination of these two factors allows the model to adapt to more scenarios.

Semantic generalization also saw significant gains. Performance on unseen object instances jumped from 29.3% to 39.6%, an increase of 10.3 percentage points—the largest absolute gain among the four dimensions. Colloquial instruction rewriting improved from 63.1% to 68.5%, a rise of 5.4 percentage points. The variety of objects and natural language diversity in egocentric videos provided the model with a richer semantic prior.

Embodied transfer warrants discussion. ARX-X5 improved from 44.1% to 51.2%, a gain of 7.1 percentage points; UR5-WSG reached 31.4% under the 3:1 mix. However, Franka remained stuck below 7%, showing no change.

It is not difficult to observe that the items performing well in terms of transfer are basically those with a hand layout more similar to humans. Franka does not resemble the dual-arm structure of a human, which also conforms to objective laws.

On the four generalization dimensions, viewpoint robustness reveals a more fundamental rule. The camera offset increased from 50.4% to 56.3%, because egocentric videos naturally involve head movement. This baseline further amplified this effect. Its camera is higher, and the robot's dual-arm layout is closer to that of a human body; the three-to-one mix surged to 51.7% here, which is 12.1 percentage points higher than the pure robot performance of 39.6%.

Back to the core question: why is training with 1:1 mixed data effective?

Xionghui provided a very clear complementary positioning: "For more precise action grounding, we focus on using real teleoperation. Ego provides us with more diverse scenarios, tasks, instructions, and some coarser but aligned actions. This prevents the model from overfitting to the narrower distribution of teleoperation data."

From an overall experimental perspective, this represents a two-pronged approach in data dimensions: real-robot data ensures action precision, while synthetic data provides coverage and anti-overfitting capabilities.

Tried it on the actual hardware

Researchers tested the impact of synthetic data on task performance using real machines.

In the experimental setup, the real robot used was the ARX ACone dual-arm platform. Five long-horizon tasks were constructed: placing fruit into a basket, putting blocks into a drawer, folding towels, sweeping up trash, and tightening screws. For each task, only 20 teleoperation demonstrations were provided. Additionally, approximately 35 minutes of ego-centric operation videos (ego play) were recorded. These videos were processed through the complete pipeline to generate synthetic demonstrations in the ACone configuration.

Three conditional comparisons were made for the real-robot demonstration data: Robot-only (pre-training on robot data alone + fine-tuning with 20 demonstrations), Mix (1:1 pre-training mix + fine-tuning with 20 demonstrations), and Mix + Ego2R Play (Mix baseline, where during fine-tuning, teleoperation demonstrations were mixed 1:1 with ego play data converted by the pipeline).

Looking directly at the results, Mix + Ego2R Play achieved the best performance across all five tasks. The success rate for placing blocks was 14 percentage points higher than Robot-only, and for tightening screws, it was 13 percentage points higher.

In fact, the step of Mix pre-training alone already surpassed Robot-only. Adding the ego play data into the fine-tuning phase further improved the real-robot performance.

Additionally, the paper's appendix includes a set of visual comparisons: the top images show original human-hand operations, while the bottom images show the rendered ACone overlay from the pipeline. The sensory difference is quite intuitive.

However, Xionghui specifically highlighted a detail regarding this experiment: the ego play recordings were not casually shot with a mobile phone. The human hand performed operations in front of the robot’s built-in head camera. "We held the robot's own camera and placed our hands in front of it to perform operations, thereby mitigating a significant portion of the viewpoint gap."

In other words, the current experiment has already controlled for the camera angle variable. What if we use a phone to shoot from any position? He directly stated, 'I think it's quite far; achieving 99.9 percent capability is possible, but there may still be a gap.'

However, this set of real-machine experiments also revealed a positive signal: apart from the camera perspective, other aspects seem to have been successfully addressed. Xionghui believes: "If we handle the camera perspective issue better, it may become easier in the future to see scenarios like a robot learning to some extent from a video recorded by a mobile phone."

Areas for Optimization

Ego2Robot demonstrates that egocentric human videos can be scaled into multi-form robot training data. Large-scale synthetic data provides the clearest gains in visual and semantic generalization. Simply put, synthetic data and real robot data are complementary: the former covers coverage gaps, while the latter ensures precision.

Of course, there is room for optimization both in the paper and outside it.

First, the data pipeline should not take the blame for the foundation model. As can be seen, the pipeline relies on SAM3 for arm segmentation and depends on open-source monocular models for depth estimation; neither of these two aspects is within the team's control.

Xionghui was very direct, saying, "Whether it's SAM3 or depth estimation, these all show obvious limitations."

Therefore, the team's current strategy is post-processing fallback. If segmentation is inaccurate, IK filtering will discard errors; if depth estimation is inaccurate, VLM final review will filter them out. Of course, this logic works on a small scale, but as data volume continues to increase, these impacts may become more pronounced.

Additionally, the gap in camera perspectives is harder to tackle than expected. In real-robot experiments, ego play uses the robot's built-in head camera; when a human hand reaches into the camera's field of view to operate, the perspective naturally aligns with that of the robot during actual operation.

This is underpinned by two layers of issues: first, the perspective and height of mobile phone shots are completely different from those of the robot's head-mounted camera; second, even if the perspectives are aligned, bridging the gap from 'can do' to 'reliably complete' requires an additional post-processing pipeline.

Regarding this, apart from the perspective alignment in the software algorithm part, Xionghui also mentioned that there is room for improvement in the hardware level of the ego recording device itself: "An ego with three perspectives (head + two wrists) is better than a pure monocular one. Currently, there are not many open-source three-perspective systems, but I believe that as the field develops, this should take data utilization to another level."

This also means that optimizing the pipeline itself is one aspect, while upgrading upstream data collection methods is equally indispensable.

Finally, there is the hard boundary of embodied transfer. Franka Panda stays below 7% across all mixed proportions, which clearly cannot be solved merely by adding more types of data.

Furthermore, the paper itself acknowledges certain limitations. The redirection capability currently supports only parallel grippers, excluding multi-finger dexterous hands; visual inpainting and depth synthesis may still produce artifacts under severe occlusion or complex lighting conditions; and the evaluation scope is limited to RoboTwin 2.0, meaning that conclusions drawn after extending to additional scenarios and embodied configurations require independent verification.

Final Thoughts

There is a discussion in the communication section worth placing at the end, regarding whether adding more data will hit a data wall.

Xionghui stated rigorously: 'Will we hit a data wall if we keep adding more data? It's hard to say, as we haven't reached that point yet. We are currently scaling up the data further to see what happens next. I can't really predict what will occur if we multiply it by 10 or even 100.'

However, he immediately added a crucial remark that highlights the value of this paper. Xionghui said: 'Simply scaling up by 10 or 100 is useless without first doing the preliminary work properly. Whether it's correctly evaluating model capabilities, having a good cleaning pipeline, or possessing effective alignment techniques, otherwise, you only get garbage in, garbage out.'

This dialogue better illustrates the position of Ego2Robot than other statements. It represents a path where clear signs have been observed, but significant hurdles still lie ahead. OOD generalization is improving, real-world performance is increasing, and knowledge sharing is occurring, so it is worth continuing forward.

Questions remain unanswered regarding where the data wall lies, which noises in the pipeline might cause failures at larger scales, and whether the boundaries of robot morphologies can be突破 (broken through).

Nevertheless, it appears that using a small amount of real-robot demonstrations to maintain accuracy while leveraging large-scale synthetic data to expand visual and semantic diversity may be the next step for robotic general-purpose manipulation to move out of the laboratory.