Silicon Valley Scrambles for Egocentric Data as Chinese AI Turns to Legacy Third-Person Videos
To address the data shortage in embodied AI, a new approach is emerging: bypassing egocentric data collection. Over the past year, egocentric human videos have become a highly contested resource for robotics companies. However, acquiring this data remains challenging. Egocentric videos typically require head-mounted cameras, and as tasks, personnel, and environments scale, the costs of equipment, data collection organization, and quality control increase accordingly.

To address the data shortage in embodied AI, a new approach has recently emerged: bypassing egocentric data collection.
Over the past year, egocentric human videos have become a highly contested data resource for robotics companies. However, acquiring this data is far from easy. Egocentric videos typically require capture via head-mounted cameras and similar devices. As the scope of tasks, personnel, and environments expands, the costs associated with equipment, data collection logistics, and quality control rise accordingly.
In contrast, the internet already hosts a massive volume of exocentric videos documenting real human manipulation across diverse scenarios. Although these videos were not recorded for robot training, they likely contain vast amounts of untapped manipulation data.
Given the critical importance of egocentric data, is it strictly necessary to record new footage? Could models be used to transform existing exocentric videos into egocentric ones?
Over the past six months, several overseas research teams have explored this possibility from various directions.
In May, a research team at the University of Texas at Austin introduced EgoExo-WM, which extracts structured human poses from exocentric videos to serve as action representations. The model then converts these videos into egocentric views to train an action-conditioned egocentric world model.
In August, another team at NVIDIA introduced Grounded-Exo2Ego, which combines 3D geometric reconstruction with semantic information to address missing spatial structures, object occlusion, and generation inconsistencies under extreme viewpoint changes.
These teams have pursued various approaches—including video generation, action recovery, and world models—to bridge the information gap between exocentric and egocentric perspectives. Cross-view generation is not a new research topic, but its value is now extending beyond the videos themselves to the production of robot training data.
This approach has recently begun to attract attention from overseas business and technology media, as well as AI content creators.
On November 17, the U.S. business and technology publication The Silicon Review published a feature article discussing third-person to first-person video conversion in the context of robot data collection costs. The article described Vita Dynamics, the company behind Vbot, as an "unusual entrant." Rather than following the conventional path of deploying more dedicated data collection hardware, the robotics company introduced Exo2EgoPolicy, which attempts to leverage existing third-person videos to generate the first-person data required for robot training.

On X, AI developers and content creators have also highlighted Exo2EgoPolicy's ability to perform cross-view generation during inference without requiring additional depth data or camera calibration, as well as Vbot's approach to integrating smartphone-based data collection with downstream processing.

These overseas discussions ultimately center on a single question: does data acquisition necessarily require increasingly complex front-end collection hardware?
Vbot's solution is to retain first-person video as an intermediate representation, introduce additional supervision for the hand structure and object interaction information required for robot manipulation, and then further process the action data through the Echo pipeline.
This approach involves a noteworthy trade-off: whereas past methods relied on dedicated hardware to capture viewpoint and action data, generative models attempt to shift a portion of this cost to backend computation and data processing.
How far this approach can go remains to be verified. For now, it is worth examining what the model has already achieved.
From spectating to experiencing
In 1925, Charlie Chaplin served a leather shoe on a dinner table.
In the film The Gold Rush, a camera positioned at one side of the table captured this absurd "dinner." The audience watches Chaplin and his companion sit face-to-face, using knives and forks to tackle the leather shoe on the plate.
More than a century later, this scene can now be filmed from a perspective that did not exist in the original footage. The camera suddenly shifts to the character's eye level, with the dinner plate dominating the lower field of view and hands reaching out from the edges of the frame, turning the person sitting across into the subject directly ahead.

The same scene shifts from watching someone else eat to feeling as though you are sitting at the dining table yourself. Similar experiences can be found in many other classic films.
In the crowded tango dance floor scene from Scent of a Woman, the original footage cuts back and forth between wide and close-up shots. In contrast, the generated first-person perspective keeps Al Pacino locked in frame, with the skeletal structure of his hands continuously adapting to the dance steps.
If treated merely as a visual effect, this is just a novel trick. But in the context of robot learning, the difficulty and value of this transformation go far beyond simply changing the camera angle.
In a third-person perspective, the camera is typically positioned outside the subject, capturing the operator, the surrounding environment, and the target object simultaneously. However, when shifting to a first-person perspective, the camera must move near the subject's head, altering the relative positions, occlusion relationships, and visible areas of the various objects in the scene.
For minor changes in perspective, traditional 3D reconstruction can provide relatively accurate geometric correspondences. However, when the viewpoint shifts from directly in front of the subject to their own first-person perspective, the unobserved areas in the source video increase significantly. The model must not only infer the spatial layout from the new viewpoint but also generate content that was not directly captured in the original footage.
This also explains why recent cross-view research increasingly incorporates generative models. While geometric information provides constraints, reconstructing large unobserved regions still requires leveraging the scene, object, and motion priors learned by the models from data.
Vbot's Exo2EgoPolicy adopts a similar approach but features targeted designs for its training methodology and the preservation of manipulation information.
Built on a pre-trained video diffusion Transformer, the model incorporates paired first- and third-person videos along with camera geometry conditions during training. This allows the model to learn correspondences between different viewpoints before gradually reducing its reliance on explicit camera geometry inputs.
As a result, during inference, the model no longer requires additional camera calibration parameters for the target viewpoint. Instead, it infers the corresponding first-person frames directly from the original third-person video and its manipulation description.
However, viewpoint translation is only one of the challenges. In movies or everyday videos, multiple people may appear in the same frame, and camera angles may shift continuously. Relying solely on video generation could cause the model to track the wrong person or even conflate the actions of different operators.
To address this, Exo2EgoPolicy incorporates a source video perception and reasoning mechanism that analyzes people, hands, manipulated objects, and action sequences. In multi-person scenarios, users can also specify the target character to track using a reference screenshot.
During training, the team also utilized multiple synchronously captured third-person camera angles, switching the input camera feed within a single continuous action. By using the first-person video of the same individual as the generation target, the model learns to maintain consistency in both person identity and action execution as the camera angle changes.
Yet for robot manipulation data, the hands present an even greater challenge.
Standard video generation models can produce realistic-looking hands, but their joint positions and motion trajectories may lack sufficient accuracy. While these imperfections might go unnoticed in entertainment videos, they can directly compromise the quality of downstream action data.
To address this, Vbot introduces two types of hand-object interaction supervision into its video generation model. The first is semantic segmentation, which trains the model to identify and distinguish between hands, arms, manipulated objects, and the background, thereby constraining their spatial relationships across consecutive frames. The second supervision signal targets the hand structure directly. The team leverages the ACE-Ego-Hand model to extract hand features, joint keypoints, and motion information, incorporating them as auxiliary training objectives during the video generation phase to ensure that finger structures and motion dynamics in the generated frames remain physically plausible.

Evaluated on a validation set of 546 clips, the introduction of ACE hand supervision improved the hand keypoint PCK@5% metric of the generated videos from 11.52% to 20.02% compared to the RGB-only baseline, while reducing the average keypoint position error from 57.09 pixels to 47.42 pixels.
This indicates that, under this test setup, hand keypoints in the generated frames are easier to extract and more accurately localized. The paper also notes that occluded fingers, object surfaces, and head movements in third-person videos can correspond to multiple plausible first-person outcomes. The model generates an inference rather than a true measurement of unrecorded information.
This further clarifies how Vbot differs from existing research. NVIDIA's Grounded-Exo2Ego focuses primarily on generation quality under geometric and semantic constraints. Work from Amazon and Stony Brook University bypasses pixel generation entirely, directly recovering hand-object poses and reporting policy-level results. Vbot, by contrast, retains the first-person video as an intermediate representation while incorporating hand and semantic supervision. Currently, Vbot's public evaluations focus on generation and hand extraction, with policy-level results yet to be disclosed.
Nevertheless, this approach of delegating view correspondence to a generative model remains highly valuable. Videos that previously could not be used directly as first-person training material can now be transformed by the model. Furthermore, the generation process need not solely pursue visual realism; it can also be optimized to preserve the specific motion information required for robot learning.
When foundation models start producing robot data
However, view transformation is never the final solution. Human videos record manipulation processes but typically lack the joint angles, control commands, and contact force information required by robots. Furthermore, morphological differences between humans and robots—such as variations in body structure, degrees of freedom, and joint constraints—prevent human motions from being directly replicated on robotic hardware.
A significant gap remains between observing someone else's manipulation and executing the task oneself.
In May this year, Vbot founder Yinan Yu told Phoenix Tech in an interview that while the first-person perspective is the currently popular approach for data collection, the third-person perspective is equally important. He reasoned that humans naturally learn by observing the actions of others and then practice from their own egocentric viewpoint, meaning the two perspectives should be combined.
Vbot's approach is to build a unified pipeline that makes diverse data usable for embodied AI. -xo2EgoPolicy first converts third-person videos into first-person visual frames; subsequently, the Vbot Echo pipeline extracts hand skeletons from these frames, retargets the motions to the robot's kinematics, and verifies physical feasibility in simulation.
This end-to-end process executes a multi-stage transformation: converting third-person videos into first-person observations, extracting human motion representations, retargeting those motions to the robot, and verifying physical consistency.
This aligns with Vbot's underlying modeling philosophy. In a previous interview with GeekPark, Yinan Yu posited that in embodied AI, the world model can be trained separately from the motion control and manipulation models, and the two can later be fused via latent space tokens.
A similar division of labor emerged as a recurring topic of discussion at the recently concluded CoRL 2026.
At the conference, Jim Fan, head of the NVIDIA GEAR team, introduced GR00T 2. Positioned by NVIDIA as System 1, this world-action model jointly predicts future video frames and robot actions. It operates in coordination with a higher-level reasoning model, which handles task decomposition, process monitoring, and error correction, while GR00T 2 executes the specific physical manipulations.
Hang Zhao, a professor at Tsinghua University, explained the necessity of this hierarchical structure across different timescales. In his framework, System 2 handles relatively slow-paced comprehension and planning; System 1 responds to physical interactions at frequencies of tens of hertz; and System 0 takes on the even higher-frequency tasks of contact feedback and low-level control.

Similar concepts have already been integrated into product architectures. In Figure's Helix 02, released this year, System 1 outputs whole-body joint targets at 200 Hz, while System 0 handles low-level control at 1 kHz.
While the specific technical architectures vary across teams, they all point to a central challenge: as general models become increasingly adept at understanding tasks and formulating plans, translating that understanding into fast, reliable physical execution is becoming the key to advancing robotic capabilities.
Yu Yinan recently drew an analogy to autonomous driving. In his view, System 2 handles high-level planning and System 0 manages low-level execution, but the intermediate System 1 is the most challenging. It must interpret high-level intent, execute specific motion planning, and continuously monitor the execution process to promptly detect and correct deviations.
He even described System 1 as the ultimate fallback layer within the robotic system.
But where do System 1's capabilities come from?
Environmental understanding and state prediction can be supervised using large-scale human videos. However, enabling a robot to actually execute actions requires bridging the gap between visual observations and physical movements. This means training data must go beyond mere video frames to include as much information as possible about hand kinematics, object states, and interaction dynamics.
This also explains why Vbot did not stop at first-person video generation, but instead went further to recover action and interaction information from the footage. Its goal is to bridge not only the viewpoint gap but also the information deficit between ordinary videos and robotic training data.
More notably, in this framework, generative models are taking on a portion of the data production workload by converting previously unusable content into new training material.
Another study drawing attention at CoRL 2026, the joint research project GRAIL by NVIDIA and UCLA, takes a different approach: it uses controllable 3D scenes and video world models to generate interaction sequences, then converts the resulting human-object motion into robot training data subject to physical validation. Vbot, by contrast, does not generate new manipulation scenes from scratch. Instead, it starts from existing third-person videos, leveraging generative models and downstream pipelines to extract previously underutilized manipulation information.
Vbot Echo is now fully open. Any team can integrate into this pipeline to convert their own third-person videos into robot training data, or use the Echo App to collect real-world scene data and feed it into this production system.

However, cross-view generation requires inferring unobserved information, and hand motion recovery can introduce errors. Furthermore, motion retargeting and physical validation add to the computational and processing costs. How much of this multi-stage transformed data will ultimately be viable for robot training still requires practical validation.
The true appeal of this approach lies in its long-term scalability.
Compared to professional data collection—which requires continuous investment in equipment, personnel, and physical venues—existing third-person videos already cover a vast array of scenarios and tasks. If models can increase the utilization of this existing video corpus at a controllable cost, they could open up new avenues for robot data production, even without replacing professional collection. Existing videos could become a viable data source, and hard-to-access environments like private homes could potentially be covered using ordinary smartphones.
This also clarifies the rationale behind Vbot’s current investment in data pipelines. It likely explains, too, why overseas researchers are beginning to rigorously test this model developed in China.
As the industry increasingly prioritizes translating understanding into execution, the value of action and interaction data becomes ever more apparent. While third-person videos are not yet a mature, low-cost data source, they offer significant long-term scalability beyond the limits of professional collection.
Scaling robot data may no longer be just about getting more people to pick up a camera. It also means transforming the massive volume of already recorded human manipulation into experience that robots can actually learn from.
Found an error? Tell us