Peking University Team Proposes Open-HumanWAM to Train Robot Foundation Models on Pure Human Video Data

On November 5, researchers from Peking University, the National University of Singapore, the Massachusetts Institute of Technology, the University of California San Diego, and NVIDIA introduced Open-HumanWAM, a framework that transfers manipulation experience from large-scale human videos to robot tasks through a standardized action representation and pre-training recipe.

Human videos contain extensive footage of picking, storing, and organizing objects. How can robots learn actionable manipulation experience from this data?

On November 5, Peking University, in collaboration with the National University of Singapore, the Massachusetts Institute of Technology, the University of California, San Diego, and NVIDIA, published the paper "Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models." The paper introduces Open-HumanWAM, an embodied model training framework that explores how to transfer manipulation experience from large-scale human videos to robotic tasks.

The paper has 34 authors, with Juncheng Ma, Jiaxin Li, Jiankai Tu, and Bo Liang serving as co-first authors. The corresponding author is Daquan Zhou, an assistant professor at the School of Information Engineering at Peking University.

In real-robot evaluations of HumanWAM-Vanilla, the team first pre-trained the model using 50,000 hours of human first-person video data, then fine-tuned it for specific tasks using robot demonstrations. Across four test scenarios involving changes to tablecloths and lighting, the model achieved an average success rate of 67.5%, compared to just 2.5% for the control group that was not pre-trained on human data. This experiment tests whether human manipulation experience can help robots adapt to environmental variations unseen during training.

In a separate experiment, HumanWAM-Goal takes a target image as model input to test whether the image can guide the robot to perform a specified manipulation. In zero-shot evaluations across 20 real-robot tasks, the model's average success rate improved from 20.0%—when using only robot data—to 30.5%. Following task-specific training, the model can also adjust its drawer selection, toy sorting, and notebook placement based on different target images.

How are human videos transformed into actionable manipulation experience for robots? The answer provided by Open-HumanWAM is to first pre-train on human videos, and then adapt to specific tasks using robot data.

Filtering 100,000 hours of video down to 50,000 hours of usable data

The research team first aggregated footage from five different sources, applied a three-stage filtering process to retain valid action clips, and converted all the data into a unified action representation before officially commencing pre-training.

The first step is data collection. The team combined five public datasets: HumanNet, EgoSuite-Open100K, EgoDex, Gen-HumanEgo, and EgoVerse. The raw data totaled 104,289 hours, or approximately 11.2 billion frames. HumanNet was the primary source, contributing 90,275 hours. It consists of filtered first-person videos from the internet, accompanied by 3D wrist poses and text descriptions.

In the second step, the data passes through three sequential filtering rules. The "hand visibility" filter projects the 3D end-effector position of the hand onto the image plane to determine if the hand is within the field of view. If only one hand is visible in a sequence, the clip is retained, but action supervision and normalization statistics are applied exclusively to that hand. The "motion and camera stability" filter identifies sudden trajectory changes; if the acceleration, jerk, or pose difference between adjacent frames exceeds a set threshold, the clip is truncated at that point. The "idle removal" filter uses a 32-frame sliding window to calculate end-effector displacement. If the hand barely moves within the window, it is classified as idle and that segment is removed. If the proportion of valid activity falls below 70%, the entire clip is discarded.

After the three filtering stages, nearly half of the original duration was removed, leaving a final set of 50,000 hours, totaling 5.4 billion frames and 35.93 million clips. Of these, HumanNet was reduced from 90,275 hours to 40,744 hours, while EgoVerse dropped from 143 hours to just 24.8 hours. EgoVerse experienced the largest reduction, reflecting significant disparities in annotation quality across the different datasets.

The third step is to unify the action representation. Human hands and robot grippers have inherently different structures, and the datasets record hand information in inconsistent formats. For the same pushing action, one dataset might represent it using a set of world coordinates, while another might use displacement relative to the previous frame. If these mixed formats are fed directly into a model, it cannot learn common action patterns.

The team mapped both human and robot hand trajectories into a unified 16-dimensional bimanual end-effector space. Each hand occupies 8 dimensions, comprising 3D position, quaternion orientation, and gripper openness, totaling 16 dimensions for both hands. Position and orientation are transformed into the camera coordinate system of the corresponding frame, with the camera axes uniformly defined as right, up, and back. The local wrist coordinate system is anchored to the base of the middle finger and the direction of the thumb; data from different sources are aligned using a fixed rotation matrix. For samples with 3D keypoints, orthogonal Procrustes analysis is used for estimation. For datasets like HumanNet, which only have monocular keypoints, alignment is achieved through projection axis and sign correction. Gripper openness is normalized based on the distance between the thumb and index finger tips, where 0 represents fully closed and 1 represents fully open.

This alignment scheme is the core design of this section. A single real-world physical action originally corresponded to several mutually incompatible numerical representations across the five raw datasets; after alignment, they share consistent geometric semantics, allowing demonstration data from multiple sources to be jointly used to train a single action prediction target.

With the datasets and standardized interfaces prepared, the most critical question remains: exactly how much model performance can this processed training duration yield?

How much does performance improve with every tenfold increase in data

The entire validation consists of seven sets of controlled experiments. First, it tests the correlation between data duration and task success rate, followed by sequential evaluations of the impact of data diversity, sample efficiency, action alignment, and action supervision. Subsequently, it directly compares human data with robot teleoperation data, and finally addresses the challenge of multi-view integration.

First set of experiments: measuring the scaling curve. The team adopted an isolated attention architecture, scaling the pre-training data from 100 hours to 10,000 hours. For each model iteration, they first performed post-training using 50 standard demonstrations from RoboTwin 2.0, and then ran 50 evaluation episodes in each of three environments: standard clean scenes, out-of-distribution scenes, and completely random scenes. In clean scenes, the success rate increased from 71.2% to 80.0%; in out-of-distribution scenes, it rose from 25.2% to 58.1%; and in random scenes, it grew from 8.4% to 28.7%. For reference, the baseline without human data pre-training achieved only 69.5%, 16.2%, and 2.1% across these three metrics, respectively.

The performance gap is primarily evident in scenes the model has never encountered. The success rates across all three tests exhibit an approximately linear relationship with the logarithm of the pre-training duration, yielding goodness-of-fit values of 0.971, 0.949, and 0.877, respectively. Extrapolating from this, every tenfold increase in pre-training data improves the success rate by 4.43 percentage points in clean scenes, 15.88 percentage points in out-of-distribution scenes, and 10.24 percentage points in random scenes.

Second set: verifying the conclusions by switching the attention mechanism. Inverse dynamics attention allows the model to refer to subsequent video frames to infer actions. Under this configuration, the improvement in clean scenes is minimal: 100 hours of pre-training corresponds to 87.4%, and scaling up to 50,000 hours yields only 89.0%; the baseline without human data pre-training still reaches 87.9%. However, in out-of-distribution scenes, the success rate rises from 49.1% to 73.2%, and in random scenes, it increases from 21.5% to 42.9%, with goodness-of-fit values reaching 0.969 and 0.955, respectively. Two entirely different attention architectures yield consistent scaling trends, indicating that the generalization benefits of scaling data are not tied to a specific model structure.

Third set: validating the value of data diversity. The team constructed two datasets based on EgoSuite-Open100K. One dataset continuously expanded its task and scene types as the duration increased, while the other kept the task and scene scope fixed, adding only different trajectories. For the same 100 hours of data, the fixed-scope version covered 2,288 tasks and 2,953 scenes, whereas the continuously expanding version covered only 62 tasks and 222 scenes. Ultimately, the scaling slope for the expanding group was higher by 2.02, 4.73, and 4.50 across the three tests, respectively. The broader the coverage of scenes and tasks, the steeper the performance curve as data scales.

Fourth set: testing sample efficiency. The team selected two pre-training tiers of 500 hours and 5,000 hours, pairing them with 5, 10, 25, and 50 demonstration samples for post-training, respectively. When using only 10 demonstrations, the success rate in clean scenes increased from 34.5% without pre-training to 44.2% and 50.1% sequentially. For the model pre-trained on 5,000 hours, using just 25 demonstrations achieved a 42.0% success rate in out-of-distribution scenes, surpassing the 37.9% achieved by the 500-hour pre-trained model paired with 50 demonstrations. This demonstrates the effect of outperforming with only half the number of robot demonstration samples.

Fifth set: confirming the practical benefits of action alignment. The researchers used the same batch of 1,000 hours of clips. One set was uniformly mapped to a standard action space, while the other retained the native action representations of each dataset. After alignment, the success rate in clean scenes increased from 72.0% to 77.5%, in out-of-distribution scenes from 43.2% to 52.2%, and in random scenes from 18.0% to 26.4%. During the pre-training phase, the action loss decreased significantly, while the video reconstruction loss remained largely unchanged.

Sixth set: determining the necessity of adding human hand action supervision. Using the same 1,000 hours of video, one group performed only video prediction, while the other added hand action supervision to half of the clips. After training for 20,000 steps, the success rate in clean scenes rose from 70.4% to 75.2%, in out-of-distribution scenes from 13.0% to 34.8%, and in random scenes from 3.6% to 16.3%. When training continued to 30,000 steps, the model performing only video prediction showed almost no further improvement, whereas the model with action supervision continued to improve.

Group 7: Direct comparison between human video data and robot teleoperation data. At the 100-hour scale, robot data slightly outperforms in out-of-distribution scenarios; however, the trend reverses after 500 hours. With 5,000 hours of human data, the model leads by 7.3 percentage points in clean scenarios, achieves 52.3% versus 45.6% in out-of-distribution scenarios, and 27.6% versus 22.4% in random scenarios. A model trained on just 1,000 hours of human data outperforms the model trained on 5,000 hours of robot teleoperation data across all three test sets.

Multi-view input is the final hurdle. If two additional wrist cameras are directly added to the robot during the post-training stage, under an isolated attention setting, the success rate in out-of-distribution scenarios drops from 46.1% to 33.0%–34.1%, and in random scenarios from 22.5% to 12.9%–14.8%. To address this, the team supplemented the training data with an intermediate dataset from EgoPro, which includes synchronized footage from both head-mounted and wrist cameras. The model takes all three camera views as conditional inputs but only predicts future video from the head-mounted perspective. After this adjustment, the success rate in out-of-distribution scenarios recovers to 61.2%, and in random scenarios reaches 30.7%, surpassing the single-view version's 28.6%.

A key experimental design is embedded in this series of controlled experiments. For each dataset, model checkpoints are saved every 10,000 steps to run the complete post-training and evaluation; furthermore, the best-performing checkpoint is selected independently for each type of test environment. The team found that the checkpoint that performs best in clean scenarios is not necessarily the best fit for random scenarios. Selecting the optimal results separately is the only way to clearly distinguish the two different scaling curves for in-distribution and out-of-distribution scenarios.

Upon completing this entire set of validations, this training recipe can be used to train the official foundation model.

Training an independent foundation model with pure human data

The foundation model is built upon Wan2.2-TI2V-5B, supplemented by an action expert module with a depth matching the video backbone network; together, they form a hybrid Transformer. The model employs inverse dynamics attention to facilitate interaction between action and video information: during training, a noisy latent variable of the real future video serves as a condition for the action module to perform denoising; during inference, the model first generates the future video latent variable and then relies on it to predict an action chunk containing 32 steps. In real-world deployment, the system executes the first 24 steps before replanning the subsequent actions.

The pre-training stage uses only the Open-HumanWAM-50K dataset. The action chunk length is set to 32, and videos are sampled at a rate of one frame per four action steps, resulting in a total of nine frames per sample. The AdamW optimizer is used with a global batch size of 8,192; the learning rate is linearly warmed up to 1e-4 over the first 5,000 steps, followed by a cosine decay down to 1e-6, for a total of 100,000 training steps.

Multi-view input capability is integrated through an intermediate training stage. During pre-training, only the head-view footage is input; subsequently, training continues on the EgoPro dataset, where current observations from all three cameras are used as conditional inputs, but the model still only predicts future video from the head-view perspective, before entering the robot-specific post-training pipeline. Specifically, RoboTwin is trained for 24,000 steps using 2,500 demonstration samples, and RoboDojo is trained for 50,000 steps using 3,500 demonstration samples; in real-robot experiments, each task is allocated 50 demonstrations and trained for 10,000 steps.

The research team conducted multiple sets of ablation experiments on a 1,000-hour dataset, yielding two counterintuitive results. By changing the action representation from relative displacement to absolute pose in the camera coordinate system, the success rate in clean scenarios improved from 83.56% to 88.48%, and in out-of-distribution scenarios from 58.08% to 62.00%. A comparison of the initialization methods for the action expert module showed that initializing via interpolation from the video weights of Wan2.2 resulted in success rates of 89.56%, 83.20%, and 51.52% for clean, out-of-distribution, and random scenarios, respectively. In contrast, random initialization achieved corresponding metrics of 90.28%, 83.92%, and 54.12%. The final official version adopted the random initialization scheme.

Real-robot experiments were conducted on the Cobot Magic dual-arm platform, featuring four tasks covering multi-step pick-and-place, deformable objects, articulated objects, and long-horizon tidying: paper cup stacking, towel folding, drawer organizing, and desktop tidying. Each task was repeated 10 times under each test condition. Out-of-distribution tests introduced environmental variations by simultaneously changing the tablecloth and lighting.

The simulation section presents a compelling set of benchmarking data. On RoboTwin 2.0, the model achieved a Clean2Clean success rate of 92.08% and a Clean2Random success rate of 68.88%, with an average of 80.48% across the two; the Clean2Clean metric ranks in the highest tier of the comparison table. For comparison, OpenWAM-α scored 89.4% and 48.7%, while PICA-WAM scored 90.68% and 74.70%. GO-3 Nano Preview achieved 81.48% under random conditions, maintaining the lead in this specific out-of-distribution metric.

On the RoboDojo benchmark, the model achieved an average success rate of 14.17% across six dimensions and a progress score of 19.00, outperforming several world-action model baselines in the table and approaching the 14.60% and 20.10% of Galaxea G0.5. Its advantages are concentrated in precision and long-horizon tasks, with corresponding success rates of 20.50% and 33.00%, whereas OpenWAM-α achieved only 9.25% and 25.33% in these two areas. The model's shortcomings lie in memory and open-instruction tasks, where DM0.5 performs better, reaching 47.44% in the memory dimension.

The real-robot results are equally impressive: the average in-distribution success rate increased from 57.5% to 87.5%, and the out-of-distribution success rate rose from 2.5% to 67.5%. Breaking this down, the out-of-distribution success rate for paper cup stacking jumped from 0% to 80%, drawer organizing from 0% to 70%, towel folding from 10% to 60%, and desktop tidying from 0% to 60%. As a baseline, a model without pre-training on human data succeeded only once in 40 out-of-distribution trials. Merely changing the tablecloth and adjusting the lighting clearly demonstrates the value of the prior knowledge learned from human manipulation.

Goal image guidance for image-driven manipulation

A foundation model trained solely on human data still relies on action demonstrations for each specific task. However, if robot-collected data is mixed into the training set and a goal image is used to specify the desired final state, the model can directly execute new tasks.

The research team first expanded the action space: in addition to the original 16-dimensional bimanual end-effector representation, they added 23 dimensions for joints, 3 dimensions for the mobile base, and 7 dimensions for the head pose, combining them into a 49-dimensional hybrid action representation. The gripper opening was changed to an absolute physical aperture, normalized using a baseline width of 0.12 meters; for human samples, the distance from the thumb to the index finger was directly used as the aperture size. The end-effector displacement was adjusted to an intra-block relative displacement anchored to the current observation, projected into the head camera coordinate system at the anchored time step, eliminating the need to recalibrate parameters when switching the robot body. For joint and base dimensions missing in the human data, training relied on a validity mask to ignore the corresponding loss. During the robot training phase, the joint branch was also randomly masked to force the shared end-effector action pathway to stably learn generalizable patterns.

The main model is replaced with Cosmos3-Nano-16B. The language component uses a frozen Qwen3-VL Reasoner, while a single generator with shared parameters handles both visual and action outputs. The target image is encoded separately and augmented with a learnable target embedding, fed into the generator as an independent conditional token to distinguish it from real-time observations. During inference, the video latent variables and the 48-step action chunk are denoised synchronously using joint rectified flow sampling.

This design eliminates the overhead of serial sampling. A purely language-guided approach requires generating a complete future video before outputting actions based on that video, creating a two-step serial process. In contrast, the target image already locks in the target spatial configuration. The research team merges these two steps, directly halving the sampling latency. The target image serves as both the task instruction and the anchor for the entire causal chain of actions, eliminating the need for the model to hallucinate all future frames before initiating execution.

The training materials are divided into multiple comparative groups. The robot data scale is progressively expanded from 2,005 hours to 4,261, 8,475, and 14,618 hours, covering eight real-robot scene domains with a total of 1.1 million interaction episodes. Human data is fixed on top of the 14,618 hours of robot corpus, with an additional 15,901 or 45,938 hours of human video overlaid, bringing the total data volume to approximately 30,500 and 60,600 hours, respectively. The target images employ an online sampling strategy: 40% are sampled from frames 2 to 8 seconds ahead, 20% from the final frame of the episode, 20% from the final frame of the next subtask, and the remaining 20% are set as empty targets, with the tokens corresponding to empty targets directly discarded.

Zero-shot testing is first conducted in a simulation environment. Without any additional adaptation for RoboTwin, the version trained solely on robot data achieves an average success rate of 11.48%; after mixing in human data, the metric improves to 13.94%. The number of tasks covered by a single weight checkpoint increases from 8 out of the original 50 to 22 after mixed training, covering a cumulative total of 29 tasks. Most models in the comparison baselines cannot even run this evaluation, lacking action normalization statistics, morphological mapping, and usable pre-trained weights; among those that can complete the test, π0.5 achieves only 1.06%, and Pelican-VLA 0.5 reaches 4.70%.

After post-training, the advantages of mixed-data training continue to manifest. As the robot data increases from 2,005 to 14,618 hours, the success rate in clean scenes rises from 87.24% to 89.84%, and in random scenes from 59.88% to 70.80%. However, the gains in random scenes plateau after 8,475 hours, with 70.96% and 70.80% being nearly identical. Incorporating human data breaks this growth bottleneck, further pushing the random scene success rates to 71.68% and 74.36%, while clean scene performance remains stably around 89%.

Combined with photometric augmentation and post-training recipe fine-tuning, this version achieves 92.70% on Clean2Clean and 84.60% on Clean2Random on the leaderboard, with a two-item average of 88.65%. This is 3.64 percentage points higher than the previously leading GO-3 Nano Preview, outperforming it by 4.16 and 3.12 percentage points in clean and random scenes, respectively.

Real-robot experiments switch to the Astribot S1, a robot equipped with a wheeled base and a movable torso. A total of 20 tasks are set up, divided into 10 easy tasks and 10 hard tasks, with each task tested repeatedly for 10 trials. The zero-shot average success rate increases from 20.0% with pure robot data to 26.0% and then 30.5%. Among these, easy tasks improve from 32.0% to 42.0%, and hard tasks rise from 8.0% to 19.0%. Hard tasks tend to hit a bottleneck under the default sampling ratio, as human clips account for as much as 76% of the training mixed samples. After forcibly adjusting the batch ratio to 1:1, the hard task success rate jumps from 19.0% to 27.0%. The overall progress score increases from 2.535 to 4.205, and the number of tasks achieving at least one successful trial grows from 10 out of 20 to 16.

The most valuable capability is image-guided in-context learning. Using the same set of frozen weights and simply changing the target image, the robot can place a bottle on the desk in front of a green block or on top of a purple block. This capability is preserved after post-training and supports the input of multiple target images: it can swap the trash bins corresponding to a toy car and a plush toy, or simultaneously modify the stacking order and placement of books, with the entire action sequence adjusting accordingly. Even with target images taken by ordinary people that show human hands in the frame, the robot can still comprehend the spatial relationships in the image and place objects precisely, such as first putting a lid on a coffee cup and then delivering it next to a monitor or beside a file rack.

The model is also capable of executing long-horizon tasks. In a desktop organization task, it completes 14 consecutive stages: placing a lid on a cup, organizing stationery, opening and closing two drawers to store a tape measure and tape, and inserting a notebook into a bookend, all without disturbing previously arranged items. A home organization task spans multiple areas: the robot first organizes toys and books at the main desk, moves to a side table to discard empty bottles, collects clothes from two chairs into a laundry basket, and finally returns to the desk. This entire sequence is executed across 28 keyframes.

Open-HumanWAM distills raw first-person human video from 100,000 hours down to 50,000 hours. By leveraging a standardized action representation, it integrates five diverse datasets to establish a reproducible training paradigm. The research team measured in a closed-loop simulation environment that the model's downstream task success rate follows a log-linear scaling law with respect to pre-training data duration. Trained under this framework using exclusively human data, HumanWAM-Vanilla achieves competitive results on the RoboTwin 2.0 and RoboDojo benchmarks after post-training. Furthermore, the model maintains stable manipulation performance in real-robot tests even when tablecloths are changed and lighting is adjusted.

The study ultimately draws two core conclusions.

At the data level, the performance scaling curve maintains a steeper slope only when the coverage of tasks and environments is expanded simultaneously. At the data source level, human video provides broad interaction priors, while robot-collected data supplies the specific control details for the corresponding embodiment. By properly balancing these two types of samples within training batches, performance can continue to improve for both zero-shot generalization and post-training fine-tuning. The team has also released the dataset, training code, model weights, inference scripts, and a complete suite of simulation evaluation tools to enable future research to build upon this work.