Zongqing Lu’s Team at BeingBeyond Introduces Being-M0.7 for Humanoid Whole-Body Mobile Manipulation

Relying on over 10,000 hours of human first-person videos and body motion data, Being-M0.7 learns joint priors for visual frames and full-body movements. It then performs mid-training on robot trajectories and post-training on actions to output control commands for humanoid full-body locomotion and manipulation. The model achieved a comprehensive success rate of 71.1% in the SIMPLE simulation benchmark, ranking first among baselines; in real-robot tests on the Unitree G1, its performance matched that of the strongest baseline.

On October 8, the BeingBeyond team published the paper "Being-M0.7: A Latent World-Action Model for Humanoid Robots." The paper has 13 authors, with Junpeng Yue, Boyuan Li, and Yuxuan Wang as co-first authors. The corresponding author is Zongqing Lu, founder of BeingBeyond and tenured associate professor at the School of Computer Science, Peking University.

For humanoid robots to perform full-body locomotion while manipulating objects, they must perceive the current environment, predict future scenes and their own movements, and coordinate all limbs. The scarcity of real-robot demonstration samples remains the core bottleneck in this field.

Human first-person videos and motion data can supplement samples at scale, but the data modalities are inconsistent: some samples contain only video, others only motion sequences, and human motions cannot be directly transferred to robots for execution.

Being-M0.7 adopts a "learn from human data, then transfer to robots" approach: during pre-training, video and motion are mapped to a shared representation space to jointly predict future visual latents and actions, embedding kinematic information into visual representations; during the robot mid-training phase, these learned priors are adapted to the robot's observation perspective and morphology; in the action post-training phase, an action expert generates full-body control commands based on the predicted context output by the frozen priors.

01 Unifying Human Video and Motion Representations

This work targets humanoid full-body locomotion and manipulation tasks. As robots move while grasping objects, they must simultaneously plan forward direction, torso rotation, and hand-foot coordination, maintaining body stability when reaching out to interact. Such tasks rely not only on real-time environmental observations but also on predicting future scenes and the robot's own motion state.

Collecting real-robot demonstrations is the core bottleneck in this direction. Acquiring synchronized first-person footage, proprioceptive states, and executable full-body control command trajectories requires complex teleoperation systems and is constrained by safety constraints, hardware limitations, and scene diversity. Human first-person videos and motion capture data can provide large-scale visual dynamics and full-body behavioral information, but the data modalities are not unified: some samples contain only video, others only motion sequences, and human motions cannot be directly converted into executable commands for humanoid robots.

The team organized the data into three types of supervisory signals based on over 10,000 hours of raw human-centric data, prior to filtering and time-slice processing:

• Video-action paired samples: Supervised joint prediction of future frames and full-body actions

• Pure video samples: Supervised visual dynamics in scenarios

• Pure action samples: Supervised full-body kinematic structure

Paired samples provide cross-modal joint supervision, while unpaired samples supplement marginal supervision for each respective modality, enabling a training coverage far greater than approaches that rely solely on paired corpora. The dataset sources include publicly available datasets such as Ego4D, Xperience, Nymeria, Bones-SEED, SnapMoGen, HumanML3D, and Lafan1, along with subsets from prior work on Being-H0.5 and Being-M0.5, as well as commercially procured data.

The research designs a shared, compact action representation for humans and robots. This representation is a 22-dimensional vector with the head as the root node: it includes 3 dimensions for root translation, 6 dimensions for continuous 6D root rotation, 1 dimension for initial head height, plus 3 dimensions each for the local positions of the left hand, right hand, left foot, and right foot, totaling 12 dimensions. This representation is extracted from human motion capture data and reconstructed on the robot side via forward kinematics from joint quantities, enabling both humans and robots to share the same representation interface. The temporal dimension of the data is downsampled to 5 Hz; action channels are normalized per channel according to the mean and standard deviation of each dataset, with standard deviations below 10⁻⁵ replaced by 1.

The vision branch does not process raw pixels directly; instead, it employs a frozen DINO encoder to convert images into latent variables. These latent variables serve as both the model's input and its prediction target. By eliminating the need to reconstruct pixel-level details, the model concentrates computational resources on scene semantics, object layout, and the temporal evolution of the scene. During pre-training, the model jointly predicts future visual latents and actions. Visual features must simultaneously encode changes in imagery and underlying kinematic shifts, which is key to transferring human visual-motor priors to downstream robotic control.

02 Joint Future Prediction with a Mixture of Transformers

The prior model must process image and action modalities simultaneously. The team employs a Video-Action Mixture of Transformers (MoT) architecture for joint modeling. Visual latent variables and action tokens undergo learnable input projection, temporal positional embedding, and modality embedding to map into a unified latent space. Task instructions are encoded independently by a frozen DistilBERT, with both visual and action branches reading instruction information via cross-attention.

Within each MoT module, the visual and action branches retain their respective layer normalization, QKV projection, and feed-forward networks, interacting only through shared attention modules. This allows visual state evolution and action dynamics to influence one another. Sinusoidal embeddings of the flow timestep τ modulate each modality’s self-attention, language cross-attention, and feed-forward networks through modality-specific adaptive layer normalization. Block attention masks are applied between modules: observed historical tokens interact only within the historical sequence, unable to access future information; noisy future tokens can read both history and other future tokens within the same chunk, enabling parallel denoising of the entire future sequence.

The architecture explicitly handles missing modalities. The model enforces a fixed interleaved arrangement for visual and action tokens: pure video samples replace action features with a learnable "missing action" embedding, while pure action samples replace visual features with a "missing image" embedding. Modality presence indicators, combined with temporal validity masks, restrict access to valid tokens: valid queries cannot read keys/values from missing modalities, and invalid queries interact only with themselves. Their outputs participate neither in supervised calculation nor in feeding the action expert. During post-training and deployment of the action expert, all action presence indicators are turned off, action states are initialized to zero and receive no further flow updates; the system only integrates and caches future visual states.

The action expert is an independent flow-matching Transformer, decoupled from the prior model. This module receives noisy action tokens and current observation information: first-person images in observations reuse the same frozen DINO encoder, while proprioceptive states are encoded via learnable projection. The expert first fuses action and observation tokens using local self-attention, then reads the frozen prior’s latent states via gated cross-attention at specified network layers, injecting prediction context containing kinematic information into the multi-layer network. Information flows unidirectionally; action tokens do not participate in the prior model’s computation.

The gating mechanism operates on individual tokens. An action token obtains feature x after local observation processing at a specific layer of the expert, with its world cross-attention projection update amount denoted as r. The model scales this residual term using a sigmoid-gated coefficient g, multiplies it by the scale provided by the flow condition, and adds it back to the action token. This mechanism allows each token to autonomously decide how much prediction context to absorb, while the action token’s reading of current observations remains guaranteed by local self-attention. The prior model comprises 30 layers with a hidden width of 768 and 24 attention heads; the action expert also has 30 layers, a hidden width of 1,024, and 32 attention heads. Connections between the expert and corresponding layers of the prior model are established at layers 3, 6, ..., 30, with the action sequence block length set to 30.

03 Three-Stage Training Followed by Closed-Loop Inference

The overall training process consists of three stages: pre-training uses human data to learn coarse-grained video-action priors; robot mid-training adapts these priors to the robot’s observational perspective and proprioceptive motion characteristics; and action post-training trains the action expert on top of the frozen adapted prior to output full-body control commands executable by the robot.

The pre-training phase employs a flow-matching scheme that adds noise to clean future tokens, enabling the model to predict target velocities. For video-action paired samples, visual and action branch losses are computed simultaneously; for pure video or pure action samples, only the corresponding modality’s loss is activated, providing targeted supervision based on available modalities. The action branch incorporates two additional geometric auxiliary losses: one constrains the decoded global head displacement trajectory, and the other constrains the local end-effector velocities of the left/right hands and left/right feet, with weights set at 0.1 and 1.0 respectively. These geometric losses are calculated only for valid future frames within motion tokens; if no valid indices exist, the loss is zeroed out.

Mid-training and post-training reuse the same robot trajectory dataset. Mid-training adapts the priors to robot-specific video, action sequences, and task text, using a combined loss of visual and action components. The action loss is computed only when trajectory data includes associated action information. Through a unified action representation, the priors learned during pre-training transfer effectively to robot scenarios and body dynamics. After mid-training completes, the prior model’s weights are frozen, and the process moves into the post-training stage with action supervision.

In the post-training stage, the prior starts from Gaussian noise and generates future visual sequences via 3-step integration of the flow ODE, disabling explicit action prediction. Along this denoising trajectory, hidden states from specified network layers of the prior are extracted and fed as predictive context into an action expert, which also receives current observations. Training the expert on these self-generated predictions aims to reduce the conditional distribution gap between training and inference. The action expert applies noise to action chunks, optimizing using flow-matching squared error as the loss function. The frozen prior model does not receive gradient updates from action losses in either its visual or action branches.

During inference, the prior generates low-frequency future visual plans and caches intermediate hidden states, serving as key-value pairs for the policy module. The action expert reuses this cache across multiple action queries. Each query allows the expert to read the latest image and proprioceptive state, applying gated caching to denoise action blocks. The cache updates before context exhaustion, separating low-frequency scene and full-body intent prediction from high-frequency responsive control generation. Temporally, each prior window comprises 1 observation frame plus 24 future frames, yielding a 5 Hz prediction rate covering 4.8 seconds. The underlying controller executes commands at 50 Hz. Each action block contains 30 steps, lasting approximately 0.6 seconds, with new queries triggered at roughly 1.67 Hz. Regarding sampling steps, the prior uses 3-step sampling during post-training, while both the prior and action expert use 10-step sampling during inference.

Engineering configuration utilizes 8 GPUs across all three training stages. Single-card batch sizes are 256, 64, and 64, resulting in effective batch sizes of 2,048, 512, and 512. Learning rates are set sequentially to 2 × 10⁻⁴, 10⁻⁴, and 10⁻⁴. The optimizer is AdamW with weight decay of 0.05, gradient clipping threshold of 1.0, and FP16 mixed precision enabled. Post-training initializes from the 50,000-step checkpoint of mid-training, with DINO and DistilBERT kept frozen throughout the entire pipeline.

04 Validation Results in Simulation and on Real Hardware

Evaluations were conducted on the SIMPLE simulation benchmark and the Unitree G1 real-robot platform. All comparative baselines and ablation variants employed a unified multi-task training configuration, ensuring fair evaluation conditions and credible results.

The real-robot test platform was built on the Unitree G1 humanoid robot, featuring 7 degrees of freedom per arm equipped with Linker Hand O6 dexterous hands. An Intel RealSense D435i camera mounted on the head captures first-person observational footage. Model inference runs on a single NVIDIA RTX 4090 GPU, while the underlying controller interacts with the policy server in a closed loop via the ZMQ protocol.

Real-robot training data was collected using a complete VR teleoperation system: operators wore PICO VR headsets, ankle-tracking devices, and hand controllers to input full-body motion commands. XRoboToolkit parsed the VR data stream in real time, solved for the operator’s SMPL human pose, and then used the SONIC module to output stable 29-degree-of-freedom (DoF) full-body control commands, driving the G1 robot at 50 Hz. Throughout data collection, first-person RGB footage, robot body states, and complete full-body commands including hand controls were recorded synchronously, forming a high-quality real-robot trajectory dataset.

All baseline models and ablation variants adopted a unified multi-task evaluation scheme: in simulation, they underwent joint training on six tasks from the SIMPLE benchmark, with a single model evaluated across all tasks; in real-robot scenarios, they underwent joint training on three full-body manipulation tasks, with a single model adapted to all test tasks. The SIMPLE simulation benchmark includes six tasks: BendPick, Handover, Mobile P&P, Grasp, XMoveBendPick, and XMovePick. Each task featured three difficulty levels—easy, medium, and hard—with 10 repetitions per level, resulting in 180 simulation trials per model.

Simulation evaluation results showed that Being-M0.7 achieved the highest overall success rate, succeeding in 128 out of 180 trials for an aggregate success rate of 71.1%, significantly outperforming mainstream baselines under the same conditions: Ψ0 reached 64.4% (116/180), π0.5 reached 52.8% (95/180), and GR00T-N1.6 achieved only 33.9% (61/180). In task-specific breakdowns, the model cleared all three difficulty levels in Handover (9/9/10) and performed excellently in Grasp and XMovePick, scoring 10/10/8 respectively. However, the model exhibited targeted weaknesses: cross-difficulty summaries indicated that Ψ0 had an advantage in BendPick, while π0.5 performed better in Mobile P&P and XMoveBendPick.

The study conducted comprehensive ablation experiments under the same six-task, three-difficulty-level simulation framework, with each variant completing 180 control trials to precisely validate the effectiveness of core modules. Removing human-data pre-training caused the model’s aggregate success rate to drop sharply from 71.1% to 48.9% (88/180), a decline of 22.2 percentage points. Removing the mid-stage robot adaptation phase reduced success to 59.4% (107/180), a drop of 11.7 percentage points. Replacing the prior model’s predicted context with pure task-language features lowered success to 58.3% (105/180), a decrease of 12.8 percentage points.

Performance bottlenecks for two key ablation variants concentrated in the BendPick task, where success rates plummeted from the original model’s 22/30 to 0/30 and 2/30 respectively. The pure-language-feature variant underperformed in four of the six tasks, with the most significant gap in XMovePick, where the original model’s 28/30 success rate far exceeded the variant’s 8/30. The sole exception was XMoveBendPick, where the language-feature variant performed better (21/30 vs. 9/30); the team attributed this to high visual similarity across multiple tasks causing visual feature interference, which temporarily allowed pure language conditioning to gain an advantage.

Real-robot evaluation focused on three high-difficulty full-body mobility-and-manipulation tasks: Walk to Fish (W2F), Walk to Mirror (W2M), and Walk to Doll (W2D). Each task was tested 5 times, totaling 15 trials per model. Robot starting positions and table locations were fixed, while target objects were randomly placed in left, center, or right positions. Each trial had a 30-second time limit, with success or failure determined by standardized human judgment: W2F required the robot to scoop up a toy fish using a net, while W2M and W2D required complete grasping and transfer of the corresponding dolls.

Real-robot testing results showed that Being-M0.7 succeeded in 13 out of 15 trials, achieving an 86.7% success rate, fully matching the performance of the current strongest baseline, GR00T-N1.6, and significantly outperforming π0.5 (7/15) and Ψ0 (3/15). Task-specific performance aligned completely with top-tier baselines: W2F succeeded 3/5, W2M 5/5, and W2D 5/5, demonstrating strong stability in precise dual-hand manipulation and coordinated mobile grasping scenarios.

The team also clarified the model's current limitations: this evaluation only covers short-horizon whole-body locomotion and manipulation tasks; longer temporal sequences and more complex continuous whole-body interaction scenarios have not yet been validated. Additionally, the end-effector tracking accuracy of the existing SONIC whole-body controller has an upper limit, which increases the difficulty of collecting high-precision real-world demonstration data to some extent and restricts the model from achieving its ultimate performance.