A joint team from Peking University, Tsinghua University, and other institutions has proposed HAF-VLA, a hierarchical framework that enables humanoid robots to generate coordinated full-body actions by sequentially planning movements along the kinematic chain to avoid compensatory instability.

Shanghang Zhang et al. propose HAF-VLA to enable humanoid robots to generate coordinated full-body motions hierarchically

For a humanoid robot to autonomously walk to a target, bend down, and reach out to grasp an object, the bottleneck lies in the coordinated generation of full-body motions. The legs, torso, head, and arms are connected in a single kinematic chain: if the base shifts by a few centimeters or the torso angle differs by a few degrees, the arms must extend further to compensate. Continuous compensation makes upper-body motions increasingly stiff and progressively degrades stability. Vision-language-action models deployed on fixed bases or wheeled platforms do not face this issue, but when these solutions are transferred to humanoid robots, errors amplify sequentially along the kinematic chain.

On October 19, a joint team from the National Key Laboratory of Multimedia Information Processing at the School of Computer Science, Peking University, the Beijing Humanoid Robot Innovation Center, Nankai University, Xi'an Jiaotong University, and Tsinghua University published the paper "HAF-VLA: Hierarchical Action Flow Framework for Humanoid Vision-Language-Action Control." The paper has 15 authors, with Langzhe Gu, Chengkai Hou, and Meng Li as co-first authors, and Jian Tang, Shanghang Zhang, and Zhengping Che as corresponding authors. Shanghang Zhang is currently an assistant professor at the School of Computer Science, Peking University, focusing on open-world generalizable machine learning and computer vision. She previously conducted postdoctoral research at BAIR, University of California, Berkeley, and her work on the Informer model won the Outstanding Paper Award at AAAI 2021.

The core concept of vision-language-action (VLA) models is to process visual inputs and language instructions to directly output actions. RT-2 and OpenVLA transfer vision-language pre-training to robotic arm manipulation, while π0 and π0.5 adopt a flow matching mechanism, progressively denoising from random noise to generate a continuous action chunk in a single pass. This approach avoids repeated frame-by-frame inference, ensuring temporal coherence in actions. It performs robustly in tabletop and wheeled robot tasks, demonstrating strong generalization to novel objects and environments. World model approaches, such as Cosmos Policy, take a different route: they first learn environmental dynamics and then translate predictions into control signals via an action head. Despite their different methodologies, both approaches aim to establish an end-to-end pipeline from perception to action.

However, most research in the humanoid robotics field avoids directly modifying general VLA models. Some approaches learn skills in latent space and then stitch them together into full-body motions; others design cross-embodiment action representations to fuse datasets from multiple robots; and some teams rely on first-person human videos to train humanoid foundation models from scratch. Meanwhile, teleoperation research has accumulated extensive full-body motion data, with the focus placed on data collection interfaces and low-level control.

While each of these approaches has its strengths, they all share critical limitations: they either require fully retraining the model, or their action outputs remain in latent space and cannot be directly mapped to robot joint commands.

Directly applying general models is equally difficult to deploy in practice. Humanoid robots must simultaneously control locomotion, posture, gaze, and bimanual manipulation in a high-dimensional space. If a single denoising step outputs all 26 action dimensions at once, the legs, torso, and arms will conflict along the same trajectory, as there are no sequential constraints on the joint movements. A tiny shift in the base will tilt the torso, forcing the arms to extend significantly to compensate, which simultaneously degrades both motion accuracy and stability.

The team is targeting exactly this structural conflict: can action generation proceed in stages along the humanoid kinematic chain, rather than solving all dimensions simultaneously?

HAF-VLA reuses the vision-language prefix encoder and action expert weights of π0.5, breaking a single denoising step into three cumulative denoising passes.

The first stage activates only the legs, skill mode, and head; the second stage adds the waist; the third stage then enables the arms and hands.

After each denoising pass, the resulting action prediction is encoded into a KV cache and passed to the next stage for further refinement. Across six real-robot tasks, the model's average normalized score improved from 53.3 for π0.5 to 68.3.

Step first, then tilt the waist, and finally move the hands

HAF-VLA predicts a full chunk of 100 future steps in a single pass. The model's action dimension is 32, with 26 active dimensions: 14 dimensions for arm joint targets, 2 dimensions each for left and right gripper commands, 3 dimensions for chassis velocity, 2 dimensions for the head, 4 dimensions for the waist corresponding to roll, pitch, yaw, and height, and 1 dimension for skill mode; the remaining 6 dimensions are zero-padded.

All dimensions are normalized using the 1st and 99th percentiles of the teleoperation data. Training and deployment use the same forward and inverse transformations, compressing joint targets, gripper values, velocity commands, and skill commands into a unified numerical range.

The model does not directly output torque for the chassis and waist. Leg walking and waist posture are handled by an AMP-style reinforcement learning controller, while kneeling motions use a BeyondMimic-style motion tracking controller.

The skill mode dimension distinguishes among three states: walking, standing, and kneeling. The model predicts this as a continuous scalar, which is denormalized during deployment and mapped to three discrete modes: 0, 1, and 2. During the transition from walking to standing, the wrapper layer inserts an approximately 2-second zero-velocity pause to prevent abrupt mode changes. The upper body uses a position control interface, with the arms outputting target joint positions and the hands outputting normalized gripper values. The head policy interface is fixed at two dimensions: Tiangong 3.0 drives two degrees of freedom, whereas Tiangong 2.0 supports only pitch, and the platform wrapper directly ignores any excess dimensions.

The team partitions the 26-dimensional action space into three progressively expanding subspaces.

• The first stage activates only the legs, skill modes, and head, first determining the direction of travel and line of sight;
• The second stage incorporates the waist, adjusting posture and upper-limb reachability based on the established motion plan;
• The third stage finally activates both arms and hands to complete the full action sequence.

All three stages share the same action space and dimension mask, which is broadcast along the time dimension. Inactive dimensions are zeroed at the input, and their target velocities are set to zero. The loss function is averaged across all 32 dimensions, including those with zero targets. The model continuously learns within a unified space, eliminating the need to learn separate representations for each stage. The dimension ordering remains unchanged; deployment simply requires switching the mask without altering the action vector structure.

The action chunk is set to 100 steps, but only the first 40 are actually executed. This trade-off balances whole-chunk prediction capability with replanning frequency:

• If the action chunk is too short, frequent policy inference increases control loop latency;
• If the action chunk is too long, subsequent predicted steps become invalid as soon as the environment changes.

Under the configuration of 100-step prediction and 40-step execution, the three inference stages take approximately 0.12 seconds in total. A single inference supports 40 action steps, matching the walking and manipulation cadence of humanoid robots. The inference results from the first two stages are not executed; they serve only as conditions for the third stage, meaning the two additional denoising rounds do not bottleneck the execution pipeline.

The core mechanism linking the three stages is the cross-stage KV cache. The vision-language prefix is computed only once, and all three stages reuse the same image-text conditions. After each round of flow-matching denoising produces a clean action prediction, this prediction is paired with the stage embedding and fed back into the shared action expert to extract the self-attention KV cache, which serves as the input condition for the next stage. The cache must be extracted from the final denoised action of the previous stage, not from intermediate denoising results; only the former represents a complete, usable motion plan.

Ultimately, only the output of the third stage is executed, while the first two stages retain only intermediate representations of motion and posture. During training, this cache is re-encoded from the masked ground-truth action chunk, with gradients truncated at this point. During inference, it is replaced by the previous stage's prediction, ensuring the dependency structure remains consistent across both phases.

The three stages share a single action expert, using learnable stage embeddings to identify the currently activated subspace. The team avoided assigning separate experts to the legs, waist, and arms primarily due to data constraints: with only 120 demonstration samples per task, splitting parameters across multiple experts would leave each module undertrained. A shared expert allows the action representations from different stages to reuse one another.

The stage embedding marks the current stage, the vision-language cache holds perception and instructions, and the action cache records the established motion plan. By combining these three conditions, a single set of weights can generate outputs for each stage.

This approach differs fundamentally from methods that construct whole-body actions in latent space.

• Latent space approaches first learn skill encodings, where a higher-level module selects the encoding and a lower-level module decodes it into joint actions, keeping the policy output confined to low-dimensional latent variables;
• In HAF-VLA, all three stages directly output executable physical quantities such as joint positions, velocities, and gripper openings. Action representations are passed between stages, bypassing the decoding step and allowing the policy to interface directly with the low-level controller. The trade-off is that each stage must perform denoising in a 26-dimensional space, concentrating the computational cost on the action dimensions.

Later stages can make minor corrections to the dimensions determined by earlier stages, marking the boundary between "soft refinement" and "hard concatenation." During the denoising of the third stage, the leg and waist dimensions from the first two stages remain within the action space. This allows the model to fine-tune the parameters without significantly altering the overall trajectory.

Ablation studies show that freezing the earlier stages and preventing modifications by later stages drops the score to 66.7, while calculating the outputs of the three stages independently and directly concatenating them drops the score to 46.7. Allowing subsequent stages to fine-tune the existing plan outperforms both locking the plan and calculating the stages independently.

This temporal design stems from the kinematic chain properties of humanoid robots: the legs and waist first lock the base position and body height, allowing the arms to execute grasps within the established reachable workspace, which prevents the upper body from having to passively compensate for base deviations. Sequence ablations confirm this logic: prioritizing the waist or placing the arms in the first stage significantly reduces performance on the box-to-table task.

Building a hierarchical action framework based on π0.5

HAF-VLA is not trained from scratch. Instead, it builds on the π0.5 checkpoint from openpi, specifically utilizing the gemma_2b_lora branch. The PaliGemma vision-language backbone remains frozen, with training restricted solely to the LoRA parameters.

The LoRA rank is set to 16, alpha to 16.0, and the scaling factor to 1.0, with no dropout applied. LoRA adapters are attached to the query, key, value, output, and gating modules in both the attention and feed-forward layers, as well as the up-projection and down-projection modules. The action expert, based on gemma_300m, undergoes full-parameter training. The newly introduced phase embeddings, action input and output projections, and temporal MLP are also trained, while the vision encoder remains unfrozen in this configuration.

This adaptation strategy relies on the premise that π0.5 has already achieved weight convergence. The action expert acquires the ability to model continuous actions from massive mobile manipulation datasets, while the vision-language branch establishes semantic alignment between visual frames and text instructions.

HAF-VLA reuses these two sets of weights and introduces a new hierarchical phase structure to relearn the temporal allocation of actions, fully inheriting the visual perception and language understanding capabilities of the original model. Training updates only the LoRA and newly added layer parameters. The PaliGemma backbone and the core of the action expert remain unchanged, concentrating the optimizable parameter space entirely on the new structure.

The training uses the AdamW optimizer with β1 set to 0.9, β2 to 0.95, weight decay to 1e-10, and gradient norm clipping at 1.0. The learning rate follows a cosine decay schedule: a 1,000-step warmup, a peak of 2.5×10⁻⁵, a 30,000-step decay interval, and a final learning rate of 2.5×10⁻⁶. The global batch size is 64, and the default training runs for 50,000 steps. When constructing cross-phase caches during training, the model uses masked ground-truth action chunks rather than its own predictions, truncating gradients at the cache positions to stabilize the optimization process.

The temporal dependency structure during inference remains identical to that of training, with caches still drawn from the previous phase's output. Training uses ground-truth inputs while inference relies on model predictions; the two data sources differ, but the underlying structure must remain unified—once the cache source changes, the generated actions will diverge.

The losses from the three stages are simply summed. In each stage, the model regresses a velocity field over the full 100-step, 32-dimensional action sequence, normalized by the product of the batch size, sequence length, and dimensionality. Target velocities for inactive dimensions are set to zero, teaching the model to remain stationary in those dimensions. During deployment, when the mask switches, the outputs for the corresponding dimensions remain near zero, preventing any abrupt jumps during phase transitions. This unified objective function allows all three stages to share a single loss, eliminating the need to design additional loss terms for the newly introduced hierarchical structure.

The training data is collected via isomorphic teleoperation. The kinematic structure of the master and slave arms maps one-to-one with the humanoid's dual arms, allowing direct control of arm and hand movements. Joysticks control the chassis velocity and waist motion, an IMU captures head orientation, and an onboard first-person camera provides visual observations.

120 demonstration samples are collected per task, totaling approximately 840 across seven tasks. Data collection deliberately diversifies initial poses: the laundry loading task samples from three starting points on the bed; the clothes retrieval task arranges garments across four positions along the drying rack; the desk tidying task collects data from two rows in front of the desk, with three lateral offset points per row; and the ball throwing task places the ball and target box at nine marked locations. This extensive pose coverage prevents the policy from relying on a fixed standing position as a default.

The state vectors for the two robots are assembled independently.

• Tiangong 3.0 is 99-dimensional, comprising left and right arms, hands, tactile feedback, head, leg joints, standing state, and inertial measurement unit features;
• Tiangong 2.0 is 36-dimensional, covering arms, hands, waist pose, and leg motion velocity.

Both state sets are percentile-normalized, split according to the discrete state input rules of π0.5, and then concatenated into the prompt, with token limits of 385 and 192, respectively. Despite the vast difference in observation dimensions, both share the same policy interface; hardware-level differences are absorbed by the observation encapsulation layer.

The same 4-dimensional waist command carries different physical meanings on the two robots.

• The waist of Tiangong 3.0 consists of three joints combined with a squat height parameter, where the command directly maps to joint targets;
• Tiangong 2.0 outputs the overall waist pose, relying on leg-waist coordination to achieve the target posture.

The same applies to head control, which features two and one degrees of freedom for the two robots, respectively. All hardware differences are fully encapsulated within the deployment layer, while the policy layer reads only a unified 26-dimensional interface. Switching robot platforms requires only adjusting the encapsulation, leaving the model itself unchanged.

Inference is executed sequentially in stages. Each stage starts from independently sampled noise and performs 10 steps of ORA denoising; upon completing all three stages, the output from the third stage is used. The robot first executes the first 40 steps of the action sequence before initiating the next round of inference. On a single RTX 5090 GPU, the total inference time across the three stages is approximately 0.12 seconds. For reference, a single 10-step inference stage in π0.5 has a latency of 0.075 seconds, which increases to 0.115 seconds when scaled to 30 steps. While this latency is comparable to that of HAF-VLA, the performance score drops from 70.0 to 30.0.

The team notes in the appendix that this baseline comparison aligns only the number of denoising updates, not the computational cost. The hierarchical approach introduces additional cache computations, and the attention context length varies across stages. Merely increasing the number of denoising steps fails to recover performance; temporal action allocation is far more critical than the total number of denoising steps.

The baseline models and HAF-VLA use the exact same teleoperation dataset, humanoid action interface, and deployment wrapper. All predicted action chunks map to a unified interface for the base, waist, head, arms, and hands, ensuring that the controlled variables isolate the policy itself.

All four baselines were tuned to their optimal settings based on public parameters: ACT uses a 200-step action horizon and a 60-step temporal aggregation window, trained for 37,000 steps on eight A100 GPUs with a per-GPU batch size of 64; GR00T N1.7 uses an action chunk length of 50, trained for 20,000 steps with a global batch size of 160; Cosmos Policy uses an action chunk length of 40, trained for 15,000 steps with a global batch size of 160; and π0.5 undergoes full-model fine-tuning, trained for 50,000 steps on eight A100 GPUs with a per-GPU batch size of 8.

Hierarchical strategy yields performance gains

Real-robot experiments were conducted on two full-sized humanoid robots, Tiangong 3.0 and Tiangong 2.0, relying solely on an onboard egocentric camera without external motion capture equipment or additional cameras.

Six core benchmark tasks cover household and industrial scenarios: loading the washing machine, fetching clothes, clearing a table, watering plants, throwing a ball, and carrying boxes. An additional Box Carry task is included specifically for ablation studies. All tasks require the robot to navigate, adjust its posture, and then execute the manipulation. Each rollout lasts from a few seconds to over ten seconds, primarily testing long-horizon whole-body mobile manipulation capabilities.

These tasks place stringent demands on whole-body coordination.

• Loading the washing machine requires picking up clothes of varying shapes from a bed, walking to the washing machine, aligning the clothes with the hatch to load them, and then closing the lid;
• Fetching clothes requires raising an arm to the height of a clothesline to retrieve the garments, moving to a wardrobe to hang them up, and closing the door; standing too far away prevents the arm from reaching, while standing too close causes the body to collide with the rod;
• In the plant watering task, the robot must first support a flowerpot to water a tall plant, then lower its body by squatting or bending over to water a shorter potted plant, a task that cannot be accomplished by arm extension alone.

Throwing a ball and carrying a box test the precision of the final motion.

• Throwing a ball requires navigating to the correct position and releasing the ball accurately toward the bin opening;
• Carrying a box requires the robot to maintain the box's orientation while walking and turning, then aligning and placing it down at the edge of the table.

Each task includes a specific text instruction, which is fed into the policy model alongside first-person visual observations and the robot's state. For instance, the instruction for the laundry-loading task is "Pick up the clothes from the bed, walk to the washing machine, and put the clothes into the washing machine". The instruction specifies only the final goal, leaving the navigation path and turning maneuvers to the policy's autonomous decision-making.

The evaluation employs a milestone-based scoring mechanism. Each task is broken down into multiple stages with a maximum raw score of 15 points, which is then normalized to a 100-point scale based on completion progress.

For the laundry-loading task, placing one garment earns 5 points, placing two earns 10 points, and closing the lid adds another 5 points; for the ball-throwing task, a successful grasp earns 5 points, the ball touching the bin wall earns 5 points, and the ball landing inside the bin earns 10 points. Even if the robot fails midway, it still accumulates milestone points, ensuring comparability across different long-horizon tasks. Each method and configuration is evaluated over 10 real-robot rollouts, with initial states aligned as closely as possible to the data collection distribution; if an unsafe action is triggered, the trial is immediately terminated and flagged as Unsafe.

The average normalized score across the six tasks reached 68.3, which is 15.0 percentage points higher than the π0.5 baseline of 53.3; GR00T N1.7 scored 37.8, ACT scored 25.6, and Cosmos Policy scored 25.0.

HAF-VLA achieved the best or tied for the best score across all six tasks. The most pronounced performance gaps occurred in box carrying (93.3 vs. 60.0), ball throwing (56.7 vs. 33.3), and laundry loading (66.7 vs. 53.3). The failure modes of the baselines differ significantly: π0.5 and GR00T N1.7 are more prone to base displacement, attempting to compensate forcefully with their arms, which causes the motion to continuously degrade; Cosmos Policy exhibits significantly higher inference latency in this deployment environment. This performance gap widens further in tasks requiring "long-distance navigation + precise alignment", which is precisely the core advantage scenario for the framework's independent planning of base and torso postures.

Examining the results task by task, the performance improvements are not uniform.

Clearing the table scored 80.0 versus 70.0, and watering plants scored 60.0 versus 50.0, reflecting a 10-point improvement in each. These tasks involve stable walking but require limited spatial extent;

Fetching clothes scored 53.3 versus 53.3, resulting in a tie. This task primarily focuses on arm lifting and stance adjustment, yielding limited gains from chassis planning.

The three tasks that show the largest performance gaps all require long-distance navigation: moving boxes, throwing balls, and loading laundry. Each requires the robot to first navigate to the target location before executing fine-grained alignment. Cosmos Policy scored 73.3 in moving boxes, the highest among the four baselines, but scored only between 0 and 30 on the other five tasks, indicating that the world model approach lacks stability across this action interface.

The overall task completion success rate provides a more practical evaluation metric for real-world deployment.

• Moving boxes: 80% compared to 30% for π0.5;
• Clearing the table: 60% compared to 40%;
• Watering plants: 50% compared to 40%;
• Fetching clothes: 40% compared to 30%;
• Throwing balls: 40% compared to 20%;
• Loading laundry: 30% compared to 20%.

The paper includes 95% Wilson confidence intervals; however, the intervals are wide due to the limitation of only 10 rollouts. The authors note that these results serve only as a reference for uncertainty and are insufficient to support statistical significance testing, with the data used primarily to identify performance trends.

Summarizing the overall task completion success rates of the baselines reveals clear performance differences.

The success rates for π0.5 across the six tasks range from 20% to 40%, while GR00T N1.7 ranges from 10% to 20%. For Cosmos Policy, aside from a 30% success rate in moving boxes, the remaining tasks fall mostly between 0% and 20%. ACT achieves 30% only in moving boxes, 10% each in clearing the table and throwing balls, and 0% across the other three tasks. In contrast, HAF-VLA achieves success rates of 80%, 60%, 50%, 40%, 40%, and 30% across the six tasks, respectively, all exceeding the upper bounds of the baseline ranges.

The ablation studies were uniformly conducted on the box-to-table task, using the same set of 120 demonstrations and the identical deployment wrapper.

The sequencing of action stages has the most significant impact on performance: the native legs-waist-arms sequence achieves a score of 86.7; moving the waist stage forward drops the score to 70.0; and moving the arms stage forward reduces it to 60.0.

The team ruled out interference from supervision frequency by applying inverse-frequency weighting based on the number of activated segments per dimension, assigning weights of 1/3 for the legs and head, 1/2 for the waist, and 1 for the arms, which yielded a score of only 76.7. This proves that the performance gains stem from the stage timing itself. Furthermore, the All-Joint setting, which applies synchronous denoising across all joints, scored 73.3, demonstrating that the effective gains arise from segmented activation rather than repeated full-body denoising.

Comparisons of the denoising budget further illustrate the core conclusion. For π0.5, single-segment 10-step denoising achieves a score of 70.0; increasing this to 30 steps actually causes the score to plummet to 30.0, while latency increases from 0.075 seconds to 0.115 seconds.

In contrast, HAF-VLA allocates 10 steps to each of its three stages, maintaining the same total number of denoising steps, but achieves a score of 86.7 with a latency of 0.123 seconds. Cross-stage caching is equally critical: removing the cache during both training and inference drops the score to 50.0, while removing or clearing the cache only during inference directly triggers unsafe actions. The conditional structures for training and inference must remain consistent. Clearing the cache effectively cuts off prior planning information, forcing the subsequent stages to independently infer the preceding actions, which is highly prone to trajectory deviation.

The refinement strategy also involves trade-offs: freezing the preceding stage actions without allowing modifications yields a score of 66.7, while directly concatenating the outputs of the three stages scores only 46.7; both approaches underperform the soft refinement scheme, which permits minor corrections in the subsequent stages. The paper also plots the forward velocity prediction curves for the three stages during the box-moving process, showing that the overall trajectories largely overlap, with differences concentrated in local fine-tuning.

The generalization tests introduce five types of environmental disturbances.

In the laundry-loading scenario, placing an unseen office chair on the path reduces the score from 66.7 to 40.0, a drop comparable to that of π0.5. For the clothes-retrieval task, moving the robot's starting point backward by 20 centimeters results in HAF-VLA retaining a score of 43.3, a decrease of 10.0, whereas π0.5 drops by 16.7. The box-to-table tests incorporate three sets of disturbances: dimming the lights, changing the box dimensions from 40×25 centimeters to 33×20 centimeters, and translating the target table by 30 centimeters. The resulting scores are 83.3 versus 60.0, 76.7 versus 50.0, and 73.3 versus 40.0, respectively; the corresponding score drops are 3.3 versus 10.0, 10.0 versus 20.0, and 13.3 versus 30.0. The model demonstrates a clear advantage under spatial and geometric disturbances, though both approaches exhibit shortcomings when faced with changes in visual appearance.

In the office chair obstacle test, both strategies showed a comparable drop in performance, indicating that unknown static obstacles remain a challenge for both.

Hierarchical generation optimizes the controllability of the base and body posture, but it does not enhance environmental perception or obstacle avoidance. This approach reuses the original weights of π0.5, modifying only the action generation sequence: it plans full-body motions step-by-step along the kinematic chain, first locking the base and posture. For humanoid robots operating in a high-dimensional action space, the sequence of action planning itself serves as transferable prior knowledge.