Apple Integrates Humanoid Perception Into Simulation for Zero-Shot Home Deployment With HALO

Apple's robotics research team introduces HALO, an autonomous humanoid control system trained entirely on synthetic simulation data and deployed on a Unitree G1. The framework enables zero-shot execution of complex household tasks in unscanned, unseen real-world environments without relying on real-world training data.

Just because a robot can reproduce pre-choreographed movements does not mean it can autonomously complete tasks in an unfamiliar home. To perform household tasks, a robot must understand instructions, locate target objects, plan movements based on its environment, approach the target, and execute the interaction.

How, then, can a robot be trained using only simulation data to achieve this in real homes it has never seen or scanned?

On November 4, Apple's robotics research team published the paper Perceptive Humanoid Motion from Simulation, introducing HALO. This autonomous control system for humanoid robots is trained in simulation and deployed directly on the physical robot. It sequences tasks, generates whole-body motions, and controls execution based on language instructions and egocentric images.

HALO automatically synthesizes training data using procedural scenes and a motion diffusion model, then filters the resulting motion trajectories through geometric verification. The entire control stack requires no real-world home data for training and no room scanning prior to deployment. During execution, a tracking module integrates data from a head-mounted depth camera to correct trajectories within a 50 Hz control loop based on the real-world environment.

The team deployed HALO on a Unitree G1, conducting 360 real-robot tests across six unfamiliar environments. The system achieved an average success rate of 91% on single tasks and 70% on chained multi-task executions. These results demonstrate that a robot can transfer motions learned in simulation to unfamiliar environments, though sequentially completing multiple tasks remains more challenging.

To understand how HALO transitioned from simulation to the physical robot, we must first examine how the training data is generated, then see how the system translates a single instruction into whole-body motions, and finally review how the team trained and validated this control system.

01 How simulation rooms become training data

The complete data pipeline consists of four main stages: first, tasks are decomposed into motion segments to synthesize motion trajectories; next, these trajectories are filtered and refined; then, the motions are placed into procedurally generated rooms; and finally, egocentric images are rendered and output.

The first step is motion synthesis. The research team divides each task into multiple motion segments, assigning a text prompt, optional waypoints, and an end keyframe to each. For instance, the task of sitting in an armchair can be split into walking forward, turning, and sitting down, using keyframes to fix the relative position between the final sitting posture and the target furniture. This task decomposition scheme needs to be written only once and can be reused across all procedurally generated rooms.

Motions are generated by the motion diffusion model Kimodo, which outputs dense whole-body motion sequences conditioned on text, waypoints, and keyframes. Because Kimodo only performs kinematic solving, the generated trajectories are prone to foot sliding, mesh clipping, and floating bodies. The team samples multiple candidate trajectories for each segment and performs geometric verification via forward kinematics to detect anomalies such as foot sliding, ground penetration, floating, and joint limit violations. They retain the highest-scoring trajectory and concatenate the segments into a coherent motion sequence.

The second step is scene generation. The team generated 1,200 rooms using the Infinigen indoor generator, 1,178 of which completed the full pipeline. They selected 1,000 for training and reserved the remaining 178 for simulation evaluation, including a 100-room evaluation subset. The scenes cover living rooms, offices, and bedrooms, with room dimensions ranging from 3.5 to 9.5 meters. The number of furniture items per room ranges from 20 to 80, featuring random combinations of floor, carpet, door, window, and wall materials. To accommodate navigation and door-opening tasks, an additional 966 two-room and multi-room scenes were used.

The third step is motion placement. The team extracts a 2D occupancy grid of the room layout and searches for the translation and rotation parameters of the rigid body within the grid space, testing up to 800 positions and 24 orientations. They select the first configuration that satisfies the constraints: the pelvis maintains a 0.35-meter safety distance from all obstacles while allowing the target furniture to be placed correctly. Each data sample is uniquely determined by a random seed and placement parameters, which decouples motion generation from rendering. Consequently, changing the camera, resolution, or renderer does not require regenerating the motion trajectories.

The fourth step is image rendering. The team replays the kinematic trajectories in Isaac Sim 5.1 using a fisheye lens identical to the robot's head-mounted camera, outputting 1280 × 800 resolution frames with a 128° × 81° field of view and a 25° downward pitch. They augment the visual diversity of the training samples by randomly adjusting exposure, contrast, color temperature, motion blur, and sensor noise. For the Go-To, Crawl, and Crouch tasks, the brightness and color of the dome light are also adjusted individually during the rendering phase.

The most ingenious aspect of the pipeline is its lighting scheme. Because per-frame path tracing is computationally prohibitive, the team pre-bakes the direct and indirect lighting of each room directly into the objects' emissive maps. The median time to bake a single room is 38.5 minutes. For unseen viewpoints, the average difference between the baked renders and path tracing is 3.8 grayscale levels, or approximately 1.5%. Once the lighting is baked, any rasterization renderer can generate images in real time, and the dataset supports cross-simulator re-rendering. MuJoCo and Isaac Sim use the same baked assets, keeping the rendering error compared to Blender within 1.5 and 2.0 grayscale levels, respectively.

The final dataset covers six skill categories, comprising 162,698 motion samples with a total duration of 296.3 hours and an average length of 6.6 seconds per sample. The Sit task consists of three segments—walking, turning, and sitting—averaging 7.1 seconds. The Crawl and Crouch tasks average approximately 9.4 seconds and 9.1 seconds, respectively. The Go-To task is the shortest, averaging just 3.4 seconds.

By generating motions with a diffusion model, controlling quality through geometric verification, and building rooms with a procedural generator, the entire pipeline eliminates the need for manual data collection. The motion scale and total duration can scale with compute, while the number of rooms independently dictates visual and geometric diversity. These two factors are decoupled, allowing researchers to isolate and verify their respective impacts through ablation studies.

02 From a single instruction to full-body motion

The entire technology stack is divided into three layers: the top layer decomposes natural language instructions into subtasks; the middle layer combines these subtasks with real-time visual input to output a 5-second full-body motion trajectory; and the bottom layer translates this trajectory into joint control commands for the robot.

The first layer is task scheduling. Relying on a vision-language model (VLM), the research team decomposes free-text instructions into up to eight subtasks, with each subtask bound to a specific skill and target object for interaction. User instructions often omit the target—for example, simply saying "sit down"—requiring the system to identify the corresponding chair from the real-time visual feed. Instructions may also imply prerequisite actions; if a table blocks the path ahead, the robot must first crouch and navigate around it before approaching the target. The scheduler continuously reads the head-mounted camera feed, advancing to the next task upon completing the current step, and retrying if it fails. If passage is blocked, it can automatically insert new subtasks.

The scheduler is deployed on a Mac mini paired with the robot, running the Qwen3.6-35B-A3B model with 4-bit quantization and MLX-based inference at a frequency of approximately 0.5 Hz. It shares the same inference engine with the motion generator, supporting offline operation; however, it can switch to a cloud-based model if stronger planning capabilities are required. The prompts explicitly specify the trigger conditions and completion criteria for each subtask. The output follows a fixed JSON format, and step names are strictly limited to skills already implemented in the system.

The second layer is trajectory generation. The generator receives a single head-mounted RGB image and the current subtask, outputting a 5-second full-body motion plan comprising 150 frames at 30 Hz. Each frame includes 29 joint angles, along with the 3D translation of the pelvis and a quaternion, totaling 36 control variables. The encoder uses a frozen Qwen3.5-2B vision-language model, while the decoder is a diffusion Transformer trained from scratch: 8 blocks, a width of 512, and 8 attention heads. It is trained using flow matching, and inference employs a 15-step Euler integration, with 49.3 million trainable parameters.

The core innovation of this module is its asynchronous generation mechanism. A single generation takes over 100 milliseconds; if the system waited for the current trajectory to finish executing before computing the next one, the robot would stall during the inference gap, unable to perform dynamic movements smoothly. The team chose to have the generator pre-compute the next trajectory while the tracker executes the current plan, though this naturally creates a seam at the junction of the two segments. Fixing this continuity during the inference stage would not only make it difficult to guarantee a smooth trajectory but also require calculating gradients at every denoising step, further increasing latency.

Instead, the team integrated continuity constraints directly into the training process. The first 15 frames of the new trajectory—representing the fixed 0.5-second motion the robot is about to execute—are fed directly into the generator as a deterministic prefix. The model then denoises only the remaining 135 frames, and the loss function is calculated exclusively for this portion. During training, the model learns to continue generating trajectories conditioned on any motion prefix, making smooth plan switching a native capability of the network without relying on extra patching during inference. During deployment, the first 15 frames of the new plan are copied directly from the trajectory currently being executed and synchronously overwritten after each denoising step, allowing the two trajectories to connect seamlessly.

The third layer is trajectory tracking. The tracker operates at 50 Hz, receiving the motion plan from the generator, real-time depth maps, proprioceptive sensor data, and IMU signals, to output 29 target joint positions. The depth information comes from a RealSense D435i camera mounted below the head-mounted camera. The team disabled the infrared dot projection and fed the raw stereo infrared images into a lightweight stereo matching network. Running on a Jetson platform with TensorRT FP16, it outputs a high-quality 48 × 84 depth map, avoiding the noise and holes inherent in the camera's native depth output.

Integrating depth perception into the tracker is the key differentiator of this approach from prior work. In traditional methods, perception-less trackers strictly replicate predefined footstep locations; with depth integrated, unplanned obstacles such as stairs, carpets, and furniture edges can be adjusted in real time within the 50 Hz control loop. This is also a primary reason the robot can walk stably in unfamiliar, unscanned rooms.

The three-tier architecture is connected via minimalist interfaces: the upper layer passes only subtask text downward, the middle layer outputs only motion trajectories, and the bottom layer feeds back execution status. These narrow interfaces facilitate the iterative replacement of individual modules, but they also mean the tracker does not understand task intent and is solely responsible for faithfully following the trajectory.

03 From expert policies to real-robot control

The tracker employs a three-stage training pipeline: first, training an expert policy with access to privileged information; then, using knowledge distillation to derive a student model that relies solely on onboard sensors; and finally, fine-tuning via multi-critic reinforcement learning. The generator, meanwhile, follows an independent imitation learning pipeline.

The training data is drawn from 1,000 procedurally generated rooms, totaling 300 hours. The entire synthetic task took 48 hours on 250 H100 GPUs, requiring no human intervention. The generator completed 50 epochs of training on 64 B200 GPUs, with a total of 31.0 million samples and a batch size of 5,120. The three training stages of the tracker utilized one, one, and four H100 GPUs, respectively, with the number of parallel simulation environments set to 8,192, 4,096, and 16,384.

The first stage is expert policy training. The team trained a separate PPO expert for each skill category. The expert can read privileged information in the simulation environment, including proprioceptive data, the next 10 frames of reference trajectories, and local height maps of the ground and furniture, outputting 50 Hz joint position targets for execution by a PD controller. Relying solely on pose imitation is insufficient for interactive tasks like pushing a box: if the robotic hand stops next to the box without displacing it, the pose error remains very small. Therefore, experts for object interaction receive two additional rewards: hand-object distance and object displacement.

The second stage distills the expert into a student model. The student receives only onboard signals: motion plans, head-mounted depth maps, proprioceptive data, and IMU readings, and cannot access privileged information such as object coordinates and terrain height. Each simulation sample is routed to the corresponding expert based on the current skill, and the student does not average or fuse actions from multiple experts. The training combines behavior cloning and DAgger, with 20% of the rollouts driven by the expert and the remaining 80% executed autonomously by the student, allowing the model to learn corrective behaviors from its own generated deviations.

The third stage uses reinforcement learning to directly fine-tune the student model, integrating depth rendering into the training loop. The reward magnitudes for different skills vary significantly; object interaction tasks include additional object rewards. If a single critic were shared, high-reward tasks would squeeze the learning space for the remaining skills. The team configured a separate critic for each skill category while sharing the actor weights; each critic is updated using only the samples for its corresponding skill, and the advantage function is independently normalized per skill.

The research team narrowed the sim-to-real domain gap using two approaches. First, they analytically derived the PD gains for each joint based on the motor rotor inertia to maintain a consistent natural frequency and damping ratio across all joints, scaling the action amplitude according to the joint torque limits. Second, they fitted the stiffness, damping, rotor inertia, torque limits, and transmission delay on the real robot using open-loop trajectories. After aligning the training distribution with the identified real-robot parameters, the domain randomization range can converge close to real-world operating conditions, preventing the policy from becoming overly conservative due to excessively large random intervals. Following this fitting process, the joint tracking error decreased from 26.9% to 10.1%.

Domain randomization was applied in two tiers. The first two training stages used narrower randomization intervals to preserve the precision of the expert policy. In the third stage, the constraints were relaxed to randomize the PD gains, rotor inertia, torque limits, and link masses, along with command delays of 15 to 40 milliseconds, while also perturbing ground friction, ground stiffness and damping, object mass, and friction coefficients. The depth stream additionally introduced camera mounting offsets, random occlusions, distance-increasing stereo noise, 3% pixel loss, and 1 to 3 frames of latency.

The real-robot platform used was the Unitree G1, weighing 37.56 kg in total. A protective cover was added to its back to carry a Mac mini M4 Pro, and an OAK-1W wide-angle camera was mounted on its head. The upper-level scheduler and generator ran on the Mac mini, while the lower-level tracker was deployed on the robot's built-in Jetson Orin NX, with the two layers communicating asynchronously via ZeroMQ. The tracker was implemented as a C++ real-time process running on a PREEMPT_RT kernel to ensure the control loop remained unblocked.

A single inference pass for the generator took approximately 152 milliseconds, or about 155 milliseconds when sharing compute with the scheduler. A single transition decision for the scheduler took about 0.72 seconds when running independently, or approximately 1.46 seconds when sharing compute. The motion plan has a timing window of 333 milliseconds; the scheduler reserves at least 60% of the GPU resources to prioritize trajectory generation, and scheduling calls can be temporarily paused between layers.

04 Real-robot testing in six unfamiliar homes

The entire validation process was divided into four stages: first, testing was conducted on six independent single tasks; next, three multi-task chains were verified; then, ablation studies were performed on each module; and finally, the robot's generalization ability was evaluated in completely new rooms with unfamiliar furniture and different human targets.

The experiments ran the full deployment pipeline in MuJoCo. The head-mounted camera feed input to the generator was rendered according to the OAK-1W fisheye camera model, the simulation ran in real time, and planning delays were included in the entire process. The six tasks were walking to a specified object, sitting on a chair, passing sideways through a half-open door, crawling under a table, crawling flat on the ground under a table, and pushing a box. Each single task was repeated 10 times across 100 simulation environments and 6 real-world environments, totaling 6,000 simulation trials and 360 real-robot trials.

Each task had independent evaluation criteria: any fall was recorded as a failure; a pelvic tilt angle exceeding 60°, or a pelvic height below 0.3 meters lasting for 1 second, was also classified as a fall. For Crouch and Crawl, the robot needed to pass under the table without touching the tabletop, and stand upright and stabilize after emerging. For Door, it had to pass smoothly through the doorway without bumping the door panels, and stand stably after entering the room. For Sit, it needed to reach within 0.9 meters of the chair, with a pelvic height between 0.35 and 0.68 meters, and a torso deviation from the vertical of no more than 40°. For Push, the box had to be moved at least 0.3 meters. For Go-To, the robot was required to stop within 1 meter of the target point.

The average single-task success rate reached 91% in both simulation and real-robot scenarios. Specifically, the real-robot success rate for Go-To was 92%, and Crouch achieved 100% on the physical robot. Four of the six tasks demonstrated real-robot performance on par with simulation. The identical average scores for simulation and the real robot indicate that this approach avoided the sim-to-real domain gap, where a policy works in simulation but fails on physical hardware. The physical testing environments included wood floors, tiles, concrete, thresholds, and carpets of varying thicknesses.

For the multi-task evaluation, three task chains were selected and executed 10 times each across 50 simulation environments and 1 real-world environment. The average success rates were 69% in simulation and 70% on the real robot. Notably, the real-robot success rate for Door→Go-To reached 100%, while Crouch→Go-To→Push achieved 50%. The success rate for chained tasks was lower than the product of the individual single-task success rates: the theoretical product for Door→Go-To was 89%, but the actual success rate was only 71%, demonstrating that the subtasks are not mutually independent.

The team reviewed failure replays and found that most failures stemmed from the robot's posture at the end of the preceding task. On the physical robot, Crouch→Go-To→Push failed 5 times in total; in 3 of these cases, the robot's head orientation deviated from the box after crawling, keeping the box out of its field of view and causing the robot to be misled by other objects. For Crouch→Sit, there were two instances where the robot mistakenly identified a sofa on the right as the target chair. A similar issue occurred in simulation for Door→Go-To: while the door-passing task itself had an 85% success rate, the robot's orientation upon exiting varied depending on the doorway's position. This could cause the target object to move entirely out of view, reducing the conditional success rate of the subsequent Go-To task to 83%.

The ablation study focused on the Sit task, testing each configuration 10 times across 100 held-out simulation environments. As the number of training rooms increased from 1 to 100, the success rate jumped from 43% to 81%. However, expanding the scale from 100 to 1,000 rooms yielded only an additional 3 percentage points of improvement. This demonstrates that data diversity provides significant early gains, but the marginal returns become highly limited once the number of training rooms reaches 1,000.

Trajectory continuity design had an even greater impact on performance. Without any continuity processing, every plan switch resulted in a discontinuous jump, causing the success rate to drop to 24%. Switching to a method that patches trajectory seams during inference raised the success rate to 66%, which was still 18 percentage points lower than the approach that employed prefix constraints during training.

The researchers also conducted control experiments by separately removing the decoder optimization and depth perception modules. Replacing flow matching with a regression decoder caused the success rate to plummet to 19%. Removing the depth input and relying solely on proprioceptive sensing reduced the success rate to 74%.

The most critical design aspect of this evaluation was the use of held-out, independent environments. All control experiments were conducted in completely unseen rooms, swapping the entire physical space rather than just changing the trajectory within the same room. Because every set of metrics was collected in a completely novel environment, the "zero-shot" claim holds practical significance. The six testing environments used for the physical robot were also not pre-scanned or mapped, and none of them appeared in the training set.

The real-robot generalization tests further varied the people, tables, and chairs involved: Go-To navigated toward different individuals in different rooms; Crawl involved crawling under two separate tables, one indoors and one outdoors; Crouch required crouching under three tables of varying heights; and the Sit targets included a black armchair, a white armchair, two standard sofas, and a sectional sofa. All six skill types were executed stably in unfamiliar environments, demonstrating that the rich environmental diversity used during training effectively translated into the robot's real-world adaptability.

The discussion section of the paper adds two practical deployment insights. First, simultaneously randomizing ground friction and contact compliance enables the robot to stand firmly on thick carpets while avoiding slips on smooth floors. Second, an action smoothness penalty is more effective than the simulation reward for predicting real-robot stability; if action rate regularization is insufficient, transmission delays will cause jitter in the ankles and wrists.

HALO transforms simulation into an automated data factory. It first trains the robot to sit, pass through doorways, crouch and crawl, and push boxes across 1,000 virtual rooms, and then deploys the complete policy to the physical robot for immediate operation in real homes that have never been scanned.

The true breakthrough lies in a fundamentally different approach to data generation: it no longer relies on humans to manually teleoperate and record samples piece by piece. As long as sufficient compute is available, it can continuously generate training data. This approach has already been successfully validated in six unfamiliar homes. The next step is to tackle more complex household tasks, enabling the robot to interpret a wider range of environmental cues.