I-BFM: Unsupervised Reinforcement Learning Enables Robust Humanoid Interaction After Falls

In recent years, several impressive robot box-moving demos have emerged. But have you noticed a detail? In most of these demonstrations, everything follows a script. Boxes are placed exactly where they should be, the robot stands perfectly straight, and it succeeds on the first try without any interference. In fact, many shots do not even show a human present.

The real world is not like this. Hands may slip, boxes might be too heavy to lift, someone nearby might accidentally bump into you, or the robot itself might fall on its back.

Faced with such issues, the mainstream response has been to replan a trajectory and then get up. However, the latency introduced by planning often consumes the entire recovery window. Ziqi Han from Tongji University and his collaborators recently offered a different answer.

On October 5, Ziqi Han from Tongji University, Yitang Li from Tsinghua University, and Junhan Sun from Zhejiang University served as co-first authors. Hao Zhao, an assistant professor at Tsinghua University, was the corresponding author. Together with RoboParty Lab, Harbin Institute of Technology, ShanghaiTech University, and other institutions, they published “I-BFM: Reward-Conditioned Robust Humanoid Interaction via Unsupervised Reinforcement Learning.” According to the paper, this is the first behavior foundation model designed for humanoid interaction with objects.

I-BFM’s approach does not assign any task-specific rewards for “moving boxes.” Instead, it allows the robot to freely explore the relationships between its body, objects, and contacts in simulation. It uses forward-backward (FB) representation to compress state-transition patterns into a reusable latent space. When a new task arises, one writes a simple reward function to retrieve the appropriate latent vector command and executes it directly.

Regarding FB, F is a forward vector encoded from the “current state” and “action taken,” while B is a backward vector encoded from the “state that could potentially be reached in the future.” The dot product of these two vectors reveals which states can be reached if the current trajectory continues. The phrase “if the current trajectory continues” is determined by the latent vector z. Changing z causes the same state-action pair to lead to entirely different futures. Since z can be derived from the reward, this mechanism enables the retrieval of distinct behaviors described by a single reward signal.

In terms of performance, under undisturbed conditions, I-BFM performs on par with the planning-based OmniContact. However, when the robot is subjected to an instantaneous impact and knocked down, I-BFM maintains a success rate of 89.3%, whereas its competitor drops to just 1.3%.

With the release of this paper, 42HOW Robotics interviewed Ziqi Han, one of its co-first authors. He is a graduate student at Tongji University and an embodied AI algorithm intern at RoboParty, focusing on motion control. We primarily discussed three topics with him:

• What is the core principle that allows a robot to continue moving boxes after falling, and where does this robustness come from? Why do methods relying on reference trajectories or planning fail to achieve this;
• I-BFM currently only handles one object type: boxes. Its core approach involves searching for rewards in latent space. To handle objects in different states, what is the most critical addition needed;
• How did the idea for this work originate, and which recent works inspired it.

Based on Ziqi Han's interview and paper content, we have obtained brief answers to the three topics mentioned above:

Regarding the first point, the core principle is to let the robot decide the next step based on its current body state, box position, and contact status, rather than executing along a pre-planned trajectory. FB learns the transition relationships between states: if the box's position changes, it re-approaches; if the robot is pushed over, it first recovers its posture before continuing. The distinction with planning-based methods lies in the fact that after falling, they require replanning, which introduces latency. "Adjusting actions directly based on the current state and goal" and "replanning before execution" are two different recovery mechanisms.

Regarding the second point, expanding to more objects, his answer is that both data and state representation need to be enhanced. On one hand, this involves broadening the variety of objects and contact experiences, and designing specific rewards for new tasks; the reward function design for carrying boxes differs significantly from that for kicking boxes. On the other hand, it requires moving beyond motion-capture environments to reliably acquire object and contact states using onboard perception. He also specifically noted that while the training data consisted entirely of flat-to-flat carrying, the results could handle cross-height carrying, demonstrating the algorithm's inherent generalization capabilities.

As for inspired work, he recommended two papers. One is BFM-Zero, which he calls the pioneer of the "unsupervised reinforcement learning plus FB representation" approach; its first author, Yitang Li, is also a co-first author of I-BFM. Ziqi Han chose FB because it pre-trains capabilities into the model and extracts them at inference time, making speed and hyperparameter tuning much more efficient. The other is INTACT by Junhan Sun, which we recently discussed in Yann LeCun likes! Junhan Sun from Zhejiang University, Hao Zhao from Tsinghua University and others propose INTACT: Eliminating the "search tax" of JEPA world models; it enables the model to understand immediate physical changes while moving toward longer-term goals. This inspired LOGO's dual-objective design of predicting the next frame and the eighth frame.

In fact, when I saw this work, the first scenario that came to mind was moving items and loading/unloading in a factory. But what truly matters in a factory is not getting it right on the first try; it's being able to finish the job even when things go wrong. The path of behavior foundation models has embedded 'getting back up and continuing after a fall' into the robot's capability list.

The Gap Revealed After a Fall

Let’s first examine performance in simulations without disturbances.

In MuJoCo simulations, researchers tasked robots with moving boxes. The task involved walking to the box, picking it up, transporting it to a target location, and placing it down. Each evaluation consisted of 3 runs × 100 random episodes. Three baselines competed against I-BFM: PhysHSI (without reference trajectories), LessMimic, and OmniContact (which uses planning combined with reference trajectories).

According to the table, I-BFM achieved a success rate of 94.3%, outperforming PhysHSI’s 37.7% and LessMimic’s 26.0%. It also matched OmniContact’s 92.7% success rate, indicating that under undisturbed conditions, I-BFM performs on par with solutions that rely on planning plus reference trajectories.

The turning point came with experiments involving disturbances. Two types of disturbances were introduced: first, a downward force of 200 N was applied to the box during transport, causing it to slip and fall; second, an instantaneous impact of 2,000 N was applied directly to the robot’s pelvis, knocking it to the ground.

Looking at the results, even when the box fell, I-BFM maintained a success rate of 91.0%, whereas OmniContact dropped to 60.0%. When the robot itself fell, I-BFM still achieved an 89.3% success rate, while OmniContact plummeted to just 1.3%. In undisturbed scenarios, the close scores of 94.3% versus 92.7% suggested competition on the same level. However, in the event of a robot fall, the disparity between 89.3% and 1.3% was stark.

In fact, planning-based methods must regenerate trajectories after a fall. The planning process itself introduces latency, and the post-disturbance state may already deviate from the initial conditions assumed by the planner. By the time the trajectory is computed, the optimal window for recovery has passed. I-BFM’s policy, by contrast, outputs actions directly based on the current state and goal at every frame, eliminating the need to wait for a planning step.

Regarding these two recovery approaches, Ziqi Han explained in an interview: 'OmniContact may require re-planning. Our key difference is that our policy can adjust actions directly based on the current state and goal. Systems relying on fixed references need to re-plan. These are two distinct recovery mechanisms.'

This trend holds equally clear in both box-pushing and box-kicking tasks. When subjected to a fall disturbance during box pushing, I-BFM achieved a success rate of 74.3%, compared to 2.7% for OmniContact. LessMimic managed only a 1.7% success rate even under undisturbed conditions. The disparity was even more extreme in the box-kicking task: OmniContact succeeded in just 1.3% of trials without disturbances, whereas I-BFM reached 70%, far surpassing the former.

Crucially, I-BFM’s real-robot performance aligns with simulation results. Researchers deployed the policy on a Unitree G1 robot, running the same strategy at 50 Hz in a motion-capture arena.

The robot could retry automatically if it failed to lift a box; if pushed or if the box’s position changed, it would first stabilize its body, approach the box again, and then lift it. If the box was kicked away mid-execution, the robot immediately updated its actions to approach the new location, all without pausing to 'think' as planning-based systems do.

I-BFM not only completed tasks like 'box lifting' without interference but also significantly outperformed other baselines in task success rates across various disturbances, thanks to its robust recovery capabilities.

Recovery Capabilities

This ability to "fall and immediately get back up" is hidden in two design elements: the FB framework and LOGO.

Let's start with FB, the forward-backward representation mentioned at the beginning. During pre-training, no specific tasks are set; instead, unsupervised reinforcement learning allows the robot to freely explore an interactive environment. The FB framework compresses the state-transition patterns discovered during this exploration into a shared latent space.

Meanwhile, walking data and human-box interaction data are mixed together for training, with pushing, kicking, and carrying even sharing the same task label. The resulting latent space functions as a vast capability library: when a new task arises, one only needs to write a simple reward function—such as "raise the box height"—retrieve from this library, and directly compute a latent vector command to feed into policy execution, all without retraining for the new task.

The choice of the FB technical route is largely driven by engineering considerations. Tuning a box-carrying strategy using traditional per-task reinforcement learning requires restarting the entire training cycle if reward parameters are unsuitable. In contrast, the FB approach first trains capabilities into the model and extracts them when needed.

Ziqi Han calculated this engineering cost in an interview: "I chose FB for box carrying because tuning it with AMP was too complex. I designed a simple function, ran it for a day or two, and found the parameters unsuitable. But with FB, the model already possesses these capabilities; we just need to extract them. Therefore, both its speed and parameter-tuning efficiency are much stronger than pure AMP."

This may also be the advantage of FB's approach, which can reduce training strategy time.

Notably, I-BFM's state explicitly incorporates contact information, including whether the hands are touching the box, their current positions, and the distance to the box. With this contact state, the policy can learn that "the same posture with different contact" corresponds to "different actions," laying the foundation for subsequent closed-loop interaction and recovery after contact changes.

The box’s pose, velocity, and target displacement are all included in the state. This means the reward can be defined directly on the object itself, optimizing for whether the box has reached its destination rather than forcing the robot to follow a reference trajectory. Traditional tracking methods struggle to bypass this intermediate step of mimicking the reference motion.

In fact, the pre-training objective relies on three parallel signals. FB organizes the transition patterns of interaction states into the latent space; a discriminator distinguishes between human action data and actions generated by the policy, continuously imposing motion priors so that the policy maintains human-like movements during free exploration; and an auxiliary critic provides stable environmental signals to ensure smoother control, which is key to the successful transfer of simulation performance to real robots.

This mechanism inherently possesses robustness. Because the policy observes the current state—its own body configuration, the box’s position, and whether contact is made—at every frame before deciding the next action, it executes a state-driven closed-loop response rather than playing back a predetermined trajectory.

Consequently, recovery behaviors emerge naturally within the unified policy. The source is the extensive exploration of the state space during unsupervised reinforcement learning, where the robot encounters various unstable, ungraspable, or severely misaligned states in simulation beforehand.

Ziqi Han also described a test conducted by the team: the robot was given a single 'lift' latent code throughout the entire process, with no command switches midway, to see how far it could go. 'We provided only one lift latent to the robot without switching to approach, turn, or stand-up latents. At a distance of 2.4 meters, it could still approach the box and lift it. This far exceeds the distribution range of our dataset.'

This phenomenon indicates that a single latent instruction can drive behavior that adapts to the state under test conditions; actions such as approaching and bending are decided ad hoc by the policy.

Regarding LOGO (Local–Goal Objective Geometry Operator), it addresses a finer-grained problem: how to simultaneously manage immediate physical contact and distant goals.

The original approach used the average features of the next eight frames as the instruction. However, averaging inevitably blurs critical information, because the same averaged value could correspond to entirely different initial actions.

LOGO replaced the '8-frame average' with a dual-target objective: the next frame and the eighth frame. Specifically, the next frame focuses on immediate physical contact, while the eighth frame monitors the broader task direction. Through extensive experimentation, a 4:3 weight ratio was determined for these two targets and incorporated into training.

Ablation studies demonstrate LOGO's impact. Removing LOGO caused the box-moving success rate to drop sharply from 94.3% to 51.0%. In fact, LOGO did not significantly improve action tracking precision; the joint position and velocity tracking errors in the subsequent columns remained nearly unchanged.

What it truly improved was the stability of contact transitions—specifically, when the hand should grasp the box, when to release it, and when to reach again.

Regarding the inspiration behind this design, Ziqi Han mentioned Junhan Sun's INTACT in an interview: 'INTACT inspired me that the model must both understand immediate physical changes and know how to move toward a more distant goal. Therefore, we changed the original 8-frame average processing method to use the next frame and the future eighth frame as targets, allowing the policy to handle current contacts while continuously considering subsequent task objectives.'

More Than Just One 'Move' Action

The value of I-BFM lies not only in its robustness against interference but also in its ability to perform other tasks using the same policy by simply changing the reward function.

Let’s start with pushing and kicking. The approach breaks long-horizon interactions into several stages, assigning a simple stage-specific reward to each. The reward for pushing a box centers on contacting it from behind with the hands, sliding it along the ground, and bringing it to a stable stop near the target. For kicking, the reward selects impact actions that feature aligned stance, no hand contact with the box, and the ability to remain balanced after the kick.

With these two reward functions retrieving their respective latents, the robot transforms from a rigid porter into a flexible worker capable of both pushing and kicking heavy objects. Simulation results show a 75% success rate for pushing and 70% for kicking. A key premise underpins these figures: the training involved no specialized instruction for either pushing or kicking. In the pre-training data, moving, pushing, and kicking all shared the same task label, meaning the policy learned a single, shared interaction representation.

Further generalization lies in height variations. While the training data consisted entirely of flat-to-flat transport, evaluations placed start and end points on platforms at different elevations. The robot successfully completed cross-height transport, moving boxes from various initial poses and orientations to a single target pose. This ability to handle unseen scenarios stems from the combination of Foundation Models (FB) and unsupervised reinforcement learning.

This makes chaining long tasks straightforward. After achieving each sub-goal, the reward description updates, and the latent switches accordingly. As shown in the figure, the robot first pushes the box halfway, stops, bends down to pick it up, carries it to the destination, and sets it down steadily. Throughout this process, there is no policy switching or re-planning.

However, this path has its ceiling. The size of the replay buffer dictates the scope of capabilities. Ziqi Han clarified this boundary in an interview: 'Its upper limit depends on how much of the real-world state space exploration covers, and how close the simulation is to reality. For example, if my dataset lacks states involving throwing a box, my algorithm cannot perform such a task.'

The True Gap Revealed After a Fall: I-BFM Trains Physical Interaction into Foundation Models

The demos in simulation and on real hardware are impressive, but the industry is more concerned with how far this technology is from actual factory deployment. In an interview, Ziqi Han outlined several hurdles that must be overcome, presenting a pragmatic view of the specific challenges ahead.

The first hurdle is object diversity. Factories do not handle only one type of box; weight, dimensions, and placement vary constantly. Currently, I-BFM interacts only with boxes. Expanding to other objects requires changing the state representation itself. While a box’s state is easy to describe, how does one represent a door handle? Or an object thrown into the air? These remain open questions.

The second hurdle is perception. Current real-robot experiments rely on external motion capture, which is neither cheap nor practical to install throughout a factory. Ultimately, this must be replaced by onboard perception, enabling the robot to reliably perceive objects and contact states using its own sensors.

Notably, I-BFM’s architecture already accommodates this transition. During training, only the critic has access to privileged omniscient information. The deployed actor relies solely on proprioceptive data and historical observations from external measurements. This means the policy itself never depends on a "God’s-eye view"; the change required is merely in the source of object and contact state data.

The third hurdle is higher-level decision-making. Current task rules are still manually scripted—for example, moving an item from point A to point B. In a factory environment, obstacles, detours, and unexpected situations arise. This likely requires adding a planning layer above the current system, allowing the robot to decide what to prioritize and how to handle failures.

The fourth hurdle is the most fundamental: economic viability. Factories must calculate cycle times, continuous operation duration, battery replacement intervals, and safety compliance. These are the operational metrics algorithms must address when leaving the laboratory.

Nevertheless, it demonstrates that interaction capabilities can be pre-trained at scale, much like knowledge in large models, stored in a latent space, and retrieved during use via reward-based prompting.

Moving, pushing, and kicking are three prompts. Following this logic, writing appropriate reward functions and retrieving latent representations should theoretically enable physical interactions such as opening doors or handing over tools.

Ziqi Han’s expectations for this framework extend beyond boxes: 'I believe BFM has immense potential. Applying it to a larger-scale foundation model is feasible, and there is still much to explore in that direction.'

The paper’s conclusion serves as a fitting footnote for the entire endeavor: enabling general-purpose humanoid robots to interact with the physical world through reusable, robust behavioral representations marks a step forward for I-BFM.