Alibaba DAMO Academy Introduces RynnUnify, a 16B Unified Embodied Model Pairing 8B Reasoning with 8B Video Generation

On October 23, Alibaba DAMO Academy and Hupan Lab published a paper on RynnUnify, a 16-billion-parameter unified embodied model that uses an 8B reasoning module and an 8B video generator to predict future scene changes and guide sequential robotic actions.

Main: 8B for Reasoning, 8B for "Imagination": What Kind of Brain Did DAMO Academy Build for Robots?

Sub: Large Models Are Starting to Use Video for Drafting

Xiaohongshu Post

For robots, inferring the presence of occluded objects is exceptionally difficult. On October 23, a team from Alibaba DAMO Academy, in collaboration with Hupan Lab, published the paper RynnUnify: Towards Unified Embodied Brain. They introduced RynnUnify, an embodied AI model with just 16B parameters. The model predicts the final frame of each task segment and uses that frame to initiate the next round of reasoning. Rather than merely outputting actions, it also anticipates the resulting changes in the scene.

The paper has 23 authors in total. Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang, Yifei Yang, Zhikai Wang, Tianyi Zhang, and Kehan Li are the core contributors, with Chengju Liu and Jianfei Yang serving as co-corresponding authors. Contributing author Deli Zhao is the Chief Scientist at Alibaba DAMO Academy and the founder of the academy's embodied AI team.

The DAMO Academy team's approach integrates environmental understanding and future prediction into a single conditional framework to build RynnUnify, a 16B-parameter embodied model. This design creates a closed loop between language reasoning and video generation. The reasoning module breaks down tasks and defines the next actions, while the video module predicts scene changes and feeds the final frame of its prediction back into the next reasoning step. Beyond simply outputting actions, the model also anticipates how those actions will alter the scene.

Most current mainstream unified embodied models separate the understanding model and the world model into two independent modules, relying on text or coordinate information to interface between them. While flaws are hard to detect in short-horizon, single-step tasks, problems emerge as the task chain lengthens: the model continues to extrapolate from the erroneous frame of the previous step, compounding the error with each iteration.

In contrast, the RynnUnify model feeds the final frame of each prediction into the next round of reasoning. It answers not only "what to do now" but also "what the scene might look like after execution."

Paper: https://arxiv.org/abs/2610.32965

#Alibaba DAMO Academy #Unified Multimodal Model #World Model #Embodied AI

---

Cover

Summary

For robots, inferring occluded objects remains exceptionally difficult. On October 23, a team from Alibaba DAMO Academy, in collaboration with Hupan Lab, published the paper RynnUnify: Towards Unified Embodied Brain. They introduced RynnUnify, an embodied model with just 16B parameters. This model predicts the final frame of each task segment and then initiates the next round of reasoning based on that frame. Rather than merely outputting actions, it also anticipates the scene changes those actions will cause.

The paper lists 23 authors in total. Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang, Yifei Yang, Zhikai Wang, Tianyi Zhang, and Kehan Li are the core contributors, while Chengju Liu and Jianfei Yang serve as co-corresponding authors. Co-author Deli Zhao is the Chief Scientist at Alibaba DAMO Academy and the founder of the DAMO Academy Embodied AI team.

The goal of a unified embodied model is to enable a single model to simultaneously accomplish three tasks: understand the current scene, anticipate subsequent environmental changes, and translate abstract intents into concrete actions.

Most mainstream industry approaches currently deploy the understanding model and the world model separately. One handles scene understanding and outputs instructions, while the other generates visual frames, with the two exchanging information via text or coordinates. Both modules perform well when trained individually, but once integrated, there is no mechanism to hold them accountable for each other's outputs.

The shortcomings of this decoupled architecture become fully apparent in long-horizon tasks. The understanding model cannot verify whether the visual frames corresponding to its instructions are plausible, nor does the world model know which specific decision step its generated content is meant to support. If video prediction is implemented as one-shot generation, or if all tasks are indiscriminately required to undergo virtual simulation first, inaccurate dynamic information and distorted spatial geometries will propagate into the next reasoning step.

To address these two core issues, the DAMO Academy team introduced RynnUnify, a 16B unified embodied model. It comprises an 8B autoregressive reasoner and an 8B video generator, with the two modules connected via joint attention.

This architecture drives two capabilities simultaneously:

Specifically, the reasoner outputs constraint conditions, and the generator predicts videos based on these conditions. The final frame of the generated video is then fed back to the reasoner as the environmental observation for the next round, enabling a continuous, step-by-step autoregressive loop. The generator also incorporates two additional memory caches: a four-frame scene memory to preserve the scene's appearance and spatial layout, and the terminal frames of the previous video to record recent motion and interaction states. This allows the model to continuously track targets even if they temporarily leave the camera's field of view.

The team also distilled an 8B agent from the reasoner, using reinforcement learning to autonomously determine which tasks benefit from video prediction first and which can directly output actions. Following optimization, the model's average score on Rynn-GEU improved from 44.0 to 50.0. On their proprietary benchmark, Rynn-VP, the overall score increased from 45.05 to 54.80, with the planning intelligence sub-score rising from 30.50 to 45.66. When extended into a world-action model, the video features bypass image decoding and are fed directly into the action head to output control signals. This approach achieved an average success rate of 46.7% across three real-robot tasks on the Astribot S1.

The reasoner defines the steps, the generator predicts the outcomes

The foundational architecture of RynnUnify builds upon the Mixture-of-Transformers framework from NVIDIA Cosmos 3. The reasoner and generator branches maintain independent Transformer parameters, interacting solely through joint attention, which allows the generator to directly read the reasoner's representations. The reasoner's internal structure remains unmodified; instead, the team focused its adjustments on the generator's conditional interface by introducing cross-segment scene memory and enabling first- and last-frame constrained generation. The entire architecture is built on the Cosmos 3 backbone, with all subsequent embodied fine-tuning and capability extensions implemented on this foundation.

In each step, the reasoner outputs three pieces of information: the action to be executed next, the corresponding scene change for this step, and the approximate duration of the action.

The first two are expressed in text, while the last is a numerical value; together, they serve as the input conditions for the generator. Since the action duration is not a fixed value, the model is trained on 4n+1 frame sequences, retaining the native skeleton's 3D rotational positional encoding and frame rate modulation. During cross-segment continuous rollout, the duration calculation only counts the newly generated segment and excludes the preceding historical prefix. By packaging the action target, expected scene change, and execution duration into a single set of conditions, the reasoner and generator share a unified representation baseline.

The core challenge in long-horizon continuous rollout lies in building the memory mechanism. The reasoner maintains four sets of reference frames: one frame is fixed as the initial observation, two frames are selected from earlier historical observations, and one frame is taken from the latest observation of the previous step. When historical frames are insufficient, the model supplements them from already generated video segments; at the very first step of a task, the initial frame is repeated to fill all four memory slots.

The four reference frames are independently encoded by a causal video VAE. Their corresponding token sequences are concatenated after the text conditions and before the main video tokens, ensuring clear boundaries so they do not mix into the main video's timeline. In addition to the sparse reference frames, the model also incorporates the tail segment of the previous video to connect with the newly generated sequence, keeping the boundary frame only once. The reference frames stabilize the scene appearance and spatial layout, while the video prefix carries recent motion and interaction information; the two work together to support multi-segment continuous scene rollout.

The model supports three generation modes in total.

• Rolling forward from a single starting frame, which belongs to image-to-video generation;
• Fixing the first and last frames to fill in the middle frames, which is first-and-last frame constrained generation;
• Using the end of the previous video as a prefix to continue generation, which is cross-segment continuation.

Under all three modes, the latent variables corresponding to the scene memory and the conditional frames within the main video remain fixed throughout the denoising process, with the model updating only the regions to be predicted. During training, the loss is computed only at the positions to be predicted; the conditional inputs and independently encoded memory items do not participate in the loss calculation. Furthermore, the normalization operation is applied only to the visual items containing the prediction targets, preventing the conditional inputs from diluting the supervision signal.

After completing a single-step plan, the last frame of the generated segment is fed back to the reasoner as a new observation, initiating the next cycle.

Two-stage training to learn when to trigger inference

The entire training dataset is divided into two branches: understanding and planning.

The understanding branch uses 10% of the RynnBrain pre-training corpus as its foundation, comprising approximately 2 million entries covering general image-text and video understanding, temporal reasoning, and instruction-following tasks. On top of this, the team added supervised data for spatial localization, object affordances, motion trajectories, and manipulation planning, supplemented by 153,000 reasoning samples and 43,000 perception samples collected in-house.

The planning branch incorporates human and robot manipulation demonstrations, including 550,000 human demonstrations, 600,000 robot demonstrations, and 87,000 samples from the RoboCasa365 dataset. The materials used for video generation total 1.083 million clips, amounting to approximately 1,370 hours of footage. The demonstration samples are first trimmed into 2- to 10-second clips. Clips lacking subtask annotations are processed by Qwen3.6-27B for temporal segmentation and annotation completion. They are then converted into structured visual prompts via GPT-5.6-Terra, which explicitly record the scene layout, interacting objects, action sequences, and final state changes.

This single set of structured prompts is reused bidirectionally: it supervises the reasoner to output standardized conditions on one hand, and serves directly as the text input for the generator on the other, ensuring that both the planning instructions and the generated frames adhere to a unified formatting standard.

The training process is split into two stages. The first stage trains the reasoner in isolation, using a standard autoregressive loss computed only on the model's output response tokens, with a learning rate of 2e-6 and a batch size of 1024. The second stage freezes the reasoner weights and updates only the generator, using a learning rate of 2e-5 for 8,000 steps with a gradient clipping threshold of 0.3. The loss function is replaced with the flow-matching loss for rectified flow, computed exclusively over the primary video region being predicted. Because each stage updates only half of the parameters, this approach prevents the understanding and video generation modules from interfering with each other or pulling in conflicting directions.

A reality gap emerges during real-robot inference: the initial frame for each segment in the inference stage comes from the model's own prediction in the previous round, whereas the original training uses ground-truth frames as conditions throughout. This inconsistency causes deployment errors to accumulate and amplify continuously.

To bridge this gap, the team designed three schemes for corrupting the conditions.

The first scheme is conditional degradation: Gaussian noise is added at the pixel level, with the noise standard deviation randomly sampled between 0 and 0.15 and applied with a probability of 0.3. There is also a 0.5 probability of applying a downsample-upsample operation to simulate image blur.

The second scheme is offline endpoint replacement: a preceding video clip is generated and cached offline in advance using fixed weights, and its final frame replaces the ground-truth starting frame of the current clip.

The third scheme replaces both the historical sequence and the starting point: the ground-truth historical video is replaced with the cached generated clip, while the scene memory and video prefix are still reconstructed from this generated content. The supervision objective for the entire process remains unchanged—continuing the sequence toward the ground truth. The only modification is the initial observation received by the model, which adapts it to continue reasoning from its own biased outputs.

The reinforcement learning module is specifically designed to address a core question: when to first generate a video for reasoning versus when to output actions directly.

The team cloned an independent agent from the 8B reasoning model. For each test question, the system samples four direct-answer trajectories and four 'reason with video first, then answer' trajectories. Within the same branch, a sequence-level GSPO algorithm compares trajectory quality. Between the two distinct branches, the difference in average rewards serves as the soft-routing optimization objective; the higher the reward, the more the model favors the video-reasoning path. The reward value is calculated by multiplying the format compliance score by the answer quality score. For numerical and geometric tasks, this is mapped to a 0 to 1 scale via normalized error. For affordance and region-matching tasks, it is scored using the symmetric closest-point distance. For trajectory-matching tasks, it is calculated using the discrete Fréchet distance.

The reinforcement learning training set comprises 22,007 questions and 883 validation questions. Each round samples 64 questions, with eight trajectories sampled per question, resulting in a total of 344 parameter updates per round. During inference, the model autonomously selects a branch and completes its output along the chosen path.

How much performance gain does video imagination actually deliver

The research team evaluated the model's comprehension capabilities across ten public benchmarks: VSI-Bench, MMSI-Bench, MindCube, ERQA, RoboSpatial, EmbSpatial, RefSpatial-Bench, the affordance and trajectory sub-tasks of ShareRobot, and Where2Place. All results are initially calculated using each benchmark's native metrics, normalized to a 0 to 100 scale, and then averaged without weighting.

Long-horizon planning capabilities were evaluated using the team's proprietary Rynn-VP benchmark. This benchmark comprises 200 first-person tasks. Each task takes an initial image and a high-level instruction as input. The model can choose to generate a video or output an ordered sequence of steps, with a maximum rollout horizon of 15 steps. The scoring metric is divided into three components: planning intelligence (weight 0.5), visual realization (weight 0.4), and image quality (weight 0.1). Evaluation was conducted using a scoring pipeline powered by GPT-6-sol running within Codex. Additionally, ten reviewers with master's degrees or higher performed 300 rounds of cross-validation to ensure result stability. The Rynn-GEU benchmark enforces scene isolation, with the training and test sets completely separated. Test samples are filtered based on whether they can be answered correctly using future video information.

In the comparison of understanding capabilities, RynnUnify ranks first among this cohort of unified embodied models, achieving an average score of 57.9 across ten benchmarks. Cosmos 3 Nano scored 53.2, RxBrain scored 44.9, and BAGEL scored only 28.0. RynnUnify leads in six of the ten benchmarks. Specifically, it scored 67.1 on VSI-Bench, 12.2 points higher than Cosmos 3 Nano. On RoboSpatial, it scored 72.8, 10.7 points higher. On RefSpatial-Bench, it scored 60.3, and on ShareRobot's affordance metric, it improved from 28.0 to 41.8; these are the best scores among all evaluated models. The trajectory error on ShareRobot is 0.27, where a lower value indicates better performance. This comparison also includes general multimodal models. The strongest in this category, GPT-5.6 Terra, achieved an average score of 52.6, which is 5.3 points lower than RynnUnify. This demonstrates that the unified model is competitive with top general large models in embodied spatial understanding. Meanwhile, DAMO Academy's proprietary dedicated understanding model, RynnBrain-8B, achieved an average score of 61.2, outperforming this 16B unified model. This also indicates that the benefits of a unified architecture do not manifest in basic understanding capabilities.

Next, consider the impact of video imagination. Across the ten subsets of Rynn-GEU, RynnUnify achieves a score of 44.0 with direct answering, already the highest level among unified models. After integrating an agentic mechanism that learns to judge timing, the average score rises to 50.0, with improvements in seven of the ten subsets. The largest improvement is in region localization, jumping from 35.9 to 66.2; the clutter pile chained interaction task improves from 8.5 to 29.1. A few metrics show a slight decline: scores for position, distance, and size—three fine-grained geometric tasks—drop slightly because the generated videos inherently contain geometric distortions that interfere with such high-precision judgments. Three public reasoning benchmarks also benefit simultaneously, with the average score rising from 41.3 to 44.6, and MindCube improving by 8.4 points. In contrast, the situation for competing models is the opposite: when an interleaved imagination rollout mechanism is forcibly added to BAGEL, RxBrain, and Cosmos 3 Nano, their average scores on Rynn-GEU drop from 11.9, 31.4, and 37.3 to 10.8, 28.6, and 33.5, respectively. Video imagination does not unconditionally add value; if the generated frames are distorted or deviate from real scenes, erroneous information propagates along the reasoning chain, dragging down subsequent inference.

Test results on the long-horizon planning benchmark Rynn-VP show that RynnUnify achieves a total score of 54.80, a 9.75-point improvement over Cosmos 3 Nano's 45.05. Breaking down the metrics, planning intelligence increases from 30.50 to 45.66, a gain of 15.16; visual realization improves from 54.58 to 60.54; however, the image quality score drops slightly, from 79.69 to 77.53. The team attributes this gain to a combination of multiple mechanisms: embodied fine-tuning, cross-segment scene memory, self-forced training, and a design that estimates action duration per subtask. However, the upper limit of externally pipelined systems remains higher. The combination of GPT-5.6 Terra and MiniMax-H3 scores 65.71, which is 10.91 points higher than RynnUnify. Therefore, the conclusion of being "the best among unified models" is strictly limited to the scope of standalone models.

Direct connection of video features to the action head

To integrate scene understanding, future imagination, and robot actions, the team additionally trained a world action model.

• The reasoner receives the instruction, the current three-camera views, and visual historical information. It can take up to 12 frames of image input and outputs a continuous sequence of subtask latents;
• The generator uses this latent variable as a condition to predict subsequent video sequences;
• The action head directly reads the video hidden state features output by the generator. It consists of 18 blocks. Relying on cross-attention, each block connects to two layers of the generator network. Through flow matching, it outputs an action chunk containing 48 steps and 12 dimensions, of which the robot actually executes only the first 16 steps.

Throughout the pipeline, the action head reads low-level video features without needing to decode and render complete frames, thereby saving the computational overhead of this intermediate step. There is also a camera pose head used only during the training phase to provide geometric supervision signals.

The robot training data is sourced from RoboCasa365, comprising approximately 1,600 hours of atomic task trajectories and an additional 177 hours of composite task trajectories. Each sample includes an instruction, three synchronized camera views, an action sequence, and subtask annotations.

The training process is divided into three stages:

In the first stage, only the language layer of the reasoner is trained while the visual encoder and generator remain frozen, enabling the model to identify the current subtask from the instruction, real-time visual frames, and historical observations;

In the second stage, the reasoner is frozen and the generator is trained independently. The video loss and native action loss are weighted equally. Each sequence segment comprises 48 frames at 20 Hz, corresponding to a duration of 2.4 seconds;

In the final stage, the generator and action head are jointly trained. The total loss comprises the action loss, the pose loss weighted at 0.01, and the cross-camera loss weighted at 0.005. During inference, a single forward pass extracts the video features. Action sampling reuses this same batch of features and then applies 16-step Euler integration to output the final action chunk.

In the simulation environment, the Selected-20 benchmark includes 20 composite mobile manipulation tasks. Each task is tested 10 times, totaling 200 experimental trials. RynnUnify achieves a 51.5% success rate; for comparison, CMA achieves 47.0%, Xiaomi-Robotics-1 achieves 46.0%, and Cosmos 3 achieves 45.0%. The gap is even more pronounced when compared to standard vision-language-action policies: π0 achieves only 3.0%, π0.5 achieves 2.0%, and GR00T N1.5 achieves 7.5%. Such policies often stall at the initial stage when faced with composite long-horizon tasks.

On the Composite-Unseen set of novel composite tasks, RynnUnify achieves a 50.0% success rate, outperforming CMA by 6.0 percentage points and Cosmos 3 by 11.0 percentage points. On the Composite-Seen set of familiar tasks, it scores 52.0%, ranking second behind Xiaomi-Robotics-1, which leads with 57.1%. Its performance is actually more stable on unfamiliar task combinations, which is the key factor that sets it apart from other baselines.

Real-robot experiments are conducted on the Astribot S1 robot, featuring three tasks with 20 repetitions each. Turn-and-grasp requires the robot to rotate its base before extending its robotic arm, testing target localization and limb coordination after a change in viewpoint. Block swap requires exchanging the positions of a small red block and a small blue block, testing the precision of grasping and placing small objects. Bowl-thrice-then-spoon involves consecutively grasping and putting back a bowl three times before grasping a spoon. Since the visual frames in each round are highly similar, the model must rely solely on historical information to determine the current execution step.

Across the three tasks, RynnUnify achieved full success rates of 55.0%, 35.0%, and 50.0%, respectively, for an average success rate of 46.7%. Compared to the baselines, GR00T N1.7 averaged 26.7% and π0.5 averaged 25.0%. In the bowl task specifically, RynnUnify outperformed the strongest baseline by 10 percentage points, directly demonstrating that the visual history memory mechanism is indeed effective.