Shanghai Jiao Tong University's Lu Cuwu, Zhejiang University teams propose Track4Action: Distilling the '3D World After Action' into VLA

VLA models can convert images and language instructions into robot actions. During training, researchers typically only evaluate whether the robot's actions resemble those of an expert. What the robot's actions will do to objects, spatial relationships, and task states is usually not explicitly taught to the model. Short tasks may mask this gap; once tasks become longer or perspectives, backgrounds, and object layouts change, robots are more prone to losing stable judgment for the next step.

In short, robots have learned actions but do not know how these actions will change the world.

On August 4, a team from Zhejiang University, Shanghai Jiao Tong University, Shanghai Innovation Institute, and Qiongche Intelligence submitted a paper titled "Track4Action: Distilling World-Centric 3D Tracker into Vision-Language-Action Policies," which proposes Track4Action to address this challenge in VLA models.

The author team is highly noteworthy. Lu Cuwu, a prominent figure in the AI academic community, is one of the authors. He currently serves as Co-founder and CEO of Qiongche Intelligence, Vice Dean of the School of Artificial Intelligence at Shanghai Jiao Tong University, and Professor. Additionally, Liang Yin Yang, Assistant Researcher and PhD Supervisor at the School of Artificial Intelligence, Shanghai Jiao Tong University, serves as the corresponding author.

The proposed Track4Action does not require the robot to run an additional large-scale 3D tracking system during execution. Instead, it places the tracking model as a 'teacher' during the training phase: it first reads changes in objects, the robot, and the camera within the 3D world from complete demonstration clips, then compresses these changes into features to teach the VLA, which only observes current inputs. After training, both the demonstration clips and the 3D tracker are removed, leaving only a VLA policy that directly generates actions from current images, language, and robot states on the deployment end.

This approach achieved an average full success rate of 67.5% across four real-world dual-arm tasks, a 25 percentage point improvement over the same architecture version without tracking alignment. In zero-shot evaluations on LIBERO-Plus without contact perturbation data, the average success rate reached 82.3%, a 7.6 percentage point increase over the unaligned version. The most significant change highlighted by the paper is not just a set of scores, but that demonstration videos now serve two types of supervision simultaneously: telling the robot what actions to take and showing how those actions should change the world.

What Action Labels Are Missing

The typical inputs for a VLA are current multi-view images, language instructions, and optional robot proprioceptive states; the outputs consist of joint commands, end-effector poses, or gripper actions. As long as the predicted actions are sufficiently close to expert demonstrations, the model can achieve a low training loss.

The problem is that identical end-effector displacements may represent entirely different events under different 3D relationships. A robotic arm moving forward could simply be traversing free space or making contact with a chili pepper and carrying it along. Similarly, an upward motion might successfully grasp an object or return empty-handed if the gripper failed to secure it. Action coordinates alone cannot distinguish between these scenarios, nor do language instructions provide frame-by-frame geometric feedback.

Demonstration videos, however, retain this information. Assuming an action sequence consists of K consecutive commands, the K+1 aligned frames fully record K state transitions: which 3D points moved, whether objects were occluded, how camera perspectives changed, and whether sustained contact was formed between the robot and objects. Track4Action's core objective is to transform this "already occurred world change" into privileged supervision during training.

Here, "privileged" has a specific meaning: the teacher can observe the complete post-action segment during training, while the student must infer 3D change representations related to the task based only on pre-action observations and task instructions. During deployment, the robot naturally cannot access future frames that have not yet occurred, so both the teacher and the complete segments disappear.

Real-World Benefits of Tracking Alignment

The paper's real-world platform comprises two six-degree-of-freedom Piper robotic arms equipped with parallel grippers, utilizing one frontal camera and two wrist-mounted cameras for RGB observation. The policy outputs 14-dimensional dual-arm actions per step, including six joint commands per arm and one gripper command.

Researchers selected four operations: placing red and green peppers into a plate; folding towels using both arms; sequentially placing two pens into a drawer and closing it; and putting a cabbage into a pot before closing the lid. These tasks cover rigid body pick-and-place, flexible object manipulation, dual-arm coordination, and multi-stage long-horizon tasks. Each task used 50 demonstrations for training, and each strategy underwent 10 test runs.

Track4Action achieved an average complete success rate of 67.5%, compared to the baseline π0.5 at 65.0% and the version without 3D tracking alignment at 42.5%. Compared to the non-aligned version, Track4Action improved by at least twenty percentage points on every task; for tasks involving multiple ordered stages such as "put pen away and close drawer" and "add cabbage to pot and cover," the improvement reached thirty percentage points.

To prevent a single late-stage failure from masking previously completed actions, the paper also divides each task into four equally weighted milestones. For example, in the drawer task, it separately checks whether two pens have been placed inside, whether the robot is holding the drawer, and whether it is successfully closed at the end. Track4Action's average process score is 75.0, which is 14.4 points higher than the 60.6 of the non-aligned version and matches π0.5. This indicates that the benefits brought by 3D change supervision appear not only in the final binary judgment of "success/failure" but also in the completion progress of multi-stage tasks.

The team also altered visual conditions not present during training in the real-world environment: moving the chili and adding distractors, changing the red towel to green, and placing a new textured background on the table. In these three out-of-distribution settings, Track4Action's average success rate increased from 23.3% for the unaligned version to 50.0%. Specifically, the chili transfer under new object layouts improved from thirty percent to sixty percent, and the cabbage task on the textured table improved from thirty percent to seventy percent.

The real-robot results provide direct support for the core judgment of the paper: although 3D tracking information only appears during training, the action representations shaped by it can indeed yield benefits on real robots without invoking the tracker.

How 3D Changes Are Distilled

The instructors of Track4Action come from Track4World. This is a world-centric 3D tracking model with frozen parameters, capable of organizing scene structure, 3D motion trajectories, visibility, and camera changes from short videos. The term 'world-centric' can be understood as follows: it does not merely track where a certain pixel has moved in a two-dimensional frame, but also strives to distinguish the camera's own movement from the actual movement of objects within the scene.

During training, Track4World reads main-view videos strictly aligned with action segments. K actions correspond to K scene transitions; the teacher encodes this video into tokens for scenes, motion, and camera, then aggregates them into a compact tracking feature. This feature is not the next frame image, nor an explicit trajectory for the robot to reproduce point by point, but a compressed representation of the entire three-dimensional world's changes.

The student side uses Qwen3.5-2B as the backbone for the VLA. Current multi-view images and language instructions are first encoded into multimodal hidden states, and a set of learnable 'track queries' then read these states via cross-attention, attempting to infer the 3D change features that the teacher has already seen from complete clips. Student features and teacher features are projected into the same space and pulled closer through an alignment loss.

These queries are not discarded after alignment is completed. Track4Action re-fuses the information learned in the queries into the VLA hidden sequence through learnable per-feature gating, which is then passed to a DiT-B-based flow-matching action head to generate action blocks. The entire model has approximately 3,300,000,000 parameters, and its training objective consists jointly of action generation loss and tracking feature alignment loss.

This connection is crucial. If 3D tracking were merely an independent auxiliary prediction head, even if the model performed well on the auxiliary task, it might not necessarily use this information for actions. Track4Action allows the same set of track queries to accept teacher supervision while directly participating in action generation, effectively placing 'how the world will change' into the representation channels that truly drive action decisions.

Teacher features themselves are not directly input into the action head. This design avoids the problem of relying on complete future clips during training but suddenly lacking inputs during deployment. During inference, what remains are the queries and fusion modules that have learned to infer relevant 3D changes from the current scene and task semantics; Track4World, complete demonstration clips, and the alignment branch are all removed from the execution chain.

Zero-Shot to Dual-Arm Simulation

If 3D change supervision only helps the model better memorize training scenes, it should not remain effective after changes in background, camera, and object appearance. Therefore, LIBERO-Plus constitutes a more targeted test for this paper.

This benchmark requires all models to be trained solely on original LIBERO demonstrations, without using perturbed data from LIBERO-Plus for adaptation, and then face seven types of variations: camera, robot appearance, language expression, lighting, background, image noise, and object layout. Track4Action's official average success rate is 82.3%, higher than LaMP's 79.3% and the unaligned version's 74.7%. Compared to the latter, none of the seven categories showed a decline; the three categories with the largest improvements were camera variation (18.3 percentage points), image noise (9.0 percentage points), and robot appearance variation (8.7 percentage points).

In the paper's tables, Track4Action ranks first in camera, lighting, noise, and layout categories, and second in robot, language, and background categories, placing within the top two across all seven perturbation types. For a method leveraging 3D motion supervision, the significant gain on camera variation particularly aligns with its design goal: the model must distinguish viewpoint changes from object motion based on learned world dynamics during training, rather than relying solely on appearance correspondences in fixed frames.

On standard LIBERO, Track4Action achieves an average success rate of 97.0%, an improvement of 2.8 percentage points over the unaligned version, surpassing π0.5 by 0.1 percentage points. The difference is more pronounced in multi-stage LIBERO-Long: with tracking alignment added, the success rate rises from 86.2% to 95.8%, an increase of 9.6 percentage points. When baseline scores for short tasks are already near saturation, maintaining task state across multiple stages better demonstrates the role of world change representation.

The team also evaluated performance on RoboTwin 2.0 using 50 dual-arm simulation tasks. Unlike the single-arm end-effector control in LIBERO, the policy here receives one external view and two wrist views, does not use robot state information, and directly predicts 14-dimensional dual-arm joint and gripper actions. The action block length also increases from 8 steps in LIBERO to 16 steps.

Track4Action achieves an average success rate of 80.44% under clean settings and 81.48% under domain randomization settings. As the strongest external baseline in the table, the unpretrained Motus model scores 77.56% and 77.00%, respectively; the same-scale version without tracking alignment scores 41.32% and 39.90%. In controlled comparisons where data, optimizer, training schedule, model scale, and execution interfaces are kept consistent, the complete tracking distillation path brings improvements of 39.12 and 41.58 percentage points, respectively.

Considering the three sets of experiments together, the location of gains is not random: when standard short tasks are already near perfect scores, incremental gains are relatively limited; however, when testing shifts to zero-shot visual variations, long-horizon tasks, different dual-arm action spaces, and real robots, the gap widens significantly. Based on this, the paper defines 3D tracking features as training supervision oriented toward actions, rather than merely an intermediate representation serving only visual reconstruction.

Spatial Intelligence Enters Actions

The method diagram of Track4Action places two types of models side by side: 3D trackers know how points in the scene change over time but do not understand what tasks humans require robots to complete; VLAs can correlate language instructions like "put the chili pepper into the plate" with the current visual frame, yet they are not explicitly required to describe the 3D motion in the next few seconds. The former leans towards space and motion, while the latter leans towards semantics and actions.

This work's approach does not involve running the two sets of models in parallel on the robot for an extended period, nor does it ask the VLA to generate additional high-definition future videos. Instead, it distills the representation capability of real-world motion from the 3D tracker into the action channel of the VLA. For deployment, the interface remains unchanged; for training, the same demonstration data gains an additional layer of supervision regarding state transitions in the world.

This perspective also offers another understanding of the value of robot data. A demonstration is not merely a pairing of images and action labels; it is a record of a real physical process. As long as pre-trained vision models can extract geometry, occlusion, camera movement, and object motion from it, historical demonstrations can be repurposed into denser training signals without requiring manual annotation for every intermediate variable.

In Conclusion

In the past, the most intuitive use of robot demonstration data was to enable models to imitate expert actions. Track4Action takes one step further: the action sequence serves as a command, and the video aligned with it represents the physical world's response to that command. Both were already present within the same demonstration, but the latter was underutilized.

For VLAs, this equates to inserting a layer of understanding of 3D state changes between "comprehending the task" and "outputting actions." When the robot goes online, it does not need to carry the teacher who observed the entire process, but it can retain the representations taught by the teacher within its policy.

If this training paradigm continues to expand to more robot bodies, more camera configurations, and longer manipulation sequences, 3D and 4D vision foundation models may not necessarily need to become controllers directly; they can instead serve as teachers for training VLAs. What Track4Action demonstrates is a clear version of this path: allowing spatial intelligence to enter training first, and then manifesting through actions.