Interview with Shanghai Jiao Tong University’s Yang Lixin: Why Robots Need a '3D Timeline'

Robots can pour water, yet after setting down the bottle, they may forget which one they just poured from. Being able to execute actions does not mean knowing how to advance a task; robots need to connect task intent, action history, and world changes into a three-dimensional timeline.

This type of failure is not entirely a problem of grasping precision. Today's Vision-Language-Action (VLA) models can map images and language into a sequence of actions, but training typically focuses on answering 'how experts move.' How task intentions translate into three-dimensional space, which step the operation has reached, and how actions will alter objects and the environment are often buried within action labels, forcing the model to figure it out on its own.

Yang Lixin, an assistant researcher at the School of Artificial Intelligence, Shanghai Jiao Tong University, summarized this gap directly: if a robot cannot determine which step it has already completed, it may choose the wrong next step. Over the past year or so, he served as the corresponding author for LaMP, ChronoFlow-Policy, and Track4Action, which were successively made public; the first two works have been accepted by ECCV 2026. All three works attempt to introduce the movement of 3D points over time into robot policies, but they target different positions: LaMP transforms task intent into future motion priors; ChronoFlow places the past, present, and future into the same type of interaction trajectory; and Track4Action uses a 3D tracker that has seen complete action outcomes to teach a VLA (Vision-Language-Action model) that only sees current observations during training.

Recently, 42 Radio Waves had an in-depth conversation with Yang Lixin, who fully outlined the research route centered on 3D point flow: how robots understand tasks, understand time, and comprehend changes in the world after actions occur.

Experiments also demonstrate that this supervision is more than an architectural embellishment. In LIBERO-Plus's seven zero-shot perturbation categories, after adding a motion expert, LaMP's average success rate increased from 71.6% to 79.3%; ChronoFlow's full model reached 93% and 61% respectively in the final stages of two "swap objects" tasks, which dropped to 20% and 11% after removing historical flows; Track4Action achieved an average overall task success rate of 67.5% across four real-robot dual-arm tasks, 25 percentage points higher than its untracked alignment ablation version with the same base.

Action commands are given to the robot's hands and feet; the three tasks filled in are spatial awareness and temporal awareness: where the target is located, how far the task has progressed, and what results a single action left behind all become the basis for the next decision.

The Three Papers Are Not Released in Series; Point Flow Is the Common Thread

Based on the chronological order of their publication, it would be easy to frame these three works as LaMP solving a problem, ChronoFlow identifying the next one, and Track4Action completing an upgrade. Yang Lixin corrected this narrative during the interview: the three research efforts largely took shape around mid-2024 and were advanced in parallel around 2025; they simply entered the public eye at different times due to varying project paces.

What is truly continuous is the "representation" problem.

Yang Lixin’s early research focused on human-hand and object interactions, aiming to find an intermediate language capable of bridging humans and diverse robotic bodies. In Motion Before Action, the team utilized six-degree-of-freedom (6-DoF) poses of objects: whether it was a human hand or a robotic arm pouring water, the bottle underwent some form of spatial transformation. However, pose representation relies on reliable pose trackers and is better suited for rigid bodies. When facing jointed objects like drawers, multi-object interactions, or flexible objects such as towels and clothes, a single rigid body pose quickly becomes insufficient.

Dense flow offers an alternative description method: it records the motion trajectories of a set of spatial points over continuous time, obtainable via depth sensing, geometric estimation, or 3D tracking. When all points move together, it can describe rigid bodies; when different parts move under constraints, it describes jointed objects; and when points deform individually, it covers flexible objects. In Yang Lixin’s view, because this representation does not directly equate to the joint commands of a specific robotic arm, it has the potential to serve as an interface between human video, different robots, and action policies.

The three papers do not use identical data structures. LaMP uses dense 3D scene flow hidden states, ChronoFlow tracks sparse 3D keypoints of objects and grippers, and Track4Action distills aggregated features from world-coordinate 3D trackers. What they share is the same overarching problem: rather than just having the robot memorize "how the arm moves," it should simultaneously learn "how the task-relevant 3D world changes with actions and time."

LaMP: Translating "What to Do" into "How Things Should Move"

LaMP originates from a gap in Vision-Language-Action (VLA) models. Pre-trained vision-language models excel at recognizing objects and understanding instructions such as "put the bread on the plate, then place the fried egg on top." The action head is responsible for outputting robotic arm commands. There is a missing layer of explicit geometric representation in between: for executing this instruction, how should the robot, bread, plate, and pan subsequently move in 3D space?

In the interview, Yang Lixin explained this bridge using the concept of "inverse kinematics": by inserting a dense flow module into the VLA, "this dense flow module predicts the future"; the subsequent Action Expert acts as the "inverse kinematics module" for the dense flow. The robot first translates the linguistic goal into the 3D changes that should occur in the scene, and then translates those changes into robotic arm movements.

Regarding the model structure, LaMP inserts a Motion Expert between the vision-language backbone and the action head. Current observations and task instructions first enter the vision-language backbone, after which the Motion Expert generates a partially denoised 3D scene flow representation. The system does not wait for it to fully reconstruct future trajectories; instead, it takes the intermediate hidden states, sends them to the Action Expert via gated cross-attention, and the latter generates action blocks.

Therefore, the statement 'LaMP enables robots to understand tasks' requires qualification. It does not specifically evaluate natural language understanding, nor does it prove that the model has acquired general causal reasoning capabilities. What the paper truly validates is: when task semantics are further constrained into 3D motion representations, action policies become more stable in terms of spatial variations and long-horizon tasks.

In LIBERO-Plus, all models are trained using only the original LIBERO demonstrations and then directly face seven types of perturbations: camera, robot appearance, language, lighting, background, image noise, and layout. LaMP achieves an average success rate of 79.3%, which is 9.7 percentage points higher than OpenVLA-OFT's 69.6% in the same table, and 7.7 percentage points higher than its own variant without the Motion Expert (71.6%). It scores highest among the four perturbation categories, but not all seven. The average success rate on standard LIBERO reaches 98.3%, only 0.6 percentage points higher than GeoVLA's 97.7% in the same table; what better reflects methodological differences is LIBERO-Long, where removing the Motion Expert causes performance to drop from 96.7% to 78.2%.

The team also specifically examined the predicted point flows. Yang Lixin recalled that their greatest concern was that 'the results predicted by the point flow might actually be chaotic, yet Action performance improves,' which would make it impossible to explain where the gain came from. The visualization results showed: 'The point flow indicates a specific motion direction, and the Action aligns with that direction.' In other words, as scores improved, this layer of 3D motion prior indeed pointed toward the actions subsequently taken by the robot.

The interpretation of real-world robot results requires restraint. The team tested grasping, towel folding, and bread making on a single Flexiv robotic arm. The primary metric was the phased cumulative 'task completion progress,' not binary success rates. LaMP achieved average progress scores of 80%, 50%, and 80% across three in-distribution tasks, with a summary average task completion progress of 62.5% across three out-of-distribution tests involving layout, objects, and backgrounds. Each in-distribution test consisted of only 10 trials, while each out-of-distribution condition involved 5 trials per task. These results indicate that 3D motion priors can yield benefits in small-scale real-world experiments, but cannot yet be extrapolated to reliability in open scenarios.

LaMP's engineering advantage lies in the fact that the robot does not need to fully generate a future trajectory before deciding on the next step. The Motion Expert performs only one partial denoising step, directly handing over 3D motion features to the Action Expert; on an RTX 4090, the complete system takes 177.1 milliseconds for a single policy query and outputs 16 steps of actions at once. Calculated per action block, its action generation rate is approximately 90Hz. Through this shorter pathway, LaMP embeds 3D geometric foresight directly into VLA action decision-making.

ChronoFlow: Robots Need to Know 'Where They Are in the Sequence'

While LaMP primarily connects task instructions to future motions, ChronoFlow addresses temporal ambiguities that cannot be resolved from a single current frame.

"If point flow can serve as an interface representing the future, could it also serve as an interface representing the past?" said Yang Lixin. Starting from this rhetorical question, ChronoFlow transforms point flow from a 'future roadmap' into an 'action history': it not only anticipates how to move next but also retains how the robot just interacted with objects.

When swapping two objects, an intermediate frame might correspond to either 'temporarily placing down the first object' or 'returning the second object to its target position'; when checking drawers one by one, closing an empty drawer does not reveal which compartment was just inspected. Such tasks exhibit non-Markovian properties: correct actions depend not only on current observations but also on past interactions.

ChronoFlow uses sparse 3D keypoints of objects and grippers to place past trajectories, current states, and predicted future trajectories into a unified coordinate system. The policy receives current RGB-D point clouds and historical ChronoFlow data; a diffusion model recovers future interaction trajectories from noise while decoding actions. It focuses on how the gripper and task objects move together, rather than incorporating every background point in the entire scene representation; for folding towels, this also avoids the difficulty of describing flexible deformation using a single object pose.

The most compelling results come from ablation studies on the same architecture with history. In real-robot Swap-Easy experiments, both the complete model and the version without history achieved 93% success in the first phase, indicating both could initiate the task; by the third phase, the complete model remained at 93%, while the no-history version dropped to 20%, a difference of 73 percentage points.

In the more difficult Swap-Hard task, both versions scored 94% in the first phase, but in the final phase, the scores were 61% versus 11%, a difference of 50 percentage points. Each task was evaluated 15 times, and Swap-Hard was evaluated 18 times; the paper did not provide confidence intervals. Compared to HistRISE, which already utilizes historical observations, the advantage is significantly reduced: for Swap-Easy, the final phase results are 93% versus 80%, and for Swap-Hard, 61% versus 56%. Both sets of results demonstrate the same advantage: when initial success rates are identical, historical point flow reintroduces previously occurred interactions into decision-making, allowing the policy to continue selecting the correct next step during the latter half of the task where frames are similar and stages are easily confused.

ChronoFlow first validates whether "time memory is truly useful" within single-task visual-motor policies. Yang Lixin summarizes this approach as: "First, thoroughly study the Policy, then apply some of its conclusions to scaling up these large models." Therefore, this version focuses on single-task visual-motor policies without language input; the converged problem setting makes the role of historical point flows more evident.

This "action record" has also entered the real-robot execution pipeline. During deployment, TAPIP3D asynchronously extracts historical object point trajectories, while gripper trajectories are calculated directly via robot kinematics; the asynchronous scheme increases the average operating frequency of the synchronous pipeline from 0.93Hz to 5.82Hz, approximately 6.3 times the original rate. ChronoFlow thus provides a clear, runnable time interface: organizing sparse 3D trajectories around task-relevant objects and grippers, placing past, present, and future into a unified representation, enabling the robot's next step to follow seamlessly from the previous one.

When point flows can serve as both a "future roadmap" and an "action record," the question can advance further: Can these 3D experiences be absorbed by the model during training, allowing only the necessary representations for action to be carried over to deployment? Track4Action chose this path.

Track4Action: Letting the "Teacher Who Has Seen the Results" Leave After Training

A robot demonstration contains two types of information. The action sequence tells the model what commands the expert issued; the video strictly aligned with these actions records what those commands actually achieved: which points moved, whether objects were lifted, how occlusions changed, and whether the camera rotated. Conventional VLA primarily uses the former, while Track4Action attempts to incorporate the latter as supervision.

The team faced a key choice: since powerful trackers have already observed the changes before and after actions, should the VLA imitate the trajectories they output, or learn their internal representations before forming those trajectories? Yang Lixin framed the issue specifically: "If this Tracker is very strong, should we use its results better, or its intermediate representations?"

Explicit trajectories compress change into discrete displacements at each point, whereas the intermediate representations of trackers can retain richer correlations among scenes, motion, occlusions, and camera changes. Track4Action chooses the latter.

During training, the frozen Track4World model reads K+1 frames corresponding to K actions, extracting scene, motion, visibility, and camera features in the world coordinate system from already recorded demonstrations, then aggregates them into a teacher representation.

The VLA student only sees the current frame, task instructions, and optional robot states; a set of learnable track queries approximates the teacher representation from this current information and enters the action head via gating. Yang Lixin summarizes this teacher-student relationship as: 'Letting a VLA that cannot see the future align with the intermediate representation of a point-flow tracking model that can.' Here, 'seeing the future' occurs during demonstrations fully recorded at training time; at deployment, the K+1 frame action-aligned segments, Track4World, teacher features, and alignment branches are no longer needed, while the VLA backbone, track queries, and gated fusion remain retained.

Four real-world dual-arm tasks provide the most direct controlled evidence. Track4Action achieves an average overall success rate of 67.5%, compared to 42.5% for the ablated version without tracker alignment on the same base, an improvement of 25 percentage points. In three real-world task-perturbation combinations—pepper task layout change, towel color change, and cabbage task desktop background change—the complete model's average success rate rises from 23.3% (no alignment) to 50.0%, an increase of 26.7 percentage points.

Among these three preset real-world visual perturbations, the tracking alignment path still yields success rate gains relative to the unaligned version, extending the effectiveness of this training signal from original scenarios to tested perturbation conditions.

Simulation results continue this trend. Track4Action achieves 82.3% on LIBERO-Plus, 7.6 percentage points higher than the unaligned version and 3.0 percentage points higher than LaMP; relative to the unaligned version, it improves across all seven perturbation categories. In RoboTwin 2.0’s 50 dual-arm tasks, the complete model reaches 80.44% and 81.48% under clean and randomized settings, respectively, outperforming the unaligned version by 39.12 and 41.58 percentage points.

In multiple experimental setups involving single-arm and dual-arm configurations, as well as simulation and real-robot scenarios, the same architecture consistently demonstrated gains in comparative controls. This indicates that this tracking distillation pathway can effectively convert action outcome information into control benefits across these various settings.

The engineering value of Track4Action thus falls on a very specific path: using a complex three-dimensional tracker to refine continuous demonstrations, compressing the world changes that were not fully utilized by action labels into the action representation of VLA. For existing data, this is equivalent to obtaining a denser layer of three-dimensional supervision from the same demonstration; for deployment, the trained track queries and gated fusion directly participate in action decisions.

Putting the three papers together reveals that they follow the same manifold line, starting from task intent, action history, and action results respectively, to supplement robotic strategies with geometric foresight, temporal memory, and world change supervision.

One explicitly provides geometric foresight, one preserves action history online, and one absorbs tracker experience during the training phase. Together, these three push point clouds from visualized motion trajectories to the robot strategy interface layer, corresponding to three engineering choices: how explicit the representation should be, how supervision scales, and how much computation remains on the deployment side.

The Role of Qiongche: Subjecting Point Flow to Company-Level Data and Real-Machine Task Validation

Qiongche stands at one end of this research pipeline, scaling academic ideas to engineering scale. Yang Lixin says that papers can first judge whether a method is effective on fixed datasets and benchmarks; at the industrial level, specific tasks may even "require you not to be so intelligent, but your success rate must be very high." Company-level data, real-robot platforms, and business tasks will continue to ask: whether gains are preserved as data scales up, and whether robots can stably complete tasks when shifted to real-world scenarios.

The joint laboratory provides the organizational foundation for this "amplification" and "verification." According to Yang Lixin, the School of Artificial Intelligence and the School of Innovation and Entrepreneurship at Shanghai Jiao Tong University are responsible for student training and research, while Qiongche Intelligent opens up data, robot platforms, computing power, and engineering resources to the joint training team. Students can leverage these conditions to conduct experiments when performing teleoperation and real-robot post-training.

The data pipeline is the most concrete link in this chain. In its Noe-0 research preview, Qongche disclosed that RoboPocket and DM3 have connected three-viewpoint acquisition, task management, quality inspection, manual and annotation review, all the way to training data delivery; the company stated that the platform has accumulated approximately 97,400 hours of data duration, covering 47 cities. For point-flow routes, the significance of this infrastructure lies in the fact that continuous demonstrations must not only be "captured" but also keep actions, visuals, and world changes aligned, thereby providing a scalable entry point for three-dimensional trajectory extraction, data filtering, and training supervision.

Positions available for integration have also emerged on the model side. Noe-0 adopts joint modeling of world and action, allowing implicit world imagination and action planning to share representations; the three papers respectively address future motion, action history, and action outcomes, offering more specific point-flow training paths. Public information does not yet show that Noe-0 has adopted these paper modules, but both routes focus on the same problem: how to bring actions and the world changes they cause into the same training process.

Beyond visual geometry, Qongche's "force-centric" research route points to the layer of contact feedback. Yang Lixin stated that the team is also exploring incorporating force into the world model; he prefers to integrate haptic and tactile signals into post-training, fine-tuning, or high-frequency response layers to handle immediate adjustments after contact occurs. The combination of point flow and force/tactile sensing remains the next technical direction, providing a clear position for connecting three-dimensional movement research with Qongche's existing accumulation.

In the interview, Yang Lixin stated that these three research efforts are currently being converted into mass production projects. The team hopes to incorporate point-flow experience into Qongche's pre-training, mid-term training, or post-training to improve policy success rates across different scenarios. Specific plans are still under exploration, with no public performance figures available yet; however, Qongche's publicly disclosed joint laboratory planning has listed real-world data collection and model generalization as collaborative攻关 directions. Data, computing power, robot platforms, and engineering feedback are汇聚 into the same cooperation framework, providing conditions for point flow to move from paper experiments to larger-scale verification.

When Training Data Begins to Record "After Actions"

This research route first changes the meaning of demonstration data. A video is no longer just a pairing of images and action labels; it also preserves the three-dimensional state transitions before and after an action. As long as tracking models can extract objects, grippers, occlusions, and camera movements from existing continuous demonstrations, old data could potentially be reprocessed into denser supervision without requiring frame-by-frame manual annotation of point trajectories. Track4Action further demonstrates that complex teacher models need only appear during the training phase, handing over distilled experience to VLA systems at deployment.

It also brings "effective information density" into the competition for data scale. Yang Lixin observed that large-scale human first-person perspective data often contains only coarse labels for entire tasks, with many similar action blocks sharing a single instruction; collector head turns, hands falling to the edge of a fisheye lens, actions difficult to map to robots, or factory data that is highly repetitive but low in informational value may all be masked as total duration increases. In his view, some data is not poor quality but may simply not be suitable for robot training. A potential value of point flow and world change labels is turning "which changes are relevant to robot tasks" into learnable signals, which might also be used for data filtering in the future.

For the future form of VLAs, Yang Lixin leans towards a hybrid architecture: vision-language models responsible for long-range task decomposition and reasoning, world-action models handling continuous operations over shorter time horizons, and high-frequency signals like force and touch entering a rapid response layer similar to System 0. The three studies respectively demonstrate three uses of point flow in such hierarchical systems: connecting task semantics to three-dimensional motion, using action history to assist current decision-making, and extracting action outcome information from complete demonstrations.

Following this layered approach, the next phase of stress testing will focus on first-person human data, dexterous hands, high-contact tasks, and autonomous error correction. Yang Lixin hopes to continue researching first-person data, dexterous hands, and point flows; these directions require models to process longer task histories, finer three-dimensional deformations, and higher-frequency contact feedback, while more fully validating the value of point flows as intermediate interfaces.

In Conclusion

Returning to the robot at the beginning that forgets which bottle it just poured from, the three works provide answers from three different angles: LaMP projects task intent into future 3D motion, ChronoFlow marks task progress using historical trajectories, and Track4Action extracts changes in the world caused by actions from complete demonstrations. Controlled ablation studies for each of the three papers show that, across the covered tasks, adding these 3D temporal signals outperforms matching versions without the corresponding pathways.

For Qionghuo, the joint laboratory provides an engineering environment to continue testing whether such supervision can adapt to larger datasets, more platforms, and complex real-world robotic tasks. The next round of VLA competition may depend not only on how much data a model has seen, but also on its ability to read what traces every action leaves in the world.