Two papers from Fudan University introduce ActiveAlign and AWAM, which align training data and representations with active camera movements to prevent distribution shifts in vision-language-action models when robots adjust their viewpoints to overcome occlusions.

Two papers from Fudan University's Yanwei Fu team: robots perceive first to execute actions from novel perspectives

When a robot reaches to grasp a cup obscured by clutter, the most intuitive solution is to rotate its wrist-mounted camera for a fresh observation. However, rotating the camera simultaneously changes the robot arm's configuration. The policy then encounters an initial state never seen during training, completely misaligning the correlations between the image, the pose, and the remaining task.

The team quantified this negative impact. After integrating a camera relocalization pipeline ahead of a mature vision-language-action model, the average success rate under the LIBERO perturbed start protocol dropped from 46.3% to 11.5%. Another backbone model's success rate fell from 30.7% to 7.0%.

On October 16, a joint team from Fudan University and the Shanghai Chuangzhi Institute published two papers: Look Before You Act: Bridging Active Perception and Vision-Language-Action Models via Observation-Aligned Data Adaptation and AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation. Jiawei Geng is the first author of both.

The research group is led by Yanwei Fu, a tenured professor at the School of Data Science at Fudan University. His long-standing research focuses on few-shot learning, generative models, 3D vision, and embodied AI. Fu received the Second Prize of the National Natural Science Award in 2023. Previously, his team explored integrating virtual viewpoint selection into vision-language-action models.

The core idea of active perception is for the robot to first plan its observation position before generating an action. The inputs to vision-language-action models are highly dependent on camera pose. In cluttered and occluded scenarios, the target may fall entirely out of the field of view. Even a high-performing action model cannot compensate for missing observational data. Adjusting the camera to a viewpoint where the target is visible is the most direct way to implement active perception.

Previous approaches face two bottlenecks.

• Physical camera approaches incur execution costs: adjusting the viewpoint synchronously changes the robot's physical state, creating an interface disconnect between perception and manipulation. Early systems addressed this by using a dedicated perception arm or a movable neck to jointly learn observation and manipulation actions.
• Virtual viewpoint approaches avoid the cost of physical movement, but they rely on preset cameras restricted to fixed positions. Reconstructed rendering cannot restore object surfaces absent from the initial geometric information. Furthermore, existing world models are primarily used to predict future states resulting from actions, rather than actively querying unobserved viewpoints.

ActiveAlign and AWAM each address different aspects of this interface.

• ActiveAlign directly addresses the issue that camera rotation alters the policy's initial state. It uses the viewpoint module employed during deployment to construct training data, aligning existing demonstration samples to the state at the end of active perception. By performing closed-loop alignment, playback, and filtering of valid samples, it then fine-tunes the policy.
• AWAM treats the viewpoint as a virtual query. A task-conditioned camera trajectory drives a geometry-based world model, which compresses the observed frames along the path into a fixed-dimensional memory. During the manipulation phase, this memory is fed into the policy alongside real-time observations. Neither approach treats the viewpoint module as an external add-on during inference. Instead, during training, both leverage the modules used in deployment to generate the corresponding data or representations, jointly covering the entire pipeline from planning observation viewpoints to executing the manipulation.

ActiveAlign improves the π0.5 baseline from 46.3% to 72.7% across 40 tasks on the LIBERO dataset. Meanwhile, AWAM achieves a 75.9% success rate on the LIBERO-Occ dataset, enabling seamless robot manipulation in occluded environments.

Training reuses the deployment camera

Both studies stem from the same core insight: when a robot's target is occluded, the most effective solution is to move the camera to acquire a visible viewpoint. However, this step directly alters the input conditions for the action policy.

Vision-language-action (VLA) models map observations and language instructions directly into actions, with the observations entirely determined by the camera pose. When the camera moves, the positions, scales, and occlusion relationships of objects in the frame change, and the robot arm configuration shifts accordingly. The policy must then infer subsequent actions based on an entirely new combination of visual and proprioceptive inputs.

The team termed the distribution shift at this handover point "first-frame distribution sensitivity," where the "first frame" refers to the observation passed from the active perception stage to the manipulation policy.

Both studies share a unified input setup: the robot is equipped with a scene camera and a wrist camera. The inputs include language instructions, proprioceptive states, depth maps, and camera calibration parameters, and the policy outputs an action chunk.

The difference between the two approaches stems from their distinct assumptions about viewpoint changes.

• ActiveAlign drives the physical rotation of the wrist-mounted camera, causing the robot body to move accordingly;
• AWAM moves only a virtual camera along a planned trajectory while keeping the robot body stationary; this trajectory is never executed on the physical robot.

The wrist-mounted camera is the root cause of this issue. Because the camera is rigidly attached to the end of the robotic arm, any change in viewpoint synchronously alters both the image and the arm configuration, deeply coupling the two variables in the training data. By switching to a virtual camera, the images can change while the robot's physical configuration remains static, effectively decoupling these two variables.

This image is an experimental data table titled "TABLE I PERTURBED-START LIBERO SUCCESS RATE (%)," showing LIBERO task success rates under different settings. The table is divided into π₀ and π₀.₅ settings, each containing three sub-items: Standard, Naive Active, and ActiveAlign. These are followed by success rate data across four dimensions: Spatial, Object, Goal, and Long, as well as the average (Avg.) success rate. The ActiveAlign setting shows significantly higher success rates across all dimensions and on average compared to the other two settings; for instance, under the π₀.₅ category, ActiveAlign achieves an average success rate of 72.7%, far exceeding the 46.3% of Standard and the lower rates of Naive Active in the same category.

The team first quantified the impact of this distribution mismatch.

Under the LIBERO perturbed-start protocol, prepending a camera relocalization module to π0.5 while keeping all other conditions constant caused the average success rate across four task suites to drop from 46.3% to 11.5%; the π0 baseline fell from 30.7% to 7.0%. The action chunking mechanism amplifies early errors, which then accumulate continuously along the executed action sequence. Even when new images become available later, the initial errors are already entrenched.

Active perception yields higher-quality observations, but an action policy can only process observations compatible with its training distribution; these two factors must be evaluated separately. The team designed a controlled experiment to decouple viewpoint information, robot state evolution, and data volume variables, demonstrating that the performance drop is not solely caused by image distribution shift.

AWAM circumvents this mismatch through an alternative approach.

Its planned virtual camera trajectory is never executed on the physical robot. The robot remains stationary with its configuration unchanged, relying solely on a world model to infer the virtual views along the trajectory. By keeping the robot still and prioritizing observation, AWAM prevents state mismatch at the source. The trade-off, however, lies in information availability: the virtual viewpoint can only extract features from the initial RGB-D back-projected point cloud. For object surfaces never captured by the physical camera, the world model must synthesize the missing details.

The two approaches establish a clear division of labor.

• ActiveAlign directly addresses the full state misalignment caused by physical camera rotations, modifying the dataset to retrain the policy on the new observation and state distributions;
• AWAM handles scenarios where the robot's proprioceptive state remains unchanged but observational information is insufficient, modifying the representation to compress virtually inferred viewpoints into a context the policy can continuously read.

Both approaches integrate the deployment-stage viewpoint module into the training data construction pipeline: ActiveAlign uses it to generate new demonstration starting points, while AWAM uses it to generate memory inputs, ensuring that training and inference follow consistent trajectory logic.

Generating trajectories to tell the robot where to look

The viewpoint module in ActiveAlign aims to derive the camera's motion path based on the current scene and language instructions.

The module splits RGB and depth information into two streams of CLIP features, encodes them alongside the language instructions, and then uses a conditional autoregressive decoder to generate quantized camera pose tokens token by token. By default, a single trajectory contains 30 poses, which can be interpolated to 120 for smoother execution. These poses are expressed in SE(3) space and then transformed into the calibrated robot coordinate frame, where they are passed to the controller to generate candidate observation actions.

The team trained this module using 20,000 simulated trajectories and 1,200 real-robot trajectories. This dataset was collected separately from the subsequent manipulation demonstration data.

The image shows active perception trajectories in LIBERO and RLBench. The top half displays LIBERO tasks, including placing a cup into an occluded rear tray and placing butter into a partially occluded basket, with the robot operating from different viewpoints. The bottom half shows RLBench tasks, such as putting money into a partially hidden safe and placing groceries into an occluded cabinet space, also featuring operations from various viewpoints. The image closely relates to the context, intuitively presenting the team's experimental results on robot viewpoint selection across different tasks.

The predicted path serves only as a candidate action; the team does not guarantee its optimality or collision-free properties. In the simulation environment, inverse kinematics moves the camera to the target pose. On the real robot, the controller drives continuous motion, and a downstream alignment and playback process filters whether the trajectory is usable. The viewpoint module proposes positions worth observing, while the effectiveness of those positions is determined by the manipulation outcome. This division of labor allows the perception and manipulation modules to be trained independently and ultimately matched at the data level, eliminating the need for end-to-end joint coupling of the two models.

AWAM's trajectory predictor employs an alternative parameterization scheme. It represents each frame as a transformation matrix relative to the initial wrist camera, then decomposes it into rotation, normalized translation, camera-assisted coordinates, and a trajectory scale variable. These are discretized into 256 tokens and generated autoregressively via a causal Transformer. The prediction process does not incorporate absolute poses in world coordinates; therefore, the output is independent of the scene origin, eliminating the need to recalibrate global coordinates when changing rooms. From a single predicted trajectory, the team selects 21 representative viewpoints based on the camera's cumulative motion: segments with intense motion are sampled more densely, while nearly static segments are sampled less frequently, preventing the viewpoint budget from being wasted on unchanging areas.

The image illustrates virtual viewpoint predictions along planned trajectories for two tasks. The left task is "Pick up the alphabet soup and place it in the basket," and the right task is "Open the middle drawer of the cabinet." The figure includes the initial robot viewpoint (yellow), predicted viewpoints (blue), and ground-truth viewpoints (green), demonstrating progressively improved visibility. Here, Warp Input is shown in orange and 1 Step Denoise in blue. Closely related to the context, this image intuitively presents the trajectory generation effects of both methods, echoing the discussion on how trajectory length impacts success rates.

Both modules draw on existing research related to camera trajectory generation.

• The ActiveAlign module is driven by language instructions: language defines the search target, and scene geometry determines visible orientations, with both types of information serving as conditional inputs;
• The AWAM predictor is initialized with a pre-trained text-RGB-D camera trajectory model. The visual and language encoders are frozen during the adaptation phase, and only the trajectory projection layer and the causal Transformer are trained.

Trajectory length affects both schemes: increasing the number of poses from 5 to 30 in ActiveAlign raises the success rate from 56.6% to 78.2%; in AWAM, increasing the number of virtual viewpoints from 7 to 21 improves the success rate from 72.4% to 75.9%, with further increases yielding diminishing returns. Both schemes lead to the same conclusion: once the trajectory reaches a certain length, the acquired information is sufficient to support the task.

The inputs for both modules are essentially identical, consisting of scene RGB-D data paired with a single language instruction, and both output a camera trajectory. The difference lies in how they are used downstream.

• ActiveAlign requires only the endpoint of the trajectory; actions after this endpoint reuse existing demonstration data to continue;
• AWAM requires a series of viewpoints along the trajectory, allowing the world model to query scene information from various orientations.

The same camera trajectory uses one end as the starting point for a completely new task and the other end as a series of virtual queries. This difference gives rise to two entirely distinct integration schemes at the data and representation levels.

Planning viewpoints, applied to the robotic arm

ActiveAlign converts trajectories into training data through an offline reconstruction process.

The team resets the simulation environment to the initial state of each demonstration sample and uses the viewpoint module to predict and execute a camera trajectory, recording the end-effector pose when it stops. Next, they search for the closest demonstration state within a prefix window, which defaults to 5% to 30% of the entire demonstration. After matching, closed-loop control compensates for the residual between the two states, and the remainder of the demonstration's manipulation trajectory is replayed. Only samples with a completely successful replay are retained. The resulting data consists of fragments showing how to proceed after observing, while data from the camera movement phase is excluded from training. These aligned fragments, along with the subsequent manipulation suffix, are jointly incorporated into the training objective, enabling the policy to learn to continue the task from this new starting point during fine-tuning.

This filtering step directly determines data usability. A close end-effector position does not guarantee that the object and contact states match perfectly; therefore, nearest-state matching is used only to generate candidate samples, with final retention determined entirely by the success or failure of the replay.

In the simulation environment, the team processed 2,000 samples and retained 1,736, achieving a retention rate of 86.8%. In real-robot experiments, they ran 100 trials and retained 86, yielding an 86.0% retention rate. Selecting the window further along the trajectory increases the retention rate but reduces the diversity of available endpoints; the intermediate range yields the best task success rate. While this data construction pipeline requires an environment that supports resetting or a workstation that can be reconstructed, neither environment resets nor replay operations are needed during deployment.

The researchers conducted parameter sweeps for both the window range and the endpoint error. Setting the window to 5%–30% achieved the highest success rate in this set at 78.2%. Shifting the window later to 10%–40% increased the retention rate from 86.8% to 88.2%, but the success rate dropped to 76.6%. A later window results in shorter retained manipulation fragments and more concentrated available endpoints, meaning the replay must reproduce more complex scene variations. Here, the retention rate and endpoint diversity trade off against each other, making the intermediate range the most stable.

During inference, the team independently perturbed the predicted endpoint camera position. A translation error of 0.02 meters decreased the success rate by 6.0 percentage points, while a 0.04-meter error decreased it by 9.0 percentage points. When the error reached 0.10 meters, performance fell below the baseline training scheme. This perturbation experiment is independent of the initial pose noise in the base protocol and specifically quantifies the cost of errors originating from the viewpoint module itself. This data construction pipeline has an upper tolerance limit for endpoint error; once the error exceeds the post-alignment distribution, the performance gains disappear.

AWAM converts trajectories into representations by constructing a memory. The team first back-projects RGB-D data captured by the initial static camera and the wrist camera to generate a colored point cloud, which is then rendered across 21 predicted viewpoints. Rendering can only reconstruct already observed object surfaces and cannot generate unseen regions; on the LIBERO-Occ dataset, the average void rate is 39.29%, with peaks reaching 74.47%. To fill these missing regions, the team employs WorldStereo as a geometry-conditioned world model, predicting intermediate features that match the corresponding viewpoints along the virtual trajectory. The study uses only these intermediate features, while the rendered RGB images serve solely for qualitative visualization. The features from each viewpoint are pooled into a 16×16 grid and then compressed by a lightweight query encoder into a fixed memory of 294 tokens, which is cached for the duration of a single task episode.

At each replanning step, the policy uses real-time observations as queries to extract keys and values from this fixed memory, performing a residual cross-attention computation. The fused representation is then combined with proprioceptive data and language instructions, and fed into a conditional diffusion Transformer to output an 8-step action chunk. The memory is constructed only once, but the retrieved content changes dynamically based on the current observation; as the robot arm moves to different positions, the retrieved predicted context also changes. This combination of a fixed memory and state-dependent dynamic retrieval is precisely what allows the method to continuously provide spatial evidence throughout the entire task episode. When training the action policy, the team also uses the predicted trajectory to build the memory rather than the wrist trajectory from the original demonstration, preventing a distribution mismatch between the input memory during training and inference.

The entire training pipeline is decoupled into sequential modules across three stages. The team first trains the trajectory predictor, then adapts the geometry-conditioned world model, and finally trains the action policy. Action policy training itself is split into two steps: first, the real-time visual projection, resampler, memory encoder, and fusion module are aligned while the pre-trained action denoiser remains frozen; second, these connecting modules are frozen, and the diffusion action model is adapted independently. During action policy training, world model features are pre-computed and cached, eliminating the need to run the foundation model at every step. This establishes a fixed representation interface between perception prediction and action learning.

The image provides an overview of the AWAM (Active World Memory) framework. Part A is task-oriented viewpoint planning, executed once per episode, which generates virtual viewpoints based on initial static RGB-D data and instructions, predicts the trajectory using an autoregressive trajectory predictor, and finally selects viewpoints. Part B is viewpoint imagination and world memory, also executed once per episode, starting from the initial dual-view RGB-D data and building the world memory through steps such as multi-view feature generation. Part C is closed-loop control; during each replanning step, it uses static/real-time RGB data and modules like visual encoders to fuse the cached world memory with real-time visual observations to predict action chunks.

The division of labor between the two approaches is most clearly delineated here.

• ActiveAlign modifies the task starting points observed by the policy, enabling the underlying model architecture to relearn action outputs under novel combinations of observations and proprioceptive states;
• AWAM modifies the context accessible to the policy, continuously feeding scene information beyond the current field of view into the model input as memory.

One approach encodes viewpoint information directly into the dataset, while the other embeds it into the feature representations. Although their implementation paths differ, their core objective is identical: when the observation viewpoint shifts, the policy's input conditions must be co-designed accordingly.

Occlusion scenarios: performance results for both approaches

ActiveAlign's simulation evaluation was conducted across four LIBERO suites, each comprising ten tasks evaluated over 50 rollouts per task. Gaussian noise with a standard deviation of 0.1 was added to the initial end-effector pose, and this perturbed initialization protocol was kept consistent across all experimental groups. The average success rate of π0.5 improved from 46.3% to 72.7%, showing performance gains across all four suites. The team compared three configurations—an external camera module, standard training, and data alignment—under the same initialization protocol. The performance differences stem from the training data rather than the evaluation metrics.

This data construction pipeline transfers effectively to other backbone models. The success rate for π0 improved from 30.7% to 70.7%; when swapped for OpenVLA and GR00T N1, both achieved gains of over 10%. These four backbones span three distinct action modeling paradigms: flow matching, autoregressive tokens, and diffusion Transformers. The larger the initial state offset, the more pronounced the value of this approach. Under the benchmark's nominal initial state, the performance of both groups is nearly identical. However, as perturbation increases, the standard training pipeline's performance drops steadily to 46.3%, while ActiveAlign maintains 72.7%, with the performance gap widening monotonically with the perturbation magnitude. This gain comes from training the policy on the task initializations it will actually encounter during deployment—an effect that cannot be replicated by simply scaling up the dataset.

The team designed two sets of controlled experiments to rule out alternative explanations of a 'shorter prefix' or 'more viewpoints.' When fixing the demonstration step count and suffix length while reverting the camera extrinsics to the default viewpoint, the success rate was 54.6%. When using random viewpoints and large viewpoint jitter for data augmentation, the maximum success rate reached only 61.4%. Both results fall short of ActiveAlign's 78.2%. The true driver of this gain is aligning the training distribution with the viewpoint module used during deployment.

Initial occlusion scenarios best demonstrate the advantages of this approach. In LIBERO-Occ, the target is occluded from the very first frame, which is precisely the scenario where view selection proves effective; the performance of the four suites is 28.6 to 34.6 percentage points higher than that of standard training. The real-robot experiments were configured with three levels of occlusion, with 30 trials conducted for each level, increasing the total number of successful samples from 35 out of 90 to 83 out of 90. Another set of tasks included cup grasping, container retrieval, and observing open-and-close grasp-and-place. However, the number of successful completions for these long-horizon tasks dropped to 13, indicating that a single view selection cannot cover the entire task sequence.

The image is a table showing physical occlusion success rates under the π0.5 setting, measured in successful trials/total trials (percentage). The table is divided into three columns: occlusion level (Partial, Medium, Severe) and method (Standard, Naive Active, ActiveAlign). Under different occlusion levels, the Standard method achieves success rates of 53.3%, 36.7%, and 26.7%; the Naive Active method achieves 26.7%, 16.7%, and 13.3%; and the ActiveAlign method achieves 96.7%, 93.3%, and 86.7%. In terms of overall success rates, Standard is 38.9%, Naive Active is 18.9%, and ActiveAlign is 92.2%. This table relates to the AWAM evaluation content in the document.

AWAM was evaluated on three benchmark datasets. On LIBERO-Occ, it achieved an average success rate of 75.9%, outperforming VIM, which is specifically designed for occlusion scenarios (74.0%), and also surpassing the world action model Fast-WAM (73.0%), with three of the four suites taking the lead. The Long suite is its weak point, achieving a 41.0% success rate, which is lower than Fast-WAM. This is because the memory is constructed only once based on the initial scene and is not updated throughout the process, causing it to fail in long-horizon tasks.

On the Occlusion1 subset of RLBench-OG, AWAM achieved an average success rate of 76.5% and an average rank of 2.05; five out of the ten tasks achieved a 100% success rate. Although the controller for this benchmark differs from that of LIBERO, the method remains transferable, demonstrating that the approach is not tied to a specific action interface. Across 10,030 instances in LIBERO-Plus, it achieved a 62.4% success rate, which is 8.8 percentage points higher than π0, with particularly significant gains under two types of perturbations: camera viewpoint and object layout.

Ablation studies were conducted to individually quantify the contribution of each component. Four variants maintained the same backbone network, training data, and evaluation protocol, with only specific modules removed or replaced. Removing the entire prediction branch and relying solely on real-time observations caused the success rate to drop from 75.9% to 64.4%. Replacing it with spherically uniformly sampled viewpoints reduced the rate to 70.2%, while applying only geometric transformations without generative prediction lowered it to 69.0%. The trajectories learned under task-conditioned settings outperformed spherical sampling, indicating that "where to look" inherently carries task-specific information. Furthermore, the superior performance of generative prediction over pure geometric transformations demonstrates that unobserved object surfaces must be completed using a world model.

In real-robot testing, AWAM was run 30 times each on the Apple, bottle, and block tasks, with the targets partially occluded in the initial observations. It achieved a total of 85 successful attempts, which is 19 more than the 66 successes recorded by π0. Meanwhile, Motus and Fast-WAM achieved 73 and 72 successes, respectively.

Both sets of experimental results point to the same conclusion.

• The ablation study of AWAM demonstrates that "where to look" inherently encodes task-specific information;
• The controlled experiments for ActiveAlign indicate that the quality of a viewpoint must be evaluated based on the module actually deployed.

Selecting the appropriate viewpoint is only the first half of the process; the performance gains can only be realized if the policy is trained to adapt to the new state following a viewpoint switch. The same research team completed both components on the same day. Only by combining the two can a complete interface be established for integrating active perception into manipulation policies.