Tsinghua University and Galbot Apply 552B Vision-Language Representations to Fine Robotic Manipulation

Researchers from Tsinghua University, Peking University, Shanghai Jiao Tong University, and Galbot introduce ScaleVLA, a vision-language-action model that leverages a frozen 552-billion-parameter vision-language backbone to improve data efficiency and fine manipulation in robotic tasks.

Vision-language models have acquired capabilities in object recognition, spatial understanding, and instruction following through large-scale image-text pre-training. State-of-the-art models are beginning to demonstrate robotic manipulation capabilities, yet the vision-language backbones of many vision-language-action (VLA) models remain relatively small.

Can the representations learned by large models from image-text data be translated into robotic policies that require fewer demonstrations and are better suited for fine manipulation?

On November 16, research teams from Tsinghua University, Peking University, Shanghai Jiao Tong University, and Galbot released the paper "ScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters" (translated into Chinese as "Scaling Vision-Language-Action Models to 552 Billion Parameters").

The paper has 12 authors in total. The four co-first authors are Ruixi Yu (Tsinghua University, Galbot), Dairu Liu (Peking University, Galbot), Xuchuan Chen (Tsinghua University, Galbot), and Zekun Qi (Tsinghua University, Galbot), with Zekun Qi also serving as the project lead.

The corresponding authors are He Wang, Assistant Professor at the School of Computer Science, Peking University, and Founder and CTO of Galbot; and Li Yi, Assistant Professor at the Institute for Interdisciplinary Information Sciences, Tsinghua University. Li Yi holds a Ph.D. from Stanford University, previously served as a research scientist at Google Research, and contributed to 3D vision research including PointNet++.

To investigate whether state-of-the-art vision-language representations can support robotic policies that require fewer demonstrations, the team proposed ScaleVLA. The approach freezes a 552-billion-parameter DeepSeek-V4.1-Flash vision-language backbone and uses an independently trained action diffusion Transformer to process images, instructions, and robot states, outputting continuous action chunks.

The team first compared vision-language backbones of varying scales in simulation, keeping the action module, demonstration data, and training budget fixed. They then deployed the policy on a dual-arm robot to perform tasks including charger grasping, dual-arm handover, and plug alignment and insertion, evaluating performance by varying the positions of the charger or the socket.

The paper reports that in the LIBERO-Goal benchmark, ScaleVLA achieved an 87.8% success rate using 10 demonstrations per task, outperforming the 85.6% success rate of a 2-billion-parameter backbone using 50 demonstrations. In LIBERO-PRO, the average success rate was 45.1%, compared to a maximum of 31.3% for smaller models. In real-world tests with reference layouts, the charger insertion task succeeded 9 out of 10 times, whereas the 2-billion-parameter backbone succeeded only 4 out of 10 times.

The key to these results lies in how the vision-language model translates images and instructions into information that the action module can use, while ensuring the 552-billion-parameter backbone can operate within the robot's control pipeline.

To understand ScaleVLA, we must first examine the output of the vision-language backbone and how the action head translates these features into robot instructions.

Integrating pre-trained representations into action generation

ScaleVLA divides visual understanding and action generation between two modules. The vision-language backbone processes camera feeds and task instructions, while the action head combines these representations with the robot's state to output a continuous action sequence. The former provides visual and linguistic information derived from large-scale pre-training, whereas the latter learns specific execution strategies from robot demonstrations.

The backbone is based on DeepSeek-V4.1-Flash-1017, a 552-billion-parameter mixture-of-experts model. Camera images are first converted into visual tokens by a vision encoder and then fed into the language backbone alongside the task instructions. The language backbone comprises 20 causal encoder layers and 20 decoder layers. ScaleVLA runs the causal encoder to extract the resulting visual and text features, halting computation before the decoder stage. The sequence and modality tags of each token are preserved, allowing the action head to distinguish which information originates from the images and which from the instructions.

This architecture alters the role of the large model within the control system. The paper notes that in related work such as GPT-6 Astra, general models can improve manipulation using physical feedback, but iterative reasoning and action verification increase decision-making time. Instead, ScaleVLA extracts features from the backbone and passes them to the action head for action generation, bypassing the language decoder entirely during this execution phase.

Vision-language model features cannot be directly translated into robotic arm commands. To bridge this gap, the team introduced a feature adapter between the backbone and the action head: it first normalizes and linearly projects the features, mapping each token from 5,120 dimensions down to 2,048 dimensions, and then passes them through a four-layer self-attention Transformer to enable further interaction between visual and linguistic features under action supervision. This adapter not only standardizes the input dimensions but also helps the model learn which visual cues are relevant to the current instruction and action.

The action head also incorporates the robot's current state. The paper employs separate encoders for the state and noisy actions, alongside a multilayer perceptron specifically designed for the robot's physical embodiment. Each step in the action sequence is converted into a token with positional embeddings, and the model also receives current flow time information. Consequently, action prediction simultaneously conditions on the visual input, task instructions, robot state, and the current generation step of the action sequence.

The diffusion Transformer (DiT) is responsible for generating actions. It consists of 32 alternating layers of self-attention and cross-attention: self-attention allows the robot state to cross-reference actions across different timesteps, while cross-attention reads information from the adapted image and text features. The action decoder converts the DiT output into the predicted velocity for the entire action sequence, enabling step-by-step updates in the subsequent generation process.

The team adopted the action head architecture from NVIDIA GR00T N1.7, but all modules on the action pathway started from random initialization and were then trained using robot demonstrations. The comparative experiments also did not load a pre-trained robot action expert. The vision-language backbone provides pre-trained representations, while the action head learns continuous control; this separation of roles forms the basic structure of ScaleVLA.

From demonstrations to action chunks

ScaleVLA's training does not update the 552B-parameter vision-language backbone. The team first uses the frozen backbone to extract image and instruction features from the demonstration data and caches these features; during training, only the feature adapters, state and action encoders, DiT, and action decoder on the action pathway are updated. This eliminates the need to recompute the entire set of vision-language features every time the action module parameters are updated.

The paper reports that training used 64 NVIDIA H200 GPUs distributed across 8 nodes, with a global batch size of 640. Training on the full LIBERO dataset used 5,000 steps of AdamW optimization with a learning rate of 1e-4, combined with cosine decay and a 5% warm-up; DexJoCo used 20,000 steps. Robot states and action values were normalized based on the 1st and 99th percentiles of the training data.

The action head learns to generate action chunks via flow matching. During training, the team starts with demonstration actions and random Gaussian noise, samples a flow time, and linearly mixes the noise and the demonstration action in proportion. What the model sees is sometimes close to random noise and sometimes close to the real demonstration; it needs to predict the direction of change from the noise to the demonstration action based on the vision-language features, robot state, and the current mixing ratio.

This training does not ask the model to output a sequence of robot commands verbatim, but rather to learn how an entire sequence of numerical actions gradually forms from noise. The loss function is computed over the valid action dimensions and timesteps, with both states and actions normalized using the percentiles of the training data. Because the vision-language backbone is frozen, the backbone features computed during training can be reused.

During inference, the action head starts from random noise and undergoes a four-step update in the predicted direction to obtain an action chunk. The four-step update reuses the same set of vision-language features, so the backbone runs only once per action chunk. ScaleVLA predicts the next 16 steps in LIBERO, executes the first 8, and then replans based on new observations; in DexJoCo, it predicts 30 steps and uses a timestamped instruction buffer to asynchronously generate the next action chunk while the controller executes the current one.

In RoboCasa latency tests, ScaleVLA's median action chunk latency was approximately 216 milliseconds, while Cosmos-Reason2-2B's was about 115 milliseconds. Although ScaleVLA's backbone contains 276 times more parameters than the latter, its action chunk latency did not scale proportionally with parameter count.

The times in the table represent the round-trip latency from the client sending an observation to receiving an action chunk. The action head's processing time remained approximately 70 milliseconds across different backbones. When scaling the smaller backbone to 32B parameters, the additional time stemmed primarily from vision-language backbone inference. ScaleVLA utilized four H200 GPUs, whereas the 2B backbone used only one. The number of input tokens per pass also differed, at approximately 568 and 212, respectively. These results show that the measured latency increased at a slower rate than the parameter count, but this cannot be directly interpreted as an efficiency comparison under identical hardware configurations.

Action chunk latency also does not equate to the robot's low-level control frequency. In LIBERO, the model predicts a 16-step action sequence but executes only the first eight steps before acquiring new observations to replan. The physical execution cycle further includes robot arm actuation, waiting for feedback, and inferring the next action. In a real-robot charger-plugging task, the paper measured a round-trip latency of 384 milliseconds per action chunk, which is a distinct metric from the total time required to complete the task.

Data Efficiency and Generalization

The team evaluated ScaleVLA's performance across four simulation benchmarks: LIBERO assesses tabletop manipulation guided by language instructions; LIBERO-PRO introduces variations in objects or environments relative to the original tasks; RoboCasa tests performance across diverse kitchen scenes; and DexJoCo comprises bimanual dexterous manipulation tasks. Together, these benchmarks span conditions ranging from standard tabletop manipulation to scene variations and bimanual coordination.

Under LIBERO's full demonstration data setting, ScaleVLA achieved an average success rate of 96.1% across its four task suites, outperforming Cosmos-Reason2-2B's 90.7%. However, performance across multiple models was already nearing saturation in the Spatial and Object tasks. The performance gap was more pronounced in the Goal suite, which requires understanding target instructions, and the Long suite, which involves chaining multiple steps. ScaleVLA achieved success rates of 96.8% and 90.2% in these two categories, respectively.

LIBERO-PRO evaluates policy robustness against environmental or object perturbations without retraining or fine-tuning after the initial training phase. Testing the LIBERO policies under these perturbed settings, ScaleVLA achieved an average success rate of 45.1%, compared to a maximum of 31.3% among the smaller models. However, results were not consistent across all perturbation types: under the environmental perturbations in the Object suite, ScaleVLA scored 31.2%, slightly below the 32.4% peak achieved by other backbones. While the overall average supports better transfer capabilities, performance on individual tasks remains sensitive to the specific type of perturbation.

RoboCasa evaluates policies in kitchen scenes entirely unseen during training, utilizing either 30 or 100 generated demonstrations per task. ScaleVLA achieved success rates of 30.7% and 54.9% at these two data volumes, respectively, while the best-performing smaller models reached 25.6% and 45.6%. This benchmark measures scene generalization within simulated kitchens and should not be directly equated to real-world deployment performance in actual kitchens.

DexJoCo comprises five bimanual tasks—unlocking a tablet, Tower of Hanoi, assembly, microwave, and photography—each using 100 human demonstrations. Under object randomization, ScaleVLA achieves an average success rate of 42.1%, while the best small model reaches 38.1%; under full randomization, these rates drop to 24.4% and 22.1%, respectively. Full randomization remains highly challenging, with all models showing very low success rates on the assembly task, where ScaleVLA achieves only 1.3%. Scaling up the backbone improves performance on some multi-task benchmarks but does not eliminate the inherent difficulty of the tasks themselves.

Sample efficiency tests are conducted on LIBERO-Goal and Long, using 5, 10, 25, or 50 demonstrations per task, with training steps fixed at 5,000. The table below retains only the two configurations from the Goal benchmark that best illustrate the differences in data volume:

With 10 demonstrations per task, ScaleVLA achieves a success rate of 87.8%, surpassing the 85.6% achieved by the 2B backbone with 50 demonstrations. With only 5 demonstrations, ScaleVLA reaches 77.4%, compared to 39.4% for the 2B backbone. As the number of demonstrations increases, the performance gap between the two backbones generally narrows. For the Long benchmark, which requires chaining more steps, adding more demonstrations continues to improve results.

The "10 demonstrations" figure has a specific definition: these trajectories are used for policy parameter updates, but the state and action normalization statistics for this experiment still use information from the full training set. Therefore, this number indicates the number of demonstrations used for policy updates, not that only these 10 data points were available during the preprocessing stage. Each policy was trained with a single random seed, and the demonstration subsets were drawn from the same set of nested sequences; the results require further verification across other random seeds and demonstration subsets.

The final insertion step is the hardest

The team deployed three policies on a bimanual robot to execute a charger insertion task. The right arm picks up a phone charger and hands it to the left arm, which then aligns the plug with the power strip socket, inserts it, and releases it. The robot observes the operation through a top-mounted camera and two wrist-mounted cameras. This task involves grasping, bimanual handover, spatial alignment, and contact manipulation simultaneously, serving to test the role of vision-language representations in real-world physical execution.

Real-robot training data comprises 171 full teleoperation demonstrations and 132 clips focusing on the insertion motion, totaling 303 episodes and approximately 520,000 frames, with an additional 26 episodes reserved for validation. Most full demonstrations start near the primary charger location, while a few include positional offsets. All three policies use the same action DiT, training data, and training budget. Among them, π0.5 is pre-trained on large-scale robot data, serving as a reference to evaluate whether robot pre-training can compensate for differences in vision-language backbone scale.

Each policy undergoes 10 reference layout tests and 15 out-of-distribution tests. The reference layouts start near the primary training location; the out-of-distribution setup involves moving or rotating the charger, as well as changing the position of the power strip. The success criterion is that the left gripper releases the charger only after the plug is inserted into the socket. This metric captures complete insertion outcomes, rather than counting it as a success if the robot arm merely moves the plug near the socket.

In successful trials, the time taken for grasping and handover was similar across policies, with the main differences emerging during the insertion phase. In the paper's demonstrations, the other two policies could move the plug near the socket but failed to complete the insertion, whereas ScaleVLA more frequently achieved post-contact alignment and advancement. The paper reports that all failures in the reference layout occurred during the insertion phase. In successful trials, ScaleVLA's median insertion time was 58.1 seconds, which is lower than that of the other two policies.

Per-action-chunk latency tells a different story: ScaleVLA requires 384 milliseconds, compared to 262 milliseconds for π0.5. However, the median overall completion time for successful trials is 114.4 seconds, shorter than the 132.0 seconds for π0.5. The time it takes a model to generate an action chunk is not equivalent to the time it takes the robot to complete the task. More accurate insertion motions reduce the need for subsequent adjustments, offsetting the longer inference latency.

Real-robot evidence currently focuses on a single charger insertion task, with each policy evaluated over 25 trials and trained using a single random seed; success is determined by human operators. Out-of-distribution tests primarily alter the spatial positions of the charger and power strip, and have not yet been extended to other objects, robots, or contact-rich tasks. Consequently, while these results demonstrate that ScaleVLA achieves better alignment and insertion performance in this specific dual-arm insertion task, they are insufficient to conclude that it can generalize to a wide variety of real-world manipulation tasks.

ScaleVLA demonstrates an approach to leveraging state-of-the-art vision-language representations for robotic policies: it freezes the vision-language backbone and uses an action module to read images, instructions, and robot state to generate continuous control. Simulation tests report sample efficiency and some generalization capabilities, while real-robot experiments show that the performance gains are concentrated in the post-contact alignment and insertion phases.

The paper also highlights clear constraints to this approach: the 552B backbone requires more GPUs, and real-robot validation currently covers only a single task. Current evidence supports further investigation into the role of large-scale pre-trained representations in robot learning, particularly for tasks with limited demonstrations, high scene variability, or stringent contact-control requirements.