ItStone Zhihang and Six Institutions Release OmniVTA Vision-Tactile World Model

Recently, Tashi Zhixing, in collaboration with six institutions—National University of Singapore, Fudan University, Chinese Academy of Sciences Institute of Automation, Tsinghua University, Zhongguancun College, and Beihang University—launched the OmniVTA vision-tactile manipulation framework and the OmniViTac large-scale vision-tactile dataset, while publishing related papers. This marks a key step toward enabling robots to move from passive perception to active tactile prediction and closed-loop precise control, facilitating dexterous manipulation.

In the past one or two years, the role of tactile sensing in the practical application of robotics has grown significantly. However, many traditional approaches have fallen into a dilemma: despite the widespread use of tactile sensors in robotics, the paradox of "perceiving more but performing worse" remains difficult to overcome. Robots can clearly "feel" objects, yet still "fail to utilize" this information effectively. Why does providing additional tactile perception sometimes lead to poorer performance?

The root cause lies in the long-standing lack of modeling for contact dynamics and effective utilization of tactile information in robotic manipulation. Current solutions often simply concatenate visual and tactile features before feeding them into policy networks. While this may seem reasonable, it overlooks the core characteristics of tactile signals. Unlike vision, which offers global semantics and continuous observation, tactile signals are highly localized and driven by contact events, making it difficult to provide global perception or support long-horizon planning.

More critically, contact is inherently a dynamic process that evolves over time. Operations such as wiping, peeling, plugging, and tightening involve continuously changing contact states throughout the task.

However, existing methods typically rely only on current or a few historical frames of tactile observations, lacking explicit modeling of "how contact evolves over time." As a result, tactile data is often used merely for simple contact detection or visual occlusion compensation, rather than actively participating in predicting and deciding upon contact processes. Furthermore, the absence of high-frequency tactile data prevents models from learning real-world contact patterns, severely compromising operational stability and generalization.

Inspiration from Human Behavior: The "Prediction + Feedback" Synergistic Mechanism

Neuroscience research indicates that humans rely on a synergistic "prediction + feedback" mechanism during contact operations: the brain uses forward models to anticipate sensory changes resulting from actions, while simultaneously employing real-time sensory feedback for rapid correction to counteract errors and disturbances. This "predict first, then correct" mechanism enables humans to perform stable and dexterous contact operations even in uncertain environments.

Inspired by human neural mechanisms, Tanshi Zhihang has collaborated with six major institutions to advance both the data foundation and technical framework. At the data level, the team released OmniViTac, a large-scale vision-tactile dataset. Currently containing over 20,000 operation trajectories, it covers nearly 100 task categories and more than 100 types of objects. Contact modes are systematically classified into six categories: wiping, peeling, cutting, grasping, assembly, and in-hand adjustment. The data collection process strictly ensures high-precision synchronization among vision, tactile signals, and actions, while preserving original sensor frequencies.

OmniViTac Dataset

Leveraging the multimodal data support of OmniViTac, Tanshi proposed OmniVTA—a vision-tactile manipulation framework centered on a world model. The core idea shifts from "passive tactile perception" to "active tactile prediction": robots not only perceive current tactile signals but also model and predict the evolution of future tactile states, thereby guiding action planning and closed-loop adjustments.

In system design, OmniVTA adopts a slow-fast hierarchical control structure: the slow system predicts future tactile representations based on the vision-tactile world model and generates action sequences, while the fast system performs reflexive high-frequency control using predicted tactile data and real-time tactile feedback. This achieves stable, robust, and fine-grained contact operations.

OmniVTA System Diagram

Building on this, OmniVTA constructs a unified closed-loop control system through four key modules, enabling robots to possess the ability to "predict tactile input—understand contact—correct actions," thereby reshaping their manipulation performance in complex contact scenarios:

TactileVAE: Through spatiotemporal joint encoding and implicit function decoding, this module compresses high-frequency, dense tactile 3D deformations into low-dimensional continuous latent variable representations. It preserves fine-grained contact structures in space (such as shear and normal deformations) while modeling temporal tactile dynamics, thereby effectively characterizing the evolutionary trajectory of the contact process. By significantly reducing data dimensionality and computational overhead, it provides generalized tactile representations for subsequent prediction and control, enabling robots to efficiently understand current contact states and rapidly respond to environmental changes.

TactileVAE network architecture diagram

Visuo-Tactile World Model (Prediction Module): Based on a dual-stream diffusion generative architecture, it jointly models the temporal evolution of vision and touch under shared conditional constraints. The visual branch provides global semantic and geometric priors, while the tactile branch focuses on local contact dynamics; both align synergistically in the latent space to achieve high-quality prediction of future tactile signals. By explicitly modeling the dynamic process from "pre-contact" to "during contact" to "post-evolution," this module enables robots to anticipate contact trends (such as impending contact, changes in contact intensity, or slipping risks), providing forward-looking information support for motion planning.

Adaptive Fusion Strategy (Decision Module): Introduces a Latent Tactile Differential (LTD) encoder to explicitly model the difference between current and predicted tactile data, extracting key signals of changing contact dynamics. On this basis, a gating mechanism dynamically weights visual and tactile modalities, allowing the strategy to adaptively adjust perceptual dependencies based on the contact phase: prioritizing global visual information during non-contact or far-contact phases, and enhancing dominant tactile influence during contact occurrence and evolution. This module effectively avoids information conflicts caused by simple feature concatenation, making action decisions more precise and context-adaptive.

OmniVTA Slow Policy: Visuo-Tactile World Model + Adaptive Fusion Strategy

Reflective Tactile Controller (Execution Module): Outputs single-step corrective actions at a high frequency of 60 Hz based on predicted tactile data and real-time tactile feedback, performing continuous closed-loop compensation for action sequences generated by the slow system. By modeling tactile errors (the difference between prediction and observation), this controller achieves rapid response, instantly correcting execution trajectories in cases of contact disturbances, object shifts, or friction changes, thereby significantly improving operational stability and precision. Its introduction endows the system with human-like "tactile reflex" capabilities, effectively compensating for the lag associated with low-frequency planning.

Reflective Controller Based on Tactile Features

Experimental data show that the OmniVTA visuo-tactile manipulation framework achieves optimal performance across different objects and contact modes. It demonstrates superior robustness and generalization compared to traditional methods in scenarios involving position changes, tool variations, and external disturbances.

Real-time Disturbance-Recovery Contact During Manipulation

Crucially, the model has learned transferable contact dynamics. It can adaptively adjust the weights of vision and touch based on predicted contact states, maintaining stable performance across different objects and tools. This indicates that robots are moving from "executing actions" to "understanding physical contact," gradually acquiring predictive and feedback synergy capabilities akin to humans.

Gating mechanism effect: Changes in tactile and visual weights throughout the manipulation process

As can be seen, OmniVTA centers on a "world model," using prediction as a guide and feedback as a safeguard, ultimately enabling robots to handle contact-rich tasks essential to industrial production and daily life, such as precision assembly, home cleaning, and food preparation.