0.04B Unified for Heterogeneous Cameras: Discussing the Trade-offs of X-Lens with Sui Wei from Digua Robotics

Fisheye cameras offer a wide field of view but suffer from distorted, curved edges, while pinhole cameras provide straight lines but have a narrow field of view.

A robot often carries both types of cameras simultaneously to perceive environmental depth. However, for depth estimation models like DAC and MapAnything, there are almost no solutions that can effectively adapt to both camera types and run on edge devices.

In response,地瓜机器人 (Digua Robotics), in collaboration with the University of Tokyo and Soochow University, released the technical report 'X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras' on July 15 and introduced X-Lens, a lightweight depth model for edge inference. With only 0.04B parameters, X-Lens takes heterogeneous camera data as input on the edge and directly outputs depth with physical scale.

The paper discusses X-Lens's performance: running at 22 FPS for six views and 41 FPS for single views on an H100 GPU. Under heterogeneous camera settings, it achieved an absolute relative error of 0.102, which is 41.6% lower than the AbsRel error of the previous MapAnything (1.23B).

At this point, I believe you share similar curiosity and questions such as:

• What is the current fusion scheme for fisheye and pinhole camera images, and what are its pain points? In other words, why is X-Lens needed?

• How does it achieve 'cross-camera consistency'? The paper mentions light space rather than pixel space. Since pixel space refers to the pixels of an image, how should we understand light space here? This relates to the core methodology of X-Lens.

• What is the actual performance on Digua Robotics' S600, and can it help iterate the company's data closed-loop? This concerns the collaborative adaptation between the X-Lens model and Digua's own products.

For this, we interviewed Sui Wei, one of the authors of this paper and also the Vice President of Algorithm at Digua Robotics, to discuss this paper together.

Digua Robotics constructed a 266,000-frame OmniScene six-view synthetic dataset, placing the 0.04B X-Lens in fisheye, pinhole, and heterogeneous environments, conducting comprehensive comparisons against more than ten baselines. Based on Sui Wei's interview and the technical report, we draw the following brief conclusions to the aforementioned questions:

• Existing solutions either lose field of view during de-distortion and cannot run in real-time, or run fisheye and pinhole models separately leading to misaligned scales, or rely on general large models with parameters too large to run on edge devices. No single model can simultaneously preserve field of view, scale, and real-time performance.

• Light fields are tagged based on "from which direction in 3D space the received light originates." This way, although fisheye and pinhole images have different distortions, the light ray directions describing the same wall are identical, naturally unifying the language of both camera types.

• The six-view heterogeneous setup runs at 11 FPS on Digua's own single-core S600, which is sufficient. Meanwhile, X-Lens does not rely on traditional data closed-loop systems, but its reasonable three-stage training structure means that for each scene adaptation, only stages 2 and 3 of fine-tuning are needed without altering the pinhole camera foundation. The iteration cost is a fraction of full-scale fine-tuning for large models.

To summarize, X-Lens normalizes the differences between heterogeneous cameras at the input level using light fields, directly yielding specific physical scales (such as depth distances). Combined with its lightweight nature, if its geometric features are integrated as a side-mounted encoder into a VLA, the model no longer needs to indirectly guess distances from limited action data, bringing it closer to true deployment requirements.

From H100 to S600, Is a 0.04B Small Model Sufficient?

Sui Wei expressed satisfaction and confidence in X-Lens's capabilities, stating, "The performance of the 0.04B parameter X-Lens is not inferior to large models with over 1B parameters."

The article begins with the most intuitive radar chart. The figure below compares X-Lens with models such as MapAnything, DA3, and VGGT in terms of performance on pure fisheye (KITTI-360), pure pinhole (ETH3D), and cross-camera scenarios (OmniScene-Full: six views consisting of 4 fisheye + 2 pinhole).

As can be seen, other models have their respective strengths and weaknesses. Some specialize in pinhole but perform poorly on fisheye; others can handle fisheye but exhibit significant scale drift in heterogeneous scenes. X-Lens is the only model that approaches the outer ring across all three dimensions.

The paper also provides quantitative metrics: X-Lens achieves a Scale AbsRel score of 0.12 on ETH3D and 0.067 on OmniOcc, whereas MapAnything scores 0.205 on OmniOcc—a threefold difference. Scale AbsRel refers to predicting global physical scale; it requires not just indicating which objects are deeper, but specifying exactly how deep they are. For robots requiring grasping or obstacle avoidance, this scale information is a hard requirement.

Qualitatively, comparing generated images shows that within the OmniScene dataset, each model's fisheye and pinhole views from the same frame in two scenarios reveal distinct differences. As monocular methods, DAC and UniDAC perform independent inference for each view, causing depth color mismatches for the same wall across different perspectives. Although MapAnything performs multi-view fusion, its handling of heterogeneous data is weak, leading to noticeable deviations in distant structures. In contrast, X-Lens maintains consistent depth shading between near-field fisheye and far-field pinhole views.

As shown in the figure above, when researchers back-project depth predictions into 3D point clouds, the differences become even more apparent. Point clouds from DAC and UniDAC show visible misalignment and layering at camera junctions. In contrast, X-Lens' point cloud aligns highly with Ground Truth (GT) values at marked distances such as 11.02m, 17.34m, and 31.79m. This indicates that X-Lens genuinely perceives environmental depth through vision, supporting subsequent robotic behaviors like navigation.

Of course, the paper is one thing; the above performance figures were achieved on an H100. As is well known, with the release of the Sunrise S600 chip last November, DiGu Robotics' true focus and competitive advantage have shifted to the edge.

Sui Wei revealed that after completing the paper, the team immediately performed a quantized deployment on the S600. The six-view heterogeneous setup (4 fisheye + 2 pinhole cameras) can achieve 11 FPS on a single core of the S600.

In fact, for indoor service robots or warehouse AGVs, this refresh rate enters the "usable" range when combined with SLAM and local planning. Moreover, since these numbers were achieved on a single core, there may be room for further optimization through multi-core parallelism.

Additionally, regarding implementation, Sui Wei noted that when deployed on VODPP (DiGu Robotics' multi-view perception stack), X-Lens demonstrates advantages over VGGT in two aspects.

The first is data usability: X-Lens directly outputs depth with physical units and fused point clouds in a unified coordinate system, whereas VGGT outputs relative depth. Information regarding true physical scale does impact robotic arm grasping tasks in the real world. The second aspect is resource consumption: at 0.04B versus 1.26B parameters, the smaller model significantly improves fluidity during robotic arm operations.

Of course, Sui Wei also addressed some shortcomings and improvement directions mentioned in the paper. For instance, the paper notes that the model strictly relies on input camera parameters, its generalization ability is constrained by the distribution of internal parameters in the training data, and extreme lens types (such as catadioptric cameras) exhibit a sim-to-real gap.

This indicates that X-Lens's training still depends on camera intrinsic parameters within the data distribution, meaning adaptation to different cameras inevitably leads to errors.

To address this, X-Lens employs an engineering solution: it synchronously outputs a confidence map each time, highlighting areas where the model considers itself unreliable. This allows downstream modules to know which pixels require cautious processing.

Light Space as a "Unified Language"

After discussing performance and engineering implementation, let's step back one stage—why is a dedicated model needed for heterogeneous cameras?

Sui Wei categorizes existing fusion solutions for fisheye and pinhole cameras into three types, each sacrificing at least one aspect among 'accuracy, field of view, and uniformity'.

The first type directly undistorts the image and treats it as a pinhole camera. The fisheye frame is reprojected to be 'straightened' and applied to an existing pinhole depth model. The problem is that the 180° field of view of the fisheye lens is cropped in this process, causing severe stretching at the edges. This discards the most valuable feature of fisheye lenses: their wide viewing angle. Moreover, each undistortion takes several seconds, making real-time processing impossible on this path.

The second type handles them separately and stitches them together later. One model for fisheye and another for pinhole output depths independently, which are then combined using extrinsic parameters. However, the two monocular models have inconsistent understandings of 'how long one meter is', resulting in misaligned walls and broken floors at the camera junctions in the stitched point cloud. This approach doubles the model count and maintenance costs.

The third type relies on general geometric foundation models. Models like DA3, MapAnything, and VGGT can handle various images, but their training data and positional encodings are primarily designed around pinhole optics. The strong distortion of fisheye lenses either causes a sharp drop in accuracy or requires prior undistortion. Furthermore, these models often have billions of parameters; 'inference for a single frame on a robot takes nearly one second, leaving no time for real-time control.'

X-Lens's solution is to change the space since fisheye and pinhole 'languages' are incompatible in pixel space.

To add a note: traditional depth models use the pinhole intrinsic matrix K (a 3×3 matrix) to describe the correspondence between image coordinates and 3D rays. This setting fails directly under fisheye lenses, where projection is controlled by a set of non-linear distortion coefficients that cannot be represented within a 3×3 matrix.

This leads to a key concept called ray space.

Sui Wei gave a vivid description: "The same wall appears straight in the pinhole camera view, but curved at the edges of the fisheye image. However, in light field space, they are inherently the same wall."

The specific approach of X-Lens is to not use the internal parameter K matrix, but instead adopt a general reverse projection lookup table G. G itself includes focal length, principal point, and distortion coefficients, meaning that distortion is isolated within the input-side G function. Each pixel is no longer expressed using (u, v) coordinates, but rather translated into "which direction in three-dimensional space this ray points to when it originates from the camera's optical center," i.e., a unit direction vector.

With a unified "language," the subsequent network backbone must learn cross-camera communication within this language. The realization of cross-camera geometric consistency primarily relies on two key architectural designs: multi-view calibration tokens and Jacobian bias layers. These two design choices allow the model to function effectively without requiring massive numbers of parameters.

0.04B is Structural Subtraction, Not Compression

The most notable figure for X-Lens is 0.04B parameters. VGGT has 1.26B, MapAnything has 1.23B, and DA3-Giant has 1.36B; X-Lens is 96% to 97% smaller than them.

Sui Wei was very clear about his reasoning, "Choosing small models is driven by deployment scenarios. Scaling up parameters works for language models, but physical AGI requires edge-side deployment and real-time feedback. If deep models are all pushing towards several billion parameters, how can we achieve real-time feedback in the physical world?"

The paper also provides data to support this. As shown in the figure, on the heterogeneous OmniScene-Full dataset, X-Lens with 0.04B parameters outperformed MapAnything's 1.23B model: AbsRel (predicted depth error) was 0.102 vs. 0.175, Scale AbsRel (aligned depth error) was 0.118 vs. 0.370, and RMSE (absolute distance error) was 1.60m vs. 2.18m. Crucially, its speed of 22 FPS far exceeded MapAnything's 5 FPS.

Additionally, as shown in the table above, on RoboTwin2's Scan-Object task, VO-DPP replaced DINOv3-Base with X-Lens's geometric features as the visual encoder, raising the task success rate from 50% to 72%. Under the same conditions, VGGT achieved 67%, validating X-Lens's practical utility and compatibility with VO-DPP.

Naturally, one might ask: Is X-Lens's 0.04B parameter count a structural necessity or a clever way to compress parameters?

Sui Wei's explanation points to the former: "Large models use massive parameters to memorize patterns across various cameras and scenes, relying on data diversity to mask camera differences. X-Lens normalizes camera differences at the input stage using light-ray representations, so the network only needs to learn the mapping from light rays to geometry."

The image above shows X-Lens's core architecture based on an Heterogeneous Projection Transformer, featuring two key designs.

One is multi-view calibration tokens. This approach is extremely lightweight: it injects a small sequence of learnable calibration tokens into each layer of the Transformer, specifically helping fisheye views 'self-correct.' These tokens are active only for the fisheye side, completely ignored by the pinhole side; they are cleared after use in each layer, and a new set is used in the next layer.

You can roughly understand it as the backbone continuously learning the universal skill of mapping "light to geometry," while the calibration token is responsible for converting the distorted fisheye image into a format the backbone can comprehend. Moreover, the multi-view calibration token does not contain parameters specific to a particular model; therefore, switching focal lengths or using a different fisheye lens of the same type does not require retraining.

Another key design element is the Jacobian distortion bias. The calibration token handles internal view distortion correction, while the Jacobian bias manages the "translation" across views.

To establish geometric correspondence between fisheye and pinhole cameras, X-Lens extracts a nine-dimensional relative descriptor from the inverse projection's Jacobian matrix. This descriptor encodes four types of geometric relationships: light ray angles, logarithmic differences in local scale, light displacement, and Jacobian correlation.

This descriptor is then fed into a small MLP, whose output serves as an additive bias injected into cross-view attention. In other words, X-Lens adds a geometric hint to the cross-view attention mechanism. When computing attention between any two pixels, the system considers not only whether they "look alike" but also how close their corresponding 3D light ray directions are and how much each has been stretched by the fisheye lens.

As Sui Wei stated, "Predictions from all views are ultimately back-projected into the same world coordinate system and constrained by a single global scale head. Cross-camera geometric consistency is not achieved through post-hoc stitching but is enforced internally within the network."

The diagram above intuitively validates this design and illustrates the distribution of Jacobian bias correction magnitudes: significant warm-colored signals appear at the edges of fisheye images (high-distortion regions), while corrections in the central areas and on pinhole images are overall cool-toned and spatially uniform. This distribution aligns with physical facts; fisheye lens distortion is most severe at the boundaries, requiring the greatest geometric correction, whereas pinhole projection is approximately linear and does not demand drastic spatial variation corrections.

Therefore, when asked about changing cameras, Sui Wei replied that "not a single byte of network weights needs to be changed; you simply switch the calibration during inference." This is because if the only edge case arises—a completely new distortion family outside the training distribution, such as extreme catadioptric cameras—only the backbone needs to be frozen, with lightweight fine-tuning applied solely to that set of calibration tokens.

OmniScene: The Data Philosophy Behind 266,000 Frames

Heterogeneous camera depth models have long suffered from a lack of data. Without dense ground-truth data for synchronized multi-view fisheye and pinhole cameras, models have no basis for learning.

OmniScene was created to fill this gap.

The OmniScene dataset comprises 103 scenes, 564 motion sequences, and approximately 266,000 frames of synchronized six-view data, equivalent to over 1.7 million independent images. Each frame consists of four fisheye lenses (Kannala-Brandt model, 180° FOV) and two pinhole cameras at a resolution of 504×798. It includes per-pixel dense depth ground truth, validity masks, and sky markers. Additionally, 12 scenes are excluded from training to serve as benchmarks for model capability testing.

Specifically, the data sources for OmniScene are divided into two parts. The indoor scene portion utilizes assets from Kujiale, but Sui Wei emphasized that this is only a minor source; most scenes come from professional rendering pipelines such as Unreal Engine. The core consideration for choosing traditional rendering is the physical precision of the ground truth.

Some of the six-view scenes include apartments, office buildings, shopping malls, and industrial warehouses, as well as temples and tropical rainforests. Sui Wei noted that considering generalization capabilities, the training data for X-Lens covers most scene categories found in both everyday life and more exotic environments.

As shown in the image above, X-Lens can indeed operate directly in unseen real-world scenarios such as OmniOcc and fisheye scenes, confirming its generalization capabilities.

Of course, with the growing importance of simulation and simulation data assets, Sui Wei also shared his views on digital assets generated by AIGC:

This will be a major source of data for robots in the future. Digua is currently conducting related research; TabletopGen presented at ECCV this year focuses on digital asset generation.

He further predicted that the maturation of AI-generated asset technology will further reduce the marginal cost of adding new scenes to the OmniScene pipeline. In the past, the marginal cost of adding new camera models, new scene types, and new sensor modalities was low. Continuously evolving OmniScene as infrastructure on the data side will be a clear direction.

Conclusion

Objectively speaking, another direction worth noting is temporal processing. Currently, X-Lens performs frame-by-frame forward inference without inter-frame memory, making it unable to handle fast-moving objects. Sui Wei also previewed the team's next steps: "If downstream applications require the speed and trajectory of dynamic objects, which exceeds the task boundary of single-frame depth models, this should be handled by a temporal module (SLAM/tracking) connected after X-Lens."

An interesting detail from the interview was Sui Wei's reaction when asked about the 'moat' after open-sourcing.

His exact words were that what is open-sourced are the model architecture and one set of weights, while three harder-to-replicate things remain within the system.

First, the data production pipeline. OmniScene releases 266,000 frames of finished product, but the 'data manufacturing assembly line'—including virtual camera rig design, trajectory sampling, and consistency verification—is continuously iterated upon within the team. Datasets may become outdated, but the production pipeline does not.

Second, hardware-software co-design. X-Lens features quantization adaptation and compilation optimization for BPU platforms such as Digua S600, alongside deep integration with the perception stack VODPP. These engineering synergies are not documented in papers, yet they naturally form a moat for Digua's own hardware and software infrastructure.

Third, the first-mover advantage in iterative optimization. Sui Wei noted that while X-Lens is a very recent work, it may not be so by next year. However, since their team produced this work first, they possess deeper insights into how to optimize it and how to couple it as a backbone with downstream tasks.

These points align precisely with the core ethos of the X-Lens research. Rather than aiming for top rankings on specific benchmarks, the approach leans toward trade-offs inherent in engineering thinking.

It prioritizes native unified representation for heterogeneous cameras and parameter scales deployable at the edge, while sacrificing panoramic image conversion, auxiliary reconstruction objectives, and ultra-large parameter scales. In an era where geometric foundation models are collectively 'going big,' this choice to 'go small' constitutes the primary contribution of X-Lens.