Nvidia Uses SimFoundry to Bridge the Last Mile of Robot Real2Sim

In recent years, automated data synthesis tools such as DexMimicGen and simulation platforms like PolaRiS and Isaac Sim have provided scalable alternatives to address robot data and training challenges.

However, enabling developers to manually set up simulation environments through steps such as CAD modeling, providing physical constraints, and scene placement—while achieving Real2Sim fidelity in vision, geometry, and dynamics—remains a significant challenge.

On June 26, 2026, NVIDIA and Fei-Fei Li’s team at Stanford University jointly released the technical report "SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation" and introduced SimFoundry, an automated Real2Sim infrastructure.

The core problem SimFoundry addresses is the low-cost, automated conversion of a real-world physical environment into a simulation environment with high fidelity in vision, geometry, and physics, while simultaneously testing and training robot policies within that simulation.

To achieve this, SimFoundry integrates processes such as 3D reconstruction, object understanding, physical attribute generation, and simulation environment construction into a modular automated pipeline. This significantly reduces manual modeling and scene configuration efforts, giving Real2Sim its first potential for scalable application.

How SimFoundry Achieves Real2Sim

SimFoundry is a modular, automated system that starts from an input real-world RGB video (left image), automatically reconstructs an interactive, physics-enabled simulation scene (digital twin), and further generates diverse variants across three dimensions: objects, scenes, and tasks (digital cousins, right image).

Next, this article breaks down the SimFoundry pipeline (automated workflow) in the above diagram from architectural and training perspectives to explore how it achieves two major functions: 'scene generation' and 'policy training'.

• Extraction

By default, SimFoundry extracts frame 0 of the video as a representative frame. Using DepthAnything3 and FoundationStereo depth estimation models, it converts RGB images and stereo images into RGB-D depth maps.

Combined with camera intrinsics inferred by the DA3 model, the system further upscales the RGB-D data to reconstruct a 3D point cloud of the scene.

Then, SimFoundry uses SAM3 for image segmentation to extract the desktop plane and aligns the reconstruction results to the simulator's world coordinate system. It employs the general VLM model Gemini-Pro-3 to detect objects in the scene and iteratively segments foreground objects using SAM3. Finally, it extracts 2D masks for object segmentation along with corresponding RGB and depth pixels, which are used for subsequent mesh generation and alignment.

Each time object information is extracted, SimFoundry uses image and depth inpainting techniques to remove the object from the current RGB-D observation. This process is repeated until no foreground objects remain in the scene, thereby completing video information extraction.

• Generation (Asset Generation and Physical Restoration)

For each cropped image of an object obtained from 'Extraction', a general-purpose image editing VLM model, Gemini-Pro-3-Image-Preview, is used for upsampling (increasing image resolution), followed by a 2D-to-3D mesh model, Hunyuan2.1, to generate visual meshes.

Subsequently, the object's 2D mask, corresponding RGB-D data, and geometric point clouds are aligned within the reconstructed scene. The FoundationPose model is then used for additional fine-tuning to obtain the object's pose. For articulated objects identified by the VLM, such as 'cabinets' and 'drawers', the system performs mesh segmentation via P3-SAM and then uses a VLM (Actor-Critic framework) to generate URDF-format joint parameters.

Finally, CoACD is used to generate collision geometries, and physical properties (such as mass and friction coefficients) are assigned based on the object appearances observed during 'Extraction'. Once all objects are generated, aligned, and annotated, the scene is assembled in the physics engine PyBullet. Object penetration is resolved to achieve stable configurations, and the generated simulation scene is exported to downstream robot simulators (e.g., IsaacLab).

• Augmentation (Digital Cousin Enhancement)

As the core module through which SimFoundry breaks through the limitations of traditional digital twins, it expands the completed initial scene across three dimensions: object instances, scene layout, and task specifications, to support subsequent robot policy training and evaluation.

As shown in the image above, the top row displays RGB images from the real world, the middle row demonstrates how SimFoundry reconstructs objects, layouts, and scene structures from real-world inputs, and the bottom row shows how the reconstructed digital twins are extended into plausible "digital cousins." It is evident from the bottom batch of images that these extended, reasonable "digital cousins" can significantly increase robot data volume and save substantial "expert costs" originally required for building simulation scenes.

As shown in the image above, for object instances within the reconstructed scene, SimFoundry employs prompt engineering to leverage VLMs (Vision-Language Models) for standardizing object names (removing irrelevant information such as material, size, and transient states) and decomposing objects into functional components (such as handles, lids, and bases). For each functional component, the VLM proposes candidate variants along three predefined dimensions: geometry, topology, and visual appearance, while prohibiting unrealistic variants (e.g., red bananas).

Next, SimFoundry feeds the candidate variants into an image-editing VLM model to generate candidate images. The model is instructed to modify only the specified component while keeping other parts unchanged, outputting images with transparent backgrounds, and referencing the original scene image to maintain stylistic consistency. Finally, variant data passing dual validation for "real-world plausibility" and "scene consistency" is stored for downstream simulation.

For the scene layout in the reconstructed scene, SimFoundry first uses a VLM model to anchor one object based on the task, then samples other objects from a "predefined spatial list" (such as OnTop, InSide, etc.) to generate meaningful alternative layouts. It then filters and selects distractors from the BEHAVIOR dataset based on categories like quality and volume. When placing objects, it ensures no "collision clipping" occurs to guarantee physical stability.

As shown in the image above, for task instances in the reconstructed scene, SimFoundry first obtains the 2D image of the simulation scene and a list of interactive objects. It then incorporates robot hardware constraints (such as maximum gripper opening width, single-arm/dual-arm modes) and optional object-level constraints. Information including the scene graph, object list, and physical constraints is provided to the VLM acting as a "robot expert," enabling the VLM to parse explicit tasks with state changes from the initial state in YAML format and automatically compile them into standardized files.

Subsequent ablation experiments were conducted on object cousins, scene cousins, and task cousins separately to investigate the impact of each "cousin" on the generalization capability of the entire SimFoundry system. Experiments demonstrate that the three types of "cousins" respectively lead to improved success rates for robot policies when encountering unseen objects, unseen layouts, and unattempted tasks.

Notably, for background reconstruction and alignment in scenes, SimFoundry can not only fully reuse the RGB video generated during the "Extraction" stage to create the background but also combine it with manually shot "pure background" videos (without objects) using an interactive editor (GUI). This allows users to adjust the background via keyboard until a visually perfect match is achieved.

However, as shown in the table above, the automated pipeline comprehensively outperforms the manual pipeline across all seven metrics, including PSNR, SSIM, and NCC (Normalized Cross-Correlation). The reason lies in the fact that the automated pipeline does not require "estimating" alignment but rather "deriving" it. In contrast, the manual pipeline requires human operators to align six degrees of freedom poses by eye; even a rotational error of just one degree can cause significant pixel deviations in distant scenes.

How SimFoundry Facilitates Robot Policy Training

As NVIDIA's automated Real2Sim infrastructure, Real-to-Sim policy evaluation is undoubtedly the most core and convincing experiment. The experiment was set up with seven tasks and five cutting-edge policies (π0, π0.5, GR00T N1.6/N1.7, DreamZero), comparing real-world success rates with evaluation results in the SimFoundry simulation environment. As a comparison, the SOTA (currently best-in-class) baseline used is PolaRiS.

As shown above, the article's statistical summary indicates that SimFoundry achieves a Pearson correlation of 0.911 (measuring the linear relationship between task success rates measured in simulation and those measured in the real world; higher values indicate stronger correlation) and an MMRV of just 0.018 (measuring whether simulation incorrectly ranks policy performance; lower is better), which is more than 0.59 higher than PolaRiS.

As shown in the table below, by decomposing long-horizon tasks into sub-tasks, strategy evaluation further improves the correlation from 0.90 to 0.95.

Furthermore, SimFoundry's greatest innovation lies in its 'digital cousins.' Unlike traditional 'digital twins,' which generate only single scenes and cause strategies to easily overfit, 'digital cousins' alter object instances, spatial layouts, and task objectives at the semantic level while strictly maintaining task relevance. This provides controlled and reasonable data diversity for strategies, significantly enhancing the generalization ability of trained models.

The article set up YAM dual-arm and DROID single-arm platforms. The YAM dual-arm platform used a simple flow-matching strategy trained from scratch, while the DROID single-arm platform used fine-tuned π0 and π0.5 models.

As shown in the figure above, 'Real Cousins' refers to testing entirely new objects in the real world and serves as the most critical metric for validating generalization ability. When trained using only 'Twins' (digital twins), both Sim Cousins and Real Cousins scores are low, indicating that neither simulation nor the real world can generalize well when encountering new objects at this stage.

As the number of training 'digital cousins' increases, the Sim Cousins and Real Cousins scores for various tasks show significant improvement, proving that 'digital cousins' can enhance the model's generalization ability.

Finally, the article validates SimFoundry's zero-shot transfer capability, which means directly using policies trained in simulation without incorporating real-world data. In this regard, YAM achieved a 99% success rate on the 'Pot on Stove' task (carrying a pot with both hands), and DROID achieved a 100% success rate on the 'Stack Dishware' task (picking up plates, placing them down, picking up cups, and placing them down).

Furthermore, the appendix notes that if a human operator is allowed to align the grid with the point cloud for just 3 minutes per object using a keyboard, the F1 score (which measures the accuracy of 3D geometric reconstruction) spikes directly to between 0.93 and 0.99.

In fact, the industry is reluctant to abandon its own real-world data, so the article also validates training performance after incorporating some real data. Here, '-S' denotes pure simulation training, '-R' denotes pure real-data training, and '-co-train' denotes joint training. As shown in the figure, joint training nearly maximizes the success rate on real-robot tasks (-Real).

In summary, SimFoundry's experimental design is comprehensive, and its core metrics (evaluation correlation) are persuasive, covering the complete chain from reconstruction quality to downstream policy performance. It proves that the entire closed loop of Real2Sim → Data Generation → Policy Training → Real-World Deployment is feasible.

SimFoundry acts as both the 'Creator' of digital scenes and the 'Good Coach' for robot policies.

Why NVIDIA Needs SimFoundry

SimFoundry did not appear out of nowhere; it is a key piece in the puzzle of NVIDIA's Physical AI strategy.

SimFoundry empowers underlying infrastructure, activating the 'high-fidelity engine' of the Isaac ecosystem. As NVIDIA's core physical simulation platforms, Isaac Lab and Isaac Sim possess powerful physics engines and rendering capabilities, but they require tedious manual modeling. SimFoundry bridges this gap by using automated video reconstruction technology to transform messy, irregular real-world scenes into standard USD format assets.

SimFoundry also feeds upper-layer intelligent models. The 'digital cousins' it generates provide massive amounts of high-fidelity synthetic training data, helping GR00T (NVIDIA's, and indeed the industry's most cutting-edge, general-purpose robot foundation model) achieve generalization across broader environments. Meanwhile, the 'digital twin' environments reconstructed by SimFoundry offer an extremely high-fidelity evaluation benchmark.

Notably, SimFoundry can both link to data generation tools (MimicGen) and root itself in real data sources (DROID / EgoData).

MimicGen can leverage a small amount of raw demonstration data, combined with variations of SimFoundry's object cousins, to automatically generate hundreds or thousands of diverse trajectory data samples, potentially solving the data hunger problem in robot training once and for all.

At the same time, SimFoundry draws 'physical genes' from real videos collected on the DROID platform and EgoData, replicating information such as the geometry and materials of real objects into simulations. This ensures the underlying logic of the entire tech stack: that the source of data is real and physically interpretable.

In summary, SimFoundry serves as a key hub in NVIDIA's Physical AI. It transforms the real world into simulation assets, filling Isaac's modeling gaps downwards while providing data and benchmarks for GR00T upwards. By linking MimicGen and DROID, it connects the full-chain closed loop from data collection and generation to training and evaluation.

**Final Thoughts

SimFoundry's greatest contribution is upgrading Real2Sim from a time-consuming and labor-intensive manual task to an automated, scalable pipeline, and fully unleashing the diversity potential of simulation data through the 'digital cousin' mechanism.

Compared to traditional Sim2Real, which requires extensive expert involvement in scenario design and physical parameter tuning, leading to increased complexity and significant labor costs.

Led by Real2Sim, SimFoundry reasonably leverages VLM models (such as Gemini) to deeply participate in end-to-end automated scene reconstruction tasks. Its cost structure is primarily computing power costs, with the cost of reconstructing a scene being predictable, which also aligns with Nvidia's business logic of selling computing power.

The latest news is that NVIDIA open-sourced a continuous learning framework for robots, ASPIRE, in early July. As the robot version of a Coding Agent, it can work in conjunction with SimFoundry to run tasks in massive simulation reconstruction scenarios, fix bugs from failures, and accumulate a Skill Library. Together, they form a closed loop for Embodied AI, spanning from simulation supply to skill accumulation.

Therefore, for future robot training, SimFoundry points the way: the real world provides scene templates, foundation models are responsible for understanding, simulators provide a sandbox for trial and error, and robot agents consolidate skills. This paradigm may accelerate the evolution of embodied AI towards generality.