Kosmo and Ropedia Partner to Align 3D Scenes With Human Manipulation Recordings

By Jiang Si Qi

On November 13, 3D spatial data engine Kosmo and physical AI data company Ropedia announced a strategic partnership to combine 3D scans of real-world scenes with first-person task recordings, providing physical AI models with aligned spatial structures and human manipulation data.

On November 13, 3D spatial data engine Kosmo and physical AI data company Ropedia announced a strategic partnership. The two companies will combine 3D scans of real-world environments with egocentric task recordings to provide physical AI models with data that aligns spatial structures with human manipulation processes.

To execute manipulations in real-world environments, embodied models must recognize the spatial relationships between rooms and objects, while also understanding how humans move, interact with objects, and alter their states. 3D scanning reconstructs the environment, and egocentric recordings preserve the task execution process. However, if these two data types lack spatial and temporal alignment, the model struggles to associate actions with environmental changes.

The core focus of this partnership is to integrate spatial data and human operational experience into a unified data collection and processing workflow.

Announced on November 12, the Ropedia Xperience 2 project includes 1,589 published metric-scale 3D scanned environments. The broader collection comprises 4,230 egocentric task recordings covering 74 tasks, totaling 131 hours of data.

The project employs a two-stage data collection process: first, handheld scanning devices record the physical space, and then operators wear Ropedia’s HOMIE Gen 2 head-mounted device to record task executions within those same environments. The processing pipeline estimates camera trajectories, as well as human body and hand movements, adds hierarchical language annotations to the tasks, and registers the recordings to the scene coordinates.

The same collected data can be applied across various downstream tasks. Xperience 2 demonstrates applications such as converting scanned environments into simulation settings, retargeting human actions into robot trajectories, teleoperation in virtual environments, and generating long-horizon task trajectories. The project also includes evaluations for depth and geometry, 3D object detection, and spatial intelligence to test multiple models.

On October 14, Kosmo launched the Kosmo One spatial camera and the Kosmo World cloud platform. Kosmo One captures the geometric and visual information of a scene, while Kosmo World processes this data into 3D assets such as 3DGS, meshes, and depth maps, and supports scene understanding.

Ropedia is responsible for recording human task execution within these scenes. The HOMIE Gen 2 captures egocentric video and sensor data, which Ropedia’s processing system then uses to perform temporal synchronization, spatial registration, motion estimation, task segmentation, and semantic annotation. Through this division of labor, the two companies provide scene data and operational experience, respectively, integrating them into a unified, jointly usable dataset.

The two companies plan to establish data collection protocols tailored to specific model and application requirements before launching targeted collection campaigns. Outcomes from subsequent training runs will also be used to refine future data collection strategies. The collaboration will build upon this workflow to ensure newly acquired data aligns more closely with the models' actual needs.