Lattice 4D Real-World Dataset

The Lattice 4D Dataset is an early, publicly available collection of real-world human-object interactions, showcasing how synchronized multiview captures can be transformed into temporally aligned 3D/4D reconstructions and rich, structured representations of the physical world. The dataset combines multiview RGB-D recordings, reconstructed geometry, human motion, derived egocentric views, structured world states, and LeRobot-compatible data to support robotics and physical AI research.

Why 4D Data?

Most AI models today learn from 2D images and video. The goal is to understand the physical world, but that requires more than recognizing what appears in a frame.

Depth, occlusion, object motion, and spatial relationships are fundamental to physical interactions, but are lost or difficult to recover when a space is projected to a single 2D perspective.

We believe 4D data can provide richer supervision for the next generation of world models and physical AI, allowing models to learn how the world moves and changes through space and time.

At Lattice, we're building a way to capture this information directly from real-world activity and make it available at scale.

What's in the Dataset?

The dataset captures people performing physical tasks, including ball handling and shirt folding.

Each recording provides multiple aligned representations:

• Multiview RGB-D: Synchronized video, depth, and camera calibration.
• 4D Reconstruction: Dynamic 3D scene geometry across time.
• Human and Object Motion: 3D body and hand tracking, object states, and action annotations.
• Derived Egocentric Views: Virtual first-person perspectives generated from multiview captures, without head-mounted cameras.
• LeRobot Integration: LeRobot-compatible episodes for robotics learning workflows.

All representations are time-aligned so the same physical interaction can be seen through multiple camera views, reconstructed 3D geometry, and derived first-person perspectives.

What's Next?

Our goal is to make real-world 4D data available at scale.

We're preparing deployments in textile manufacturing environments to capture complex human-object interactions involving deformable materials, from folding and sewing to ironing and packaging.

We hope to enable new research in world models, robot learning, physical prediction, and simulation.

We're actively looking to collaborate with researchers and physical AI teams to help shape future datasets, representations, and collection efforts.

Explore the Dataset | Interactive 4D Scene

— The Lattice Team