README.md

Foundation Physics Graph Model — demo outputs

Perception outputs from DROID episodes (lab AUTOLab, task "Put brick in drawer shelf and close drawer"). Everything here is generated by the repo's pipelines; nothing is hand-edited.

⚠️ Read this before trusting the numbers

A time-base bug was found after these were rendered and is still being fixed. PointWorld's scene-flow clip frame t maps to video/trajectory frame 2 * (start + t), not start + t — the annotations run at half the video rate. Verified across all 42 clips of all 5 episodes with joint-angle error 0.00e+00.

Consequences for the files here:

• 01_robot_mesh_overlay/ — unaffected. Driven purely by trajectory.h5 indexed by video frame, which is genuinely 1:1.
• 02_object_velocity/ — affected. The depth seed came from an annotation frame paired with the wrong video frame. Treat the m/s figures as provisional.
• 03_object_pipeline/ — affected (same reason), and the clip shown happens to contain no robot–object contact, so state=free throughout.

---

01_robot_mesh_overlay

Franka Panda + Robotiq 2F-85 posed from recorded joint angles by URDF forward kinematics, rendered with pyrender/OSMesa and composited on the real RGB, plus a per-frame robot mask.

Uses NVlabs/PointWorld's franka_panda_robotiq_2f85.urdf. An earlier attempt with fairo/polymetis's panda_robotiq_85.urdf was wrong: it puts the flange at z=0.045 where the real panda_link8 is at 0.107. That 62 mm shortfall was measured, not guessed — FK vs DROID's own recorded end-effector gave a residual of exactly [0, 0, 0.062] m with zero variance over 128 frames, and 0.107 − 0.045 = 0.062.

URDF	FK residual vs recorded EE
fairo/polymetis	0.062 m (std 0)
NVlabs/PointWorld	1.6e-8 m (std 7e-9)

02_object_velocity

SAM 3.1 (text prompt derived from the episode's task instruction) → TAPNext++ point tracking → rigid-body PnP lifting → metric 3D velocity in the robot-base world frame.

Depth is needed at one seed frame only: a rigid body has 6 DOF, so the 2D tracks determine every later pose. This replaced a per-frame depth lookup that was both weaker and dependent on the dataset annotating the object at every instant — which it does not (the nearest annotated point to the brick is 40.07 px away, just outside the search radius).

03_object_pipeline

Five independently re-runnable stages, each leaving its artefacts on disk:

dir	stage
1_mask	SAM 3.1 mask → tight RGBA crop
2_mesh	crop → mesh (proxy hull here; TRELLIS.2 not yet installed)
3_align	mask × depth → world point cloud → scale + 6-DOF fit
4_act	robot FK → per-frame object pose (free / pushed / grasped)
5_render	robot + object + point cloud, recorded and free-camera views

Alignment quality on this clip: RMSE 3.25 mm, inlier fraction 1.00, scale 1.009.

scene_freecam.mp4 orbits the world independently of the recording viewpoint — the recorded camera cannot show whether the object sits at the right depth, so a second viewpoint is the only honest check of the 3D placement.

The HUD's push_gain=4.0x is labelled VISUALISATION ONLY, not physical: it amplifies pushed motion (5 cm of arm travel → 20 cm of object travel) so the effect is visible. It is not a simulation result.

The object mesh here is a convex hull fitted to the observed points, not a reconstruction — MeshSource records this so a proxy is never mistaken for real geometry.

04_verification_images

Alignment checks drawn as mask outlines over the raw frames. A filled overlay hides misalignment by covering the thing it should match, so these outline views are what the robot alignment was actually judged on.