twanghcmut/ego-real2sim/basic_pick_place_clean/code - Storage Bucket

ego-real2sim

Fit an articulated hand and a rigid object to egocentric video with HaWoR, SAM3.1, SAM3D and MuJoCo. Scene inputs are the recorded camera, a flat table and hand trajectories.

Setup

uv venv --python 3.10 .venv-sam3d
uv pip install --python .venv-sam3d/bin/python torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
uv pip install --python .venv-sam3d/bin/python -e '.[object_reconstruction,optimization]'
git clone https://github.com/facebookresearch/sam-3d-objects.git external/sam-3d-objects
git -C external/sam-3d-objects checkout f91db411c50efee93d8db7aeb323885650f6f722
CUDA_VISIBLE_DEVICES="" MAX_JOBS=8 uv pip install --python .venv-sam3d/bin/python --no-build-isolation -r requirements/sam3d.txt
uv venv --python 3.12 .venv-sam3
uv pip install --python .venv-sam3/bin/python torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv-sam3/bin/python -e '.[segmentation,storage]'

SAM3D dependencies target Python 3.10, torch 2.5.1 and CUDA 12.1. Supply its checkpoint directory containing pipeline.yaml and weights. SAM3.1 uses a separate environment and the sam3.1_multiplex.pt checkpoint. HaWoR uses .venv-hawor; save its tracked keypoints with examples/track_hawor.py.

Prepare scene

MUJOCO_GL=egl .venv-sam3d/bin/python examples/prepare_scene.py outputs/basic_pick_place_clean \
  --tracks outputs/track_hawor/20261014_080552/hawor_tracks.npz

Preparation uses the recorded EgoDex camera and saved HaWoR tracks. Supply --table-height in meters to set the table explicitly; otherwise the contact interval and object height prior determine its height. Use --help to change these inputs.

Scene preparation writes:

• real_video.mp4: RGB frames with one full image per frame.
• cameras.npz: intrinsics, world camera poses, source frame indices, timestamps and image size.
• sim_trajectory.npz: hand qpos, camera poses, source frame indices and timestamps.
• scene.xml: articulated hands and a flat table.
• scene.json: table height and input provenance.

Each video frame corresponds to the same row in the saved camera and hand trajectories.

SAM3D receives the RGB image, the SAM3.1 object mask and an analytic pointmap. Camera rays intersect the calibrated horizontal table; inside the object mask, the intersection plane is raised by the object height prior, 0.045 m by default. This supplies a metric pose and scale initialization. The scaffold depends on those table and height assumptions; it does not measure the object's true depth. Reconstruction sets depth_model=None, so it requires no separate MoGe model checkpoint or depth inference.

Run inverse simulation

MUJOCO_GL=egl .venv-sam3d/bin/python examples/run_inverse.py outputs/basic_pick_place_clean \
  --sam-checkpoint /path/to/sam3.1_multiplex.pt \
  --sam3d-root external/sam-3d-objects --sam3d-checkpoints data/sam3d/checkpoints \
  --population 8 --iterations 4 --device cuda:1 \
  --upload --prefix basic_pick_place_clean --token-file /path/to/hf-token

The runner performs SAM3.1 tracking, SAM3D object reconstruction, an asset preview, contact physics search, pixel refinement, replay verification and video comparison. --reuse keeps completed stages when their dependencies have also been reused. Override the Python executables with --sim-python and --sam-python. Stage status is saved in inverse_run.json.

SAM3.1 saves persistent hand and object masks with tracking confidence. Physics renders actor masks directly from MuJoCo. The objective combines per actor IoU and boundary distance with reference and smoothness penalties. Random search and CEM receive the same candidate budget. The object moves through contact and mj_step after initialization.

Outputs include real_masks.npz, mask_overlay.mp4, sam3d/object.npz, sam3d/object.glb, physics/metrics.json, candidate scenes and rollouts, final_comparison.mp4, and the pixel trace viewer. Metrics describe each run's mask loss, stability, lift and contact forces.

The clean run completed 126 frames at 30 fps. SAM3D produced a watertight mesh with 282,808 triangles and source mask IoU 0.6844. Random search and CEM each used population 8 for four iterations, totaling 33 objective evaluations per method including initialization. Results are saved in physics/metrics.json and physics/refinement/metrics.json.

Result	Sampled objective	Full video mask loss	Maximum lift (m)
Initialization	0.562066876	0.564036387	0.113454375
Random search	0.562066876	0.564036387	0.113454375
CEM	0.551045676	0.551588883	0.108765486
Trace refinement	0.545757148	0.548917344	0.107028888

Refinement selected the right wrist using the worst observed actor's mask errors. It used eight sensitivity evaluations and 33 CEM evaluations, totaling 43 objective evaluations with baseline and final checks. Its acceptance guard requires the target actor loss and combined objective to stay at or below baseline. Sampled right-hand actor loss decreased from 0.763930532 to 0.759033189. The saved candidates are stable and record hand-object contact while lifting. Mask agreement does not establish unique physical parameters. Fifteen tests and Ruff passed.

All eight pipeline stages completed for outputs/basic_pick_place_clean, including reused SAM3.1 masks. Reset replay reproduced all 126 frames with zero object qpos error and zero mask pixel differences, recorded in physics/refinement/replay_verification.json. final_comparison.mp4 shows real video, initial physics and refined physics at 1554 × 294 pixels; the interactive trace viewer is physics/refinement/viewer.html.

Inspect a pixel

Add --frame 66 --pixel 136 53 to the pipeline command to target a mismatch. Frame numbers refer to the original video; pixel coordinates refer to the rendered mask resolution.

.venv-sam3d/bin/python examples/inspect_trace.py \
  outputs/basic_pick_place_clean/physics/refinement/refined_trace.npz \
  --video outputs/basic_pick_place_clean/real_video.mp4 \
  --metrics outputs/basic_pick_place_clean/physics/refinement/metrics.json \
  --output outputs/basic_pick_place_clean/physics/refinement/viewer.html

The viewer shows real and simulated actor IDs, geometry IDs, tracking confidence and mask disagreement. Refinement evaluates parameter sensitivity and adjusts the selected block. A pixel mismatch alone does not identify a unique physical cause.

Upload

.venv-sam3/bin/python examples/upload_results.py \
  --folder outputs/basic_pick_place_clean --prefix basic_pick_place_clean \
  --token-file /path/to/hf-token --code --sha256

Results go to the private twanghcmut/ego-real2sim bucket. Authentication uses the token file or HF_TOKEN. The uploader excludes secrets, checkpoints, caches and virtual environments, records local hashes and verifies remote file sizes. Upload status is saved locally.

License

EgoDex data is licensed CC-BY-NC-ND.