Buckets:

twanghcmut
/

ego-real2sim

Files

xet

twanghcmut/ego-real2sim / basic_pick_place_clean /code /README.md

twanghcmut

20 days ago

preview code

download

raw

7.17 kB

# ego-real2sim

Fit an articulated hand and a rigid object to egocentric video with HaWoR, SAM3.1, SAM3D and MuJoCo. Scene inputs are the recorded camera, a flat table and hand trajectories.

## Setup

```bash
uv venv --python 3.10 .venv-sam3d
uv pip install --python .venv-sam3d/bin/python torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
uv pip install --python .venv-sam3d/bin/python -e '.[object_reconstruction,optimization]'
git clone https://github.com/facebookresearch/sam-3d-objects.git external/sam-3d-objects
git -C external/sam-3d-objects checkout f91db411c50efee93d8db7aeb323885650f6f722
CUDA_VISIBLE_DEVICES="" MAX_JOBS=8 uv pip install --python .venv-sam3d/bin/python --no-build-isolation -r requirements/sam3d.txt
uv venv --python 3.12 .venv-sam3
uv pip install --python .venv-sam3/bin/python torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv-sam3/bin/python -e '.[segmentation,storage]'
```

SAM3D dependencies target Python 3.10, torch 2.5.1 and CUDA 12.1. Supply its checkpoint directory containing `pipeline.yaml` and weights. SAM3.1 uses a separate environment and the `sam3.1_multiplex.pt` checkpoint. HaWoR uses `.venv-hawor`; save its tracked keypoints with `examples/track_hawor.py`.

## Prepare scene

```bash
MUJOCO_GL=egl .venv-sam3d/bin/python examples/prepare_scene.py outputs/basic_pick_place_clean \
  --tracks outputs/track_hawor/20261014_080552/hawor_tracks.npz
```

Preparation uses the recorded EgoDex camera and saved HaWoR tracks. Supply `--table-height` in meters to set the table explicitly; otherwise the contact interval and object height prior determine its height. Use `--help` to change these inputs.

Scene preparation writes:

- `real_video.mp4`: RGB frames with one full image per frame.
- `cameras.npz`: intrinsics, world camera poses, source frame indices, timestamps and image size.
- `sim_trajectory.npz`: hand qpos, camera poses, source frame indices and timestamps.
- `scene.xml`: articulated hands and a flat table.
- `scene.json`: table height and input provenance.

Each video frame corresponds to the same row in the saved camera and hand trajectories.

SAM3D receives the RGB image, the SAM3.1 object mask and an analytic pointmap. Camera rays intersect the calibrated horizontal table; inside the object mask, the intersection plane is raised by the object height prior, 0.045 m by default. This supplies a metric pose and scale initialization. The scaffold depends on those table and height assumptions; it does not measure the object's true depth. Reconstruction sets `depth_model=None`, so it requires no separate MoGe model checkpoint or depth inference.

## Run inverse simulation

```bash
MUJOCO_GL=egl .venv-sam3d/bin/python examples/run_inverse.py outputs/basic_pick_place_clean \
  --sam-checkpoint /path/to/sam3.1_multiplex.pt \
  --sam3d-root external/sam-3d-objects --sam3d-checkpoints data/sam3d/checkpoints \
  --population 8 --iterations 4 --device cuda:1 \
  --upload --prefix basic_pick_place_clean --token-file /path/to/hf-token
```

The runner performs SAM3.1 tracking, SAM3D object reconstruction, an asset preview, contact physics search, pixel refinement, replay verification and video comparison. `--reuse` keeps completed stages when their dependencies have also been reused. Override the Python executables with `--sim-python` and `--sam-python`. Stage status is saved in `inverse_run.json`.

SAM3.1 saves persistent hand and object masks with tracking confidence. Physics renders actor masks directly from MuJoCo. The objective combines per actor IoU and boundary distance with reference and smoothness penalties. Random search and CEM receive the same candidate budget. The object moves through contact and `mj_step` after initialization.

Outputs include `real_masks.npz`, `mask_overlay.mp4`, `sam3d/object.npz`, `sam3d/object.glb`, `physics/metrics.json`, candidate scenes and rollouts, `final_comparison.mp4`, and the pixel trace viewer. Metrics describe each run's mask loss, stability, lift and contact forces.

The clean run completed 126 frames at 30 fps. SAM3D produced a watertight mesh with 282,808 triangles and source mask IoU 0.6844. Random search and CEM each used population 8 for four iterations, totaling 33 objective evaluations per method including initialization. Results are saved in `physics/metrics.json` and `physics/refinement/metrics.json`.

| Result | Sampled objective | Full video mask loss | Maximum lift (m) |
| --- | ---: | ---: | ---: |
| Initialization | 0.562066876 | 0.564036387 | 0.113454375 |
| Random search | 0.562066876 | 0.564036387 | 0.113454375 |
| CEM | 0.551045676 | 0.551588883 | 0.108765486 |
| Trace refinement | 0.545757148 | 0.548917344 | 0.107028888 |

Refinement selected the right wrist using the worst observed actor's mask errors. It used eight sensitivity evaluations and 33 CEM evaluations, totaling 43 objective evaluations with baseline and final checks. Its acceptance guard requires the target actor loss and combined objective to stay at or below baseline. Sampled right-hand actor loss decreased from 0.763930532 to 0.759033189. The saved candidates are stable and record hand-object contact while lifting. Mask agreement does not establish unique physical parameters. Fifteen tests and Ruff passed.

All eight pipeline stages completed for `outputs/basic_pick_place_clean`, including reused SAM3.1 masks. Reset replay reproduced all 126 frames with zero object qpos error and zero mask pixel differences, recorded in `physics/refinement/replay_verification.json`. `final_comparison.mp4` shows real video, initial physics and refined physics at 1554 × 294 pixels; the interactive trace viewer is `physics/refinement/viewer.html`.

## Inspect a pixel

Add `--frame 66 --pixel 136 53` to the pipeline command to target a mismatch. Frame numbers refer to the original video; pixel coordinates refer to the rendered mask resolution.

```bash
.venv-sam3d/bin/python examples/inspect_trace.py \
  outputs/basic_pick_place_clean/physics/refinement/refined_trace.npz \
  --video outputs/basic_pick_place_clean/real_video.mp4 \
  --metrics outputs/basic_pick_place_clean/physics/refinement/metrics.json \
  --output outputs/basic_pick_place_clean/physics/refinement/viewer.html
```

The viewer shows real and simulated actor IDs, geometry IDs, tracking confidence and mask disagreement. Refinement evaluates parameter sensitivity and adjusts the selected block. A pixel mismatch alone does not identify a unique physical cause.

## Upload

```bash
.venv-sam3/bin/python examples/upload_results.py \
  --folder outputs/basic_pick_place_clean --prefix basic_pick_place_clean \
  --token-file /path/to/hf-token --code --sha256
```

Results go to the private [twanghcmut/ego-real2sim bucket](https://huggingface.co/buckets/twanghcmut/ego-real2sim). Authentication uses the token file or `HF_TOKEN`. The uploader excludes secrets, checkpoints, caches and virtual environments, records local hashes and verifies remote file sizes. Upload status is saved locally.

## License

EgoDex data is licensed CC-BY-NC-ND.

Xet Storage Details

Size:: 7.17 kB
Xet hash:: 300bbba3c8135889a96453b471da51a155a1ce6cb75b8ea300b7c87eaffd7a6c

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# ego-real2sim

	Fit an articulated hand and a rigid object to egocentric video with HaWoR, SAM3.1, SAM3D and MuJoCo. Scene inputs are the recorded camera, a flat table and hand trajectories.

	## Setup

	```bash
	uv venv --python 3.10 .venv-sam3d
	uv pip install --python .venv-sam3d/bin/python torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
	uv pip install --python .venv-sam3d/bin/python -e '.[object_reconstruction,optimization]'
	git clone https://github.com/facebookresearch/sam-3d-objects.git external/sam-3d-objects
	git -C external/sam-3d-objects checkout f91db411c50efee93d8db7aeb323885650f6f722
	CUDA_VISIBLE_DEVICES="" MAX_JOBS=8 uv pip install --python .venv-sam3d/bin/python --no-build-isolation -r requirements/sam3d.txt
	uv venv --python 3.12 .venv-sam3
	uv pip install --python .venv-sam3/bin/python torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
	uv pip install --python .venv-sam3/bin/python -e '.[segmentation,storage]'
	```

	SAM3D dependencies target Python 3.10, torch 2.5.1 and CUDA 12.1. Supply its checkpoint directory containing `pipeline.yaml` and weights. SAM3.1 uses a separate environment and the `sam3.1_multiplex.pt` checkpoint. HaWoR uses `.venv-hawor`; save its tracked keypoints with `examples/track_hawor.py`.

	## Prepare scene

	```bash
	MUJOCO_GL=egl .venv-sam3d/bin/python examples/prepare_scene.py outputs/basic_pick_place_clean \
	--tracks outputs/track_hawor/20261014_080552/hawor_tracks.npz
	```

	Preparation uses the recorded EgoDex camera and saved HaWoR tracks. Supply `--table-height` in meters to set the table explicitly; otherwise the contact interval and object height prior determine its height. Use `--help` to change these inputs.

	Scene preparation writes:

	- `real_video.mp4`: RGB frames with one full image per frame.
	- `cameras.npz`: intrinsics, world camera poses, source frame indices, timestamps and image size.
	- `sim_trajectory.npz`: hand qpos, camera poses, source frame indices and timestamps.
	- `scene.xml`: articulated hands and a flat table.
	- `scene.json`: table height and input provenance.

	Each video frame corresponds to the same row in the saved camera and hand trajectories.

	SAM3D receives the RGB image, the SAM3.1 object mask and an analytic pointmap. Camera rays intersect the calibrated horizontal table; inside the object mask, the intersection plane is raised by the object height prior, 0.045 m by default. This supplies a metric pose and scale initialization. The scaffold depends on those table and height assumptions; it does not measure the object's true depth. Reconstruction sets `depth_model=None`, so it requires no separate MoGe model checkpoint or depth inference.

	## Run inverse simulation

	```bash
	MUJOCO_GL=egl .venv-sam3d/bin/python examples/run_inverse.py outputs/basic_pick_place_clean \
	--sam-checkpoint /path/to/sam3.1_multiplex.pt \
	--sam3d-root external/sam-3d-objects --sam3d-checkpoints data/sam3d/checkpoints \
	--population 8 --iterations 4 --device cuda:1 \
	--upload --prefix basic_pick_place_clean --token-file /path/to/hf-token
	```

	The runner performs SAM3.1 tracking, SAM3D object reconstruction, an asset preview, contact physics search, pixel refinement, replay verification and video comparison. `--reuse` keeps completed stages when their dependencies have also been reused. Override the Python executables with `--sim-python` and `--sam-python`. Stage status is saved in `inverse_run.json`.

	SAM3.1 saves persistent hand and object masks with tracking confidence. Physics renders actor masks directly from MuJoCo. The objective combines per actor IoU and boundary distance with reference and smoothness penalties. Random search and CEM receive the same candidate budget. The object moves through contact and `mj_step` after initialization.

	Outputs include `real_masks.npz`, `mask_overlay.mp4`, `sam3d/object.npz`, `sam3d/object.glb`, `physics/metrics.json`, candidate scenes and rollouts, `final_comparison.mp4`, and the pixel trace viewer. Metrics describe each run's mask loss, stability, lift and contact forces.

	The clean run completed 126 frames at 30 fps. SAM3D produced a watertight mesh with 282,808 triangles and source mask IoU 0.6844. Random search and CEM each used population 8 for four iterations, totaling 33 objective evaluations per method including initialization. Results are saved in `physics/metrics.json` and `physics/refinement/metrics.json`.

	\| Result \| Sampled objective \| Full video mask loss \| Maximum lift (m) \|
	\| --- \| ---: \| ---: \| ---: \|
	\| Initialization \| 0.562066876 \| 0.564036387 \| 0.113454375 \|
	\| Random search \| 0.562066876 \| 0.564036387 \| 0.113454375 \|
	\| CEM \| 0.551045676 \| 0.551588883 \| 0.108765486 \|
	\| Trace refinement \| 0.545757148 \| 0.548917344 \| 0.107028888 \|

	Refinement selected the right wrist using the worst observed actor's mask errors. It used eight sensitivity evaluations and 33 CEM evaluations, totaling 43 objective evaluations with baseline and final checks. Its acceptance guard requires the target actor loss and combined objective to stay at or below baseline. Sampled right-hand actor loss decreased from 0.763930532 to 0.759033189. The saved candidates are stable and record hand-object contact while lifting. Mask agreement does not establish unique physical parameters. Fifteen tests and Ruff passed.

	All eight pipeline stages completed for `outputs/basic_pick_place_clean`, including reused SAM3.1 masks. Reset replay reproduced all 126 frames with zero object qpos error and zero mask pixel differences, recorded in `physics/refinement/replay_verification.json`. `final_comparison.mp4` shows real video, initial physics and refined physics at 1554 × 294 pixels; the interactive trace viewer is `physics/refinement/viewer.html`.

	## Inspect a pixel

	Add `--frame 66 --pixel 136 53` to the pipeline command to target a mismatch. Frame numbers refer to the original video; pixel coordinates refer to the rendered mask resolution.

	```bash
	.venv-sam3d/bin/python examples/inspect_trace.py \
	outputs/basic_pick_place_clean/physics/refinement/refined_trace.npz \
	--video outputs/basic_pick_place_clean/real_video.mp4 \
	--metrics outputs/basic_pick_place_clean/physics/refinement/metrics.json \
	--output outputs/basic_pick_place_clean/physics/refinement/viewer.html
	```

	The viewer shows real and simulated actor IDs, geometry IDs, tracking confidence and mask disagreement. Refinement evaluates parameter sensitivity and adjusts the selected block. A pixel mismatch alone does not identify a unique physical cause.

	## Upload

	```bash
	.venv-sam3/bin/python examples/upload_results.py \
	--folder outputs/basic_pick_place_clean --prefix basic_pick_place_clean \
	--token-file /path/to/hf-token --code --sha256
	```

	Results go to the private [twanghcmut/ego-real2sim bucket](https://huggingface.co/buckets/twanghcmut/ego-real2sim). Authentication uses the token file or `HF_TOKEN`. The uploader excludes secrets, checkpoints, caches and virtual environments, records local hashes and verifies remote file sizes. Upload status is saved locally.

	## License

	EgoDex data is licensed CC-BY-NC-ND.