basic_pick_place/code/README.md

ego-real2sim

Reconstruct egocentric video with MapAnything, track hands and objects with SAM3.1, and replay articulated EgoDex or HaWoR hands in MuJoCo.

Setup

python -m venv .venv
.venv/bin/pip install -e .

The EgoDex sample code lives in external/ml-egodex as a git submodule. This package does not import it.

Usage

from real2sim import Episode, HandScene, video

episode = Episode.load("basic_pick_place", 0)
scene = HandScene.from_episode(episode)

ego = scene.render("ego")
mask = scene.render_mask("ego")
overview = scene.render("overview")

Episode.load downloads the episode from the public EgoDex archive when it is missing, so only the files you ask for are fetched. Run examples/replay_episode.py with MUJOCO_GL=egl to produce a side-by-side video, an overlay of the sim hands on the real video, and an episode.npz trajectory file. Results go to outputs/, logs to logs/.

Tracking with HaWoR

real2sim.tracking.HaWoR runs the HaWoR hand detector and motion model on a video and returns 21 MANO keypoints per hand in the camera frame. It skips DROID-SLAM and Metric3D: for EgoDex the camera pose is known, so no SLAM is needed. HaWoR needs its own environment (torch 2.4, ultralytics, smplx), so create .venv-hawor and install this package into it with pip install --no-deps -e .. Weights go under external/HaWoR/weights and the MANO pickles are symlinked into external/HaWoR/_DATA.

examples/track_hawor.py compares the tracked keypoints with the EgoDex annotations and writes a video with both skeletons drawn. examples/replay_hawor.py loads the saved tracks, turns the keypoints into a MANO skeleton with per-joint rotations (Hand.from_keypoints), and replays them in MuJoCo next to the EgoDex hands. Frames where HaWoR lost the hand are hidden below the floor.

MapAnything

uv venv --python 3.10 .venv-mapanything
uv pip install --python .venv-mapanything/bin/python torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
uv pip install --python .venv-mapanything/bin/python -e '.[reconstruction]'
HF_HOME="$PWD/data/huggingface" TORCH_HOME="$PWD/data/torch" MUJOCO_GL=egl \
  .venv-mapanything/bin/python examples/run_mapanything.py \
  --output outputs/mapanything_run --device cuda:1 \
  --tracks outputs/track_hawor/20261014_080552/hawor_tracks.npz

Outputs: reconstruction.mp4, orbit.mp4, mujoco_replay.mp4, scene.ply, scene.xml, scene.mjb, cameras.npz, sim_trajectory.npz, sim_masks.npz, and metrics JSON.

Keep MapAnything and SAM3D in a Python 3.10 or 3.11 environment with torch 2.5. SAM3D uses its own source checkout, checkpoints, and native dependencies; the reconstruction extra installs MapAnything. Keep SAM3.1 in a separate Python 3.12 environment with torch 2.10. Install the reconstruction and segmentation extras into their respective environments.

MUJOCO_GL=egl .venv-mapanything/bin/python examples/replay_mapanything.py \
  outputs/mapanything_run \
  --tracks outputs/track_hawor/20261014_080552/hawor_tracks.npz
.venv-mapanything/bin/python -m unittest discover -s tests -v

SAM3.1 masks

uv venv --python 3.12 .venv-sam3
uv pip install --python .venv-sam3/bin/python torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
uv pip install --python .venv-sam3/bin/python -e '.[segmentation]'
.venv-sam3/bin/python examples/segment_mapanything.py outputs/mapanything_run \
  --prompts examples/basic_pick_place_prompts.json \
  --checkpoint data/sam3/sam3.1_multiplex.pt --device cuda:1

The checkpoint must be available at the supplied path. The tracker saves real_masks.npz, mask_overlay.mp4, mask_preview.png, and tracking_metrics.json. The saved model tag is sam3.1_multiplex.

SAM3D objects

SAM3D reconstruction has completed for the sample clip. The asset is saved at outputs/mapanything_final/sam3d/object.npz, alongside object.glb, object_world.ply, input.png, and metrics.json. The reconstructed mesh is watertight, with 143,002 vertices and 286,020 triangles. The rendered object silhouette has an IoU of 0.6915 against the SAM3.1 mask at source frame 10. Its world coordinates use the MapAnything metric pointmap and the scene alignment in surface.npz.

Use the torch 2.5 environment for SAM3D. Install torch and the reconstruction extra with the MapAnything commands above before installing requirements/sam3d.txt. The provided dependency file targets Python 3.10, torch 2.5.1, and CUDA 12.1; it includes a Python 3.10 FlashAttention wheel. Python 3.11 requires a matching wheel. Native dependencies include PyTorch3D, Kaolin, FlashAttention, spconv, and gsplat. The installation below builds PyTorch3D without GPU extensions; the object reconstruction uses vertex colors with texture baking and layout postprocessing disabled.

git clone https://github.com/facebookresearch/sam-3d-objects.git external/sam-3d-objects
git -C external/sam-3d-objects checkout f91db411c50efee93d8db7aeb323885650f6f722
CUDA_VISIBLE_DEVICES="" MAX_JOBS=8 uv pip install \
  --python .venv-mapanything/bin/python --no-build-isolation \
  -r requirements/sam3d.txt
.venv-mapanything/bin/python examples/reconstruct_object.py outputs/mapanything_final \
  --root external/sam-3d-objects --checkpoints data/sam3d/checkpoints \
  --output outputs/mapanything_final/sam3d --frame 10 --seed 42 --device cuda:1
MUJOCO_GL=egl .venv-mapanything/bin/python examples/preview_object.py \
  outputs/mapanything_final/sam3d/object.npz

The source root may be an ignored symlink to the existing RAM checkout. The checkpoint directory must contain pipeline.yaml and the SAM3D weights referenced by that configuration. Reconstruction consumes the saved MapAnything views, camera bundle, surface alignment, and SAM3.1 object mask. The preview command produces object_orbit.mp4, object_preview.png, and alignment.png next to the asset.

Mask fitting and contact physics

The current hand fitting command optimizes wrist corrections against the SAM3.1 masks. Random search and CEM use the same population and iteration counts.

uv pip install --python .venv-mapanything/bin/python -e '.[optimization]'
MUJOCO_GL=egl .venv-mapanything/bin/python examples/fit_hand_masks.py \
  outputs/mapanything_run --population 16 --iterations 6 --stride 5 --seed 42

This produces fitted_hand_trajectory.npz, hand_fit.mp4, hand_fit_preview.png, hand_fit_metrics.json, and hand_fit_loss.png. Its metrics identify the stage as kinematic_hand_alignment.

The contact physics entry point accepts a SAM3D asset file and the reconstruction folder containing the SAM3.1 masks and hand trajectory:

MUJOCO_GL=egl .venv-mapanything/bin/python examples/fit_physics.py \
  outputs/mapanything_final --asset outputs/mapanything_final/sam3d/object.npz \
  --output outputs/mapanything_final/physics --population 12 --iterations 5 --stride 5 --seed 42

The baseline, original global refinement, and object-targeted refinement completed all 126 frames. Random search and CEM each used a population of 12 for five iterations, with 61 objective evaluations per method including initialization. The baseline summary is physics/metrics.json; the original global refinement is archived in physics/global_refinement/metrics.json, and the current object-targeted result is in physics/refinement/metrics.json. Lower mask loss is better. Maximum lift measures height relative to the object's starting position.

Result	Sampled objective	Full mask loss	Maximum lift (m)
Initialization	0.521123820	0.522445040	0.103529613
Random search	0.521123820	0.522445040	0.103529613
CEM	0.519711859	0.520399145	0.102780952
Archived global refinement	0.505381194	0.505724086	0.104426260
Object-targeted refinement	0.505849466	0.510567164	0.105025778

The object-targeted run includes source frame 66 in its 27 sampled frames. On that sampling set, the CEM baseline objective is 0.522846931 and the refined objective is 0.505849466. Earlier rows use the original sampling set; full mask losses cover all 126 frames.

These completed candidates are stable and record hand-object contact during lifting. The archived refinement selected right_wrist: its combined objective decreased, while the right-hand actor loss increased from 0.757568040 to 0.780932347 and the object actor loss decreased from 0.381381638 to 0.314526513. This global result does not demonstrate improved right-hand mask alignment. Mask agreement does not establish unique physical parameters.

Replay verification

Replay a saved candidate from its parameters and compare the simulated object joint states and rendered actor masks with its saved rollout:

MUJOCO_GL=egl .venv-mapanything/bin/python examples/replay_physics.py \
  outputs/mapanything_final --asset outputs/mapanything_final/sam3d/object.npz \
  --result outputs/mapanything_final/physics/cem

The result folder must contain rollout.npz and either parameters.npz or best_parameters.npz. Use an initial, random, cem, global_refinement, or completed refinement result folder. Verification requires object qpos error at most 1e-9 and zero mask pixel differences. A successful run saves verified_replay.mp4 and replay_verification.json in that folder. --output selects another video destination.

The saved CEM, archived global refinement, and current object-targeted refinement replays have been verified across 126 frames with zero object qpos error and zero mask pixel differences.

Render the final comparison of the real clip, initialization, and physics fitted using the pixel query:

MUJOCO_GL=egl .venv-mapanything/bin/python examples/render_comparison.py \
  outputs/mapanything_final

This writes final_comparison.mp4 and final_preview.png in the reconstruction folder. The sample video contains 126 frames at 1554 × 294 pixels, with real video, initial physics, and pixel-guided physics arranged left to right.

Pixel error traces and refinement

The refinement entry point compares the real and rendered masks, records false positive and false negative pixels, tests parameter blocks with physics rollouts, and runs CEM on the selected block. It uses the baseline parameters in physics/best_parameters.json.

MUJOCO_GL=egl .venv-mapanything/bin/python examples/refine_physics.py \
  outputs/mapanything_final --asset outputs/mapanything_final/sam3d/object.npz \
  --population 8 --iterations 4 --stride 5 --seed 42 --frame 66 --pixel 136 53
.venv-mapanything/bin/python examples/refine_physics.py outputs/mapanything_final \
  --asset outputs/mapanything_final/sam3d/object.npz \
  --query-only --frame 66 --pixel 136 53 \
  --trace outputs/mapanything_final/physics/refinement/refined_trace.npz

Outputs go to physics/refinement/: trace.npz, trace.json, trace_overlay.mp4, error_heatmap.png, their refined_ counterparts, sensitivity.json, metrics.json, the refined rollout and parameters, and comparison videos. Supply --frame and --pixel together. At source frame 66, mask pixel (136, 53) has a real object mask and simulated background in the baseline trace. This query selected object as the target actor and added frame 66 to the objective's sampled frames.

The completed object-targeted run selected right_wrist. It used eight sensitivity evaluations and 33 CEM evaluations with population eight and four iterations, recording 43 total objective evaluations including baseline and final checks. The acceptance guard requires the target actor loss to stay at or below its baseline. The sampled objective decreased from 0.522846931 to 0.505849466, full mask loss decreased from 0.520399145 to 0.510567164, and maximum lift reached 0.105025778 m with hand-object contact.

Actor loss on the targeted sampling set	Baseline	Refined
Left hand	0.417780613	0.417780613
Right hand	0.744978792	0.765795578
Object	0.404147024	0.331938167

The object target improved, while right-hand mask loss increased. The guard applies to the selected target actor.

The original global refinement remains in physics/global_refinement/. It used eight sensitivity evaluations and 33 CEM evaluations with population eight and four iterations, recording 43 total objective evaluations including baseline and final checks. Its viewer is physics/global_refinement/viewer.html. The archived values above describe that completed global run.

The query uses a source frame index and pixel coordinates in the trace image. It returns the real and simulated actor, mask disagreement, tracking confidence, and available renderer geometry ID. Parameter sensitivity guides refinement; it does not establish a unique physical cause for a mismatched pixel.

Build a standalone HTML viewer for the saved refined trace:

.venv-mapanything/bin/python examples/inspect_trace.py \
  outputs/mapanything_final/physics/refinement/refined_trace.npz \
  --video outputs/mapanything_final/mujoco_replay.mp4 \
  --output outputs/mapanything_final/physics/refinement/viewer.html \
  --metrics outputs/mapanything_final/physics/refinement/metrics.json

Open viewer.html locally, move the timeline, and click either canvas or enter X and Y. The viewer uses native mask coordinates, normally 256 × 144; entering 50, 50 selects that mask pixel. RGB backgrounds come from the replay's first panel at the matching source frame. Toggle false positive and false negative overlays, choose an actor, and inspect saved geometry IDs, tracking confidence, and observed state. Refinement metrics and sensitivity are displayed when supplied. All frame images and metadata are embedded in the HTML.

Inverse pipeline

examples/run_inverse.py starts from a completed MapAnything reconstruction and HaWoR replay. It runs SAM3.1 masks, SAM3D reconstruction, the asset preview, contact physics search, pixel refinement, replay verification, and the HTML pixel viewer in sequence, then optionally uploads the results. Each stage uses its own Python environment. --reuse keeps stages whose expected output files already exist.

.venv-mapanything/bin/python examples/run_inverse.py outputs/mapanything_final \
  --reuse --sam-checkpoint /path/to/sam3.1_multiplex.pt \
  --sam3d-root external/sam-3d-objects --sam3d-checkpoints data/sam3d/checkpoints \
  --population 12 --iterations 5 --device cuda:1 --frame 66 --pixel 136 53 \
  --upload --token-file /path/to/hf-token

The default executables are .venv-mapanything/bin/python and .venv-sam3/bin/python; override them with --sim-python and --sam-python. Supply the existing SAM3.1 checkpoint path. The paired --frame and --pixel options are forwarded to refinement; the frame is a source video index and the pixel uses native mask coordinates. Pipeline stage status is saved in inverse_run.json. With --upload, results and a code snapshot go to the private bucket under basic_pick_place with local SHA256 hashes recorded in the manifest.

Baseline fitting, the archived global refinement, and the current object-targeted refinement have completed for this sample. The current trace and viewer use source frame 66 and pixel (136, 53) for the requested query. The pipeline checks physics/metrics.json, physics/refinement/metrics.json, physics/refinement/replay_verification.json, and physics/refinement/viewer.html for their respective stages.

Upload results

The uploader uses Hugging Face SDK 2.2 storage buckets. The existing private bucket is twanghcmut/ego-real2sim.

uv pip install --python .venv-sam3/bin/python -e '.[storage]'
.venv-sam3/bin/python examples/upload_results.py \
  --folder outputs/mapanything_run --prefix basic_pick_place/latest \
  --token-file /path/to/hf-token --code --sha256

Authentication comes from --token-file or an existing HF_TOKEN environment variable. --bucket selects another existing private bucket. --code includes a snapshot of this repository under the prefix's code/ directory; --code /path/to/source selects another source folder. --sha256 records local file hashes in the remote manifest.

The uploader recursively uploads eligible files on every run, checks remote sizes, and uploads upload_manifest.json under the chosen prefix. It writes upload_status.json into the local results folder. A verified_sizes status confirms that the uploaded file and manifest sizes match. SHA256 values record the local inputs; remote contents are verified by size.

Virtual environments, Git metadata, caches, symlinks, secrets, and model checkpoints are excluded. Files containing Hugging Face tokens or private keys are also excluded. Code snapshots additionally exclude data, outputs, logs, external checkouts, and build directories. Existing remote files outside this upload remain in the bucket.

Layout

src/real2sim/
  egodex.py     download, load and save episodes
  evaluate.py   keypoint errors and 2D skeleton drawing
  tracking/     HaWoR wrapper
  reconstruction/  MapAnything, SAM3D objects, pointmaps, previews and surfaces
  hands.py      Hand: per-joint SE(3) poses, confidences, 21 keypoints
  skeleton.py   the 25 ARKit joints, parent links, keypoint order
  geometry.py   ARKit y-up to z-up, quaternion helpers
  video.py      read, write, stack and overlay clips
  paths.py      data, outputs and logs folders
  sim/rig.py    one hand as a MuJoCo body tree
  sim/scene.py  both hands, ego camera, overview camera, rendering
  sim/reconstructed.py  textured reconstructed scene and geometry IDs
examples/       runnable scripts

How the replay works

EgoDex gives an SE(3) pose for every hand joint, so no MANO fit is needed. For each hand the rig takes the median bone vector in the parent joint frame, builds a body tree with one ball joint per bone, and sets the wrist as a free joint. Each frame then sets the wrist pose and the relative rotation between each parent and child. Forward kinematics reproduces the recorded joint positions with 0.000 mm error on the sample episode, which confirms the bones are rigid in the data.

World frame is z-up, in meters. The ARKit origin sits on the floor and changes between episodes. The camera follows the OpenCV convention and is converted for MuJoCo.

License

EgoDex data is licensed CC-BY-NC-ND.