docs/DEMO.md

Demo

Pipeline Overview

Input Video → Human Detection (YOLOX) → 2D Keypoints (VitPose) → Features (SAM3D) → GEM Model → 3D Pose (SOMA)
                                              ↓                                                        ↓
                                      2D Keypoint Overlay                              (Optional) Retarget → G1 Robot Motion

All demo scripts use YOLOX + ByteTrack for person detection and tracking.

Full 3D Pipeline (demo_soma.py)

Run full inference on a video:

python scripts/demo/demo_soma.py \
  --video path/to/video.mp4 \
  --output_root outputs \
  --ckpt inputs/pretrained/gem_soma.ckpt

Note: The --ckpt argument is optional. If omitted, the script will automatically download the pretrained checkpoint from HuggingFace.

Arguments

Argument	Default	Description
--video	—	Input video path (required)
--ckpt	null	Pretrained checkpoint path
-s / --static_cam	off	Assume static camera (disables VO)
--output_root	outputs/demo_soma	Root directory for outputs
--verbose	off	Save debug overlays (bbox, pose)
--render_mhr	off	Render MHR identity model
--retarget	off	Retarget motion to Unitree G1 robot (requires soma-retargeter)

Outputs

Results are saved to <output_root>/<video_name>/:

File	Description
0_kp2d77_overlay.mp4	2D keypoint overlay on input video
<video_name>_1_incam.mp4	In-camera mesh overlay
<video_name>_2_global.mp4	Global-coordinate render
<video_name>_3_incam_global_horiz.mp4	Side-by-side (or 2x2 grid with --retarget)
preprocess/bbx.pt	Detected bounding boxes
preprocess/vitpose.pt	2D keypoints (77 joints)
preprocess/hpe_results.pt	Full 3D pose prediction
<video_name>_retarget_g1.bvh	G1 robot motion in BVH format (with --retarget)
<video_name>_retarget_g1.csv	G1 robot joint angles (with --retarget)
<video_name>_4_g1_retarget.mp4	G1 robot motion video (with --retarget)

Preprocessing Fallbacks

• When no pre-computed bbx.pt exists, the demo runs human detection via YOLOX + ByteTrack.
• If VO modules are unavailable, the demo falls back to a static camera trajectory.

Accelerated Pipeline (demo_soma_onnx.py)

ONNX/TensorRT-accelerated variant of demo_soma.py. Replaces PyTorch inference with ONNX Runtime for VitPose, SAM-3D-Body, and the GEM denoiser.

macOS: This script supports Apple Silicon via ONNX Runtime with the CoreML Execution Provider. See INSTALL_MACOS.md for setup instructions.

python scripts/demo/demo_soma_onnx.py \
  --video path/to/video.mp4

Prerequisites

ONNX models are automatically downloaded from HuggingFace on first run if not found locally. To export your own ONNX models instead:

python tools/export/export_vitpose_onnx.py
python tools/export/export_sam3db_onnx.py
python tools/export/export_denoiser_onnx.py --ckpt <path>

Arguments

Argument	Default	Description
--video	—	Input video path (required)
--ckpt	null	Pretrained checkpoint path
-s / --static_cam	off	Assume static camera (disables VO)
--output_root	outputs/demo_soma_onnx	Root directory for outputs
--verbose	off	Save debug overlays
--force_pytorch	off	Force PyTorch inference even if ONNX/TRT available
--no-imgfeat	off	Skip SAM3DB, use 2D keypoints only
--ddim	off	DDIM sampling (50 steps) instead of regression — slower but higher quality
--retarget	off	Retarget motion to Unitree G1 robot

Outputs

Same as demo_soma.py. When --retarget is used, the final composite is a 2x2 grid (kp2d, incam, global, retarget).

Humanoid Robot Retargeting (--retarget)

Retarget the recovered SOMA motion to a Unitree G1 humanoid robot:

python scripts/demo/demo_soma.py \
  --video path/to/video.mp4 \
  --retarget

This requires the soma-retargeter package (see Installation). The output includes a G1 robot motion video and joint angle CSV. When --retarget is used, the final composite video shows a 2x2 grid: 2D keypoints, in-camera mesh, global mesh, and G1 robot motion.

2D Keypoint-Only Demo (demo_2d_keypoints.py)

A lightweight demo that runs only detection and 2D keypoint extraction — no GEM model, no 3D rendering, no Hydra config.

python scripts/demo/demo_2d_keypoints.py \
  --video path/to/video.mp4

Arguments

Argument	Default	Description
--video	—	Input video path (required)
--output_dir	outputs/demo_2d_kp/<video_name>/	Output directory
--detector_name	vitdet	Human detector: vitdet or sam3
--conf_thr	0.5	Confidence threshold for visualization
--save_raw	off	Keep intermediate .pt files

Output

• <video_name>_kp2d77_overlay.mp4 — 2D keypoint overlay video

Accessing Results Programmatically

import torch

# Load 2D keypoints
vitpose = torch.load("outputs/demo_soma/<video>/preprocess/vitpose.pt")
# vitpose shape: (num_frames, 77, 3) — x, y, confidence

# Load bounding boxes
bbx = torch.load("outputs/demo_soma/<video>/preprocess/bbx.pt")
bbx_xyxy = bbx["bbx_xyxy"]  # (num_frames, 4)
bbx_xys = bbx["bbx_xys"]    # (num_frames, 3) — center_x, center_y, scale

# Load 3D prediction
pred = torch.load("outputs/demo_soma/<video>/preprocess/hpe_results.pt")