docs/MODEL_OVERVIEW.md

# Model Overview

## What is GEM?

GEM (Generalist Estimation of Human Motion) is a commercial-grade monocular video 3D human pose estimation model developed by NVIDIA. It recovers full-body 77-joint motion (body, hands, and face) from monocular video using the SOMA parametric body model.

## GEM vs GENMO

GEM is the commercial successor to the research project GENMO. Key differences:

| | GENMO (Research) | GEM (This Repo) |
|---|---|---|
| Body model | SMPL (24 joints) | SOMA (77 joints: body + hands + face) |
| Modalities | Video + text + audio + music | Video only |
| License | Research only | Apache 2.0 (commercial) |
| 2D Pose Model | COCO-17 keypoints | SOMA-77 keypoints |
| Training Data | Public + internal | NVIDIA-owned only |

## SOMA Body Model

GEM uses the **SOMA** parametric body model:

- **77 joints** covering full body, hands, and face
- **MHR identity model** for body shape representation
- Bundled as a submodule in `third_party/soma`

## Architecture

| Property | Value |
|---|---|
| Parameters | ~520M |
| Architecture | 16-layer Transformer encoder |
| Positional encoding | RoPE (Rotary Position Embedding) |
| Latent dimension | 1024 |
| Attention heads | 8 |
| Decoder | Regression-based denoising decoder |
| Optimizer | AdamW (lr=2e-4) |
| Precision | 16-bit mixed |

The model takes as input video features (from SAM-3D-Body), 2D keypoints, and camera intrinsics, and outputs per-frame SOMA body parameters in both camera-relative and global coordinate frames.

## Bundled 2D Pose Model

GEM includes a trained 2D pose estimation model:

- **Backbone:** DINOv3 (ViT-based)
- **Output:** 77 SOMA keypoints per frame (x, y, confidence)
- **Usage:** Automatically invoked during the demo preprocessing pipeline

The 2D pose model is accessed via `gem.utils.vitpose_extractor.VitPoseExtractor`.