README.md

---
license: other
license_name: embedl-models-community-licence-1.0
license_link: https://github.com/embedl/embedl-models/blob/main/LICENSE
base_model:
- facebook/sam-3d-body-dinov3
quantized_from:
- facebook/sam-3d-body-dinov3
tags:
- image-feature-extraction
- sam
- sam-3d-body
- dinov3
- quantization
- onnx
- tensorrt
- edge
- embedl
gated: true
extra_gated_heading: "Access Embedl SAM 3D Body (Quantized)"
extra_gated_description: >-
  To access this model, please review and accept the terms below.
  Your contact information is collected solely to manage access and,
  with your explicit consent, to notify you about updated or new
  optimized models from Embedl. You can withdraw consent at any time
  by contacting us (see Contact section below). See our license for full terms.
extra_gated_button_content: "Agree and request access"
extra_gated_prompt: "By requesting access you agree to the Embedl Models Community Licence and the upstream SAM 3D Body License. You must also have access to the gated upstream model facebook/sam-3d-body-dinov3."
extra_gated_fields:
  Company: text
  I agree to the Embedl Models Community Licence and upstream SAM 3D Body License: checkbox
  I consent to being contacted by Embedl about products and services: checkbox
---

# Embedl SAM 3D Body (Quantized)

Deployable version of [facebook/sam-3d-body-dinov3](https://huggingface.co/facebook/sam-3d-body-dinov3),
Meta's single-image full-body human-mesh recovery model. The repository ships
**both stages of the pipeline as ONNX models** for low-latency NVIDIA TensorRT
inference:

- **Encoder** — the 840.6 M-parameter **DINOv3 ViT-H+/16 vision backbone**
  (the dominant compute cost), mixed-precision INT8/FP16-quantized with
  hardware-aware optimizations from [embedl-deploy](https://github.com/embedl/embedl-deploy).
- **Decoder** — the SAM-3D-Body promptable decoder + **MHR parametric mesh
  head** (body-only inference path), exported to ONNX; it maps backbone
  features directly to the 3D mesh vertices and camera.

Image → 3D body mesh runs end-to-end in TensorRT in **~42 ms on an NVIDIA L4**
(see [Performance](#performance)).

## 3D Human Mesh Recovery

Below is our **INT8-quantized** backbone applied to the Gothenburg *Poseidon*
(Carl Milles): input → mesh overlay → ¾ view → side view. The INT8 mesh stays
within **~0.5 % (mean ≈ 10 mm)** of the FP32 result. Reproduce it with
[`demo3d_trt.py`](demo3d_trt.py) (all-TensorRT) or [`demo_3d.py`](demo_3d.py)
(PyTorch pipeline).

<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/sam-3d-body/SAM-3D-Body__demo.png" width="100%">

<table style="width: 100%; border-collapse: collapse; border: none;">
  <tr style="border: none;">
    <td style="width: 50%; border: none; padding: 10px;" align="center">
      <b>Rotating recovered mesh</b><br>
      <img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/sam-3d-body/SAM-3D-Body__spin.gif" width="60%">
    </td>
    <td style="width: 50%; border: none; padding: 10px;" align="center">
      <b>Backbone features (PCA)</b><br>
      <img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/sam-3d-body/SAM-3D-Body__pca.png" width="60%">
    </td>
  </tr>
</table>

<table style="width: 100%; border-collapse: collapse; border: none;">
  <tr style="border: none;">
    <td style="width: 100%; border: none; padding: 10px;">
      <p align="center"><b>Nvidia L4</b></p>
      <img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/sam-3d-body/SAM-3D-Body__l4.svg" style="width: 100%;">
    </td>
  </tr>
</table>

<a href="https://hfviewer.com/facebook/sam-3d-body-dinov3?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=facebook__sam-3d-body-dinov3_card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=facebook%2Fsam-3d-body-dinov3&amp;v=20260501clipcard"
    alt="Open facebook/sam-3d-body-dinov3 in hfviewer"
    width="100%"
  />
</a>

## Highlights

- **End-to-end TensorRT:** image → mesh with both models as TRT engines,
  **42 ms** engine time on an L4 (encoder 26 ms + decoder 16 ms), no PyTorch
  and no upstream checkpoint needed at inference time.
- **Formats:** encoder as ONNX with external weights
  (`embedl_sam3dbody_int8.onnx` + `.onnx.data`) and as a `torch.export` graph
  (`embedl_sam3dbody_int8.pt2`); decoder as ONNX (`sam3dbody_decoder.onnx`).
- **Precision:** encoder INT8 with sensitive layers (the 3-channel patch-embed
  conv and LayerNorms) kept in FP16; decoder FP16 with the MHR mesh-rig math
  kept in FP32. Both ONNX files ship in fp32 — the demo casts them at
  engine-build time.
- **Input:** a single `(1, 3, 512, 512)` ImageNet-normalized person crop →
  `(1, 1280, 32, 32)` feature map → mesh (18 439 vertices) + camera.
- **3.9× faster, 2.9× smaller, 2.7× less peak GPU memory** than the
  TensorRT-default FP32 pipeline on an NVIDIA L4 (6.3× vs strict FP32);
  recovered 3D mesh within **~0.5 %** of the FP32 result.

## Quick Start — end-to-end TensorRT demo

`demo3d_trt.py` recovers and renders the 3D body from a photo, running both
models as TensorRT engines. It only needs numpy/OpenCV-class dependencies —
no PyTorch, no upstream repo, no upstream checkpoint.

```bash
hf download embedl/sam-3d-body \
  embedl_sam3dbody_int8.onnx embedl_sam3dbody_int8.onnx.data \
  sam3dbody_decoder.onnx sam3dbody_faces.npy \
  demo3d_trt.py sample_input.png --local-dir .

pip install tensorrt pycuda onnx numpy opencv-python-headless pillow matplotlib imageio
# on CUDA-12 drivers install `tensorrt-cu12` instead of `tensorrt` (see Performance notes)

python demo3d_trt.py --image sample_input.png --bbox 180 210 700 950
# add --bbox x1 y1 x2 y2 to crop one person out of a wider scene (defaults to the full image)
```

The first run casts both ONNX models to FP16 (the decoder's mesh-rig section
stays FP32 for accuracy) and builds the engines (a few minutes, cached next
to the ONNX files). Afterwards:

```
recovered mesh: 18439 vertices (TensorRT end to end)
inference latency (engine execution only):
  encoder  mean  26.56 ms  p95  27.63 ms
  decoder  mean  15.85 ms  p95  15.96 ms
  total    mean  42.41 ms
wrote mesh_demo_trt.png and mesh_demo_trt_spin.gif
```

The decoder engine runs the SAM-3D-Body *body* inference path (equivalent to
upstream `inference_type="body"`): full-body mesh without the optional
hand-refinement passes.

### Backbone only

`infer_trt.py` (TensorRT) and `infer_pt2.py` (`torch.export`) run just the
encoder on an image, report feature statistics + latency, and save a **PCA
visualization** of the patch features — the classic DINOv3 "what the backbone
sees" image:

```bash
pip install pillow numpy onnx           # + tensorrt/pycuda (trt) or torch (pt2)
python infer_trt.py --image sample_input.png --save-pca features_pca.png
python infer_pt2.py --image sample_input.png --save-pca features_pca.png
```

## PyTorch reference pipeline (`demo_3d.py`)

`demo_3d.py` runs the **upstream eager-PyTorch pipeline** with this INT8
backbone swapped in — useful as a reference, or when you need the full
inference mode with hand refinement. It requires the upstream code and gated
checkpoint:

```bash
pip install torch matplotlib pillow numpy imageio opencv-python huggingface_hub

# upstream SAM-3D-Body pipeline (not a pip package - use via PYTHONPATH)
git clone https://github.com/facebookresearch/sam-3d-body
# its runtime deps (see sam-3d-body/INSTALL.md):
pip install pytorch-lightning yacs scikit-image einops timm dill hydra-core hydra-colorlog \
  pyrootutils roma loguru optree fvcore trimesh braceexpand webdataset "networkx==3.2.1" \
  chump jsonlines joblib pandas rich smplx torchvision

# gated upstream checkpoint (accept the licence at facebook/sam-3d-body-dinov3 first)
hf download facebook/sam-3d-body-dinov3 model.ckpt model_config.yaml assets/mhr_model.pt --local-dir sam3d_ckpt
hf download embedl/sam-3d-body embedl_sam3dbody_int8.pt2 demo_3d.py sample_input.png --local-dir .

PYTHONPATH=sam-3d-body python demo_3d.py \
  --image sample_input.png \
  --ckpt-dir sam3d_ckpt \
  --pt2 embedl_sam3dbody_int8.pt2 \
  --out mesh_demo.png
```

## Files

| File | Description |
|---|---|
| `embedl_sam3dbody_int8.onnx` | **Encoder** — quantized ONNX with precalibrated Q/DQ operations |
| `embedl_sam3dbody_int8.onnx.data` | Encoder external weights (~3.4 GB) |
| `embedl_sam3dbody_int8.pt2` | Encoder as INT8 `torch.export` ExportedProgram |
| `sam3dbody_decoder.onnx` | **Decoder** — SAM-3D-Body promptable decoder + MHR mesh head (body path), features → mesh (fp32; cast to fp16 at engine build) |
| `sam3dbody_faces.npy` | Mesh triangle indices (36 874 × 3), used for rendering |
| `demo3d_trt.py` | **End-to-end TensorRT demo** — image → mesh with both engines, reports engine-only latency |
| `infer_trt.py` | TensorRT encoder inference + latency + PCA-feature demo |
| `infer_pt2.py` | `torch.export` encoder inference + PCA-feature demo |
| `demo_3d.py` | PyTorch reference pipeline demo (upstream code + checkpoint required) |
| `sample_input.png` | Example person crop (from the SAM-3D-Body qualitative gallery) |

## Performance

### End-to-end pipeline comparison (NVIDIA L4, TensorRT 11.1)

> **Environment:** NVIDIA L4 · driver 595.71 / CUDA 13.2 · TensorRT 11.1
> (`tensorrt` cu13 build) · batch 1, 512×512 input · engine execution time
> only (no host↔device copies), 10-iteration warmup, 50 timed iterations,
> all rows measured in a single run. The FP32/FP16 baselines use the same
> encoder with Q/DQ stripped (the underlying fp32 weights); "FP32" is
> TensorRT's default mode for fp32 ONNX (TF32 enabled).

| Pipeline (image → mesh) | Encoder | Decoder | End-to-end | Throughput | Engine size | Peak GPU memory | Speedup |
|---|---|---|---|---|---|---|---|
| FP32 (TRT default, TF32) | 144.2 ms | 25.7 ms | 169.9 ms | 5.9 fps | 4105 MiB | 4610 MiB | 1.0× |
| FP16 | 45.0 ms | 16.2 ms | 61.2 ms | 16.3 fps | 2063 MiB | 2396 MiB | 2.8× |
| **Embedl INT8+FP16 (this model)** | **27.2 ms** | **16.3 ms** | **43.4 ms** | **23.0 fps** | **1425 MiB** | **1718 MiB** | **3.9×** |

Peak GPU memory is the whole-process footprint (engine weights + TensorRT
activation memory + CUDA context) sampled during inference, one pipeline per
process.

With strict IEEE FP32 (TF32 disabled) the baseline is 269.3 ms end-to-end
(encoder 241.7 ms), making the quantized pipeline **6.3×** faster. Run-to-run
variance is ~±2 ms. The standalone encoder benchmark (`infer_trt.py`, 2 s
warmup, 300 iterations) measures 26.2 ms mean / 26.8 ms p95 / 38.2 qps on the
same stack.

### Encoder precision ladder (NVIDIA L4, TensorRT 11.1)

> Same environment and timing method as above. Every row is the identical
> Q/DQ-stripped fp32 encoder rebuilt at a different precision (Q/DQ kept for
> the INT8 row); "strict FP32" has TF32 disabled, "TF32" is TensorRT's
> default mode for fp32 ONNX.

| Encoder precision | Mean latency | p95 | Speedup | What changes |
|---|---|---|---|---|
| FP32 (strict, TF32 off) | 241.7 ms | 244.8 ms | 1.0× | CUDA cores, 3.4 GB weights |
| TF32 (TRT default "fp32") | 138.6 ms | 140.2 ms | 1.7× | GEMMs on tensor cores, weights still fp32 |
| FP16 | 43.0 ms | 45.1 ms | 5.6× | faster tensor cores + half the weight traffic |
| **Embedl Deploy INT8+FP16 (this model)** | **27.2 ms** | **28.1 ms** | **8.9×** | quantized GEMMs, ~0.9 GB weights |

At batch 1 this model is substantially weight-bandwidth-bound, which is why
TF32 (compute-only change) gains less than FP16/INT8 (which also shrink the
weights). INT8 buys 1.6× over the best non-quantized deployment (FP16) with
the mesh-accuracy trade-off quantified under Fidelity.

> **Version notes:**
> - **TensorRT 11.x** (CUDA 13): TRT 11 removed the weakly-typed FP16/INT8
>   builder flags — engines must be built strongly typed, which requires
>   casting the fp32-declared ONNX to FP16 first (otherwise all non-quantized
>   layers run in FP32, ~3.6× slower). `infer_trt.py` and `demo3d_trt.py`
>   handle this cast automatically for both models.
> - **TensorRT 10.x** (CUDA 12 drivers): install `tensorrt-cu12` (a CUDA 13
>   wheel fails with `cudaErrorInsufficientDriver`) and use **TensorRT ≥
>   10.16** — older 10.x kernel selection on Ada is up to 2.6× slower on the
>   INT8 path.

## Fidelity

Because the backbone feeds a 3D-mesh pipeline, the metric that matters is the
recovered **mesh**, not raw features. Running the full SAM-3D-Body pipeline with
the INT8 TensorRT engine vs. the FP32 reference on a real person image:

| Configuration | Recovered-mesh deviation vs FP32 | Backbone-feature rel. diff |
|---|---|---|
| FP16 (TensorRT) | negligible | 0.36 % |
| **Embedl Deploy INT8+FP16 (this model)** | **mean ≈ 10 mm (~0.5 %), max ≈ 67 mm** | (calibration-dependent) |

The decoder + MHR head are robust to the backbone's INT8 quantization noise, so
the 3D body is preserved (the patch-embed conv and LayerNorms are kept in FP16).
Mesh deviation is on a ~1.8 m body. Re-calibrating on a larger, more diverse set
of person crops tightens this further.

The decoder ONNX itself is numerically faithful: as an FP32 TensorRT engine it
matches the upstream PyTorch decoder to ≤ 1e-4 relative on all outputs, and the
default FP16 engine (mesh-rig math kept in FP32) adds only ~2 mm mean vertex
deviation while running 1.7× faster. End-to-end TensorRT mesh deviation stays
dominated by the encoder INT8 kernels (mean ≈ 9 mm vs. the quantized PyTorch
pipeline on the shipped sample).

Quantization sensitivity is pose-dependent: comparing the full quantized
pipeline against the full FP32 pipeline (Q/DQ-stripped encoder + fp32 decoder)
gives mean ≈ 10 mm on a standing figure and mean ≈ 28 mm / max ≈ 63 mm on the
shipped sample's unusual lying pose.

## Deployment boundary

The encoder is the natural `torch.export`-friendly deployment boundary and is
what embedl-deploy quantizes; the decoder + MHR head (~5 % of the compute) are
shipped as a separate FP32 ONNX export covering the body inference path. Feed
`(1, 3, 512, 512)` normalized crops to the encoder and its
`(1, 1280, 32, 32)` features to the decoder (plus the camera/bbox conditioning
inputs — see `demo3d_trt.py` for the exact preprocessing) to get mesh
vertices, 3D/2D keypoints and camera translation.

You must have accepted the upstream gated license at
[`facebook/sam-3d-body-dinov3`](https://huggingface.co/facebook/sam-3d-body-dinov3)
to use this derivative.

## Creating Your Own Optimized Models

Deployment-ready models can be created from any supported base model using
[embedl-deploy](https://deploy.embedl.com), available on PyPI.
This artifact follows the
[SAM3 tutorial](https://docs.embedl.com/embedl-deploy/latest/auto_tutorials/sam3.html)
workflow applied to the SAM-3D-Body backbone.

## License

This model is a derivative of **facebook/sam-3d-body-dinov3**.

| Component | License |
|---|---|
| **Upstream (Meta SAM 3D Body / DINOv3)** | [SAM 3D Body License](https://huggingface.co/facebook/sam-3d-body-dinov3/blob/main/LICENSE) |
| **Optimized components** | [Embedl Models Community Licence v1.0](https://github.com/embedl/embedl-models/blob/main/LICENSE) *(no redistribution as a hosted service)* |

## Contact

- **Enterprise & commercial inquiries:** [models@embedl.com](mailto:models@embedl.com)
- **Technical issues & early access:** [github.com/embedl/embedl-deploy](https://github.com/embedl/embedl-deploy/)

We offer engineering support for on-prem/edge deployments and partner co-marketing opportunities.