Sapiens segmentation (fastforks)

Sapiens (Khirodkar et al. 2024) human body-part segmentation: a ViT pretrained on 300 M human images, fine-tuned with a deconvolution head on Goliath's 28 classes (face, hair, left / right limbs, clothing, lips, teeth, ...); 1024 x 768 input, the Sapiens-Lite TorchScript checkpoints.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/facebookresearch/sapiens at 2cb07227a740.

Author weight repos: https://huggingface.co/facebook/sapiens-seg-0.3b-torchscript, https://huggingface.co/facebook/sapiens-seg-1b-torchscript.

Task: segmentation. Tier: stable. Outputs: logits. Backends: portable, kernels.

Input: one RGB image. Output: per-pixel class labels (logits with logits=True).

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
sapiens-seg-0.3b-goliath@1 sapiens_0.3b_goliath_best_goliath_mIoU_7673_epoch_194_torchscript.pt2 735a9a8d63fe CC-BY-NC-4.0 no
sapiens-seg-1b-goliath@1 sapiens_1b_goliath_best_goliath_mIoU_7994_epoch_151_torchscript.pt2 33bba30f3de8 CC-BY-NC-4.0 no

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: CC-BY-NC-4.0. Licence files: LICENSE.

Training data: MAE pretraining on Humans-300M (Meta-internal, not released); segmentation fine-tuning on Goliath body-part annotations (Meta, not released).

Licence evidence: https://huggingface.co/facebook/sapiens-seg-1b-torchscript/blob/584504252a6d2a27ea6560927811db734ee42332/README.md

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): sapiens-seg-0.3b-goliath 3000x4000 (2.2 GiB peak allocated, 3.2 reserved); sapiens-seg-1b-goliath 3000x4000 (3.8 GiB peak allocated, 5.0 reserved)

Speed and memory

Segmentation, synthetic

model input time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
sapiens-seg-0.3b-goliath 1024x2048 466 173 132 92.6 1.9x 1.4x 2520 / 2724 1643 / 1762 1643 / 1762
sapiens-seg-1b-goliath 1024x2048 1189 354 275 185 1.9x 1.5x 5721 / 5906 3259 / 3384 3259 / 3382

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
sapiens-seg-0.3b-goliath 4.92 (512x512) 1.22 0.96โ€  1.02 (512x512) 0.13โ€  0.09โ€ 
sapiens-seg-1b-goliath 11.33 (512x512) 3.54 2.27โ€  0.57 (512x512) 0.27โ€  0.18โ€ 

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
sapiens-seg-0.3b-goliath 1024x2048 132 ms 118 ms 156 ms 92.6 ms 12.5 s 52.9 s 1643 / 1762 1373 / 1462 1643 / 1762 2465 / 2526
sapiens-seg-1b-goliath 1024x2048 275 ms 263 ms 244 ms 185 ms 18.8 s 84.9 s 3259 / 3384 2986 / 3084 3259 / 3382 5694 / 5742

Fixed-size speed and peak memory, all models

model mode 480x640 720x1280 1080x1920 12 MP 3000x4000
sapiens-seg-0.3b-goliath portable 131.3 ms / 1621 / 1960 132.2 ms / 1627 / 1964 133.7 ms / 1641 / 1950 146.0 ms / 2254 / 3302
sapiens-seg-0.3b-goliath kernels 92.6 ms / 1621 / 1960 93.5 ms / 1627 / 1964 95.2 ms / 1641 / 1950 107.5 ms / 2254 / 3302
sapiens-seg-1b-goliath portable 274.9 ms / 3238 / 3718 275.7 ms / 3245 / 3734 277.0 ms / 3259 / 3730 289.7 ms / 3869 / 5076
sapiens-seg-1b-goliath kernels 185.7 ms / 3238 / 3718 186.6 ms / 3245 / 3734 187.6 ms / 3259 / 3730 199.5 ms / 3869 / 5076

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Any body part counts as person; ground truth = the COCO person masks. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
sapiens-seg-0.3b-goliath COCO val2017 persons, every 10th image, person IoU % 32.60 32.61 32.61 32.62
sapiens-seg-1b-goliath COCO val2017 persons, every 10th image, person IoU % 54.22 54.22 54.22 54.19

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 939adfe8, 2026-11-06.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.05 and p99 <= 0.25 (absolute error of the output logits).

Usage

from fastforks import segmentation

# non-commercial weights: run with FASTFORKS_NONCOMMERCIAL=1 in the environment
model = segmentation.load("sapiens-seg-0.3b-goliath")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.labels  # [H, W] per-pixel class index
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: sapiens-seg-0.3b-goliath, sapiens-seg-1b-goliath.

Kernels backend

segmentation.load("sapiens-seg-0.3b-goliath", mode="kernels") runs our CUDA kernels (fastbb, cudnn_hacc, seg_up). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for fastforks/sapiens

Finetuned
(1)
this model

Collection including fastforks/sapiens