| --- |
| license: other |
| license_name: sam-license |
| library_name: cortiq |
| base_model: |
| - facebook/sam3 |
| base_model_relation: quantized |
| pipeline_tag: image-segmentation |
| tags: |
| - cmf |
| - cortiq |
| - image-segmentation |
| - sam3 |
| - video-tracking |
| - q8_2f |
| - rust |
| language: |
| - en |
| - ru |
| - zh |
| --- |
| |
| # SAM3 CMF |
|
|
| Native, single-file CMF packaging of |
| [facebook/sam3](https://huggingface.co/facebook/sam3) |
| for the Rust [`cortiq`](https://github.com/infosave2007/cmf) runtime. |
|
|
| SAM3 is Meta's Segment Anything Model 3: promptable instance segmentation |
| with a CLIP text encoder, a ViT trunk with windowed attention, a DETR-style |
| detector with 200 queries, a geometry encoder (boxes and points), a mask |
| decoder, and a video tracker (memory bank). This repository stores every |
| model section plus the vendor configuration and BPE tokenizer in one |
| `sam3-q8_2f.cmf` file: quantized q8_2f for 2-D projections, f16 for |
| convolutions, f32 for norms, geometry tables and the text tower. |
| |
| Requires `cortiq` **v0.8.34** or newer (the release that ships |
| `sam3-pack`, the `segment` CLI and the video tracker). **v0.8.35** |
| adds the CPU and GPU optimizations described under |
| [Performance](#performance). |
| |
| [English](#files) · [Русский](#русский-перевод) · [中文](#中文说明) |
| |
| ## Files |
| |
| | File | Size | SHA-256 | |
| |---|---|---| |
| | `sam3-q8_2f.cmf` | 1,962,174,217 bytes (1.96 GB) | `a7098fd061e0f884b40453fe684a4919113a2942ba9052c46fa208342fbcd3d1` | |
| | `LICENSE` | upstream Sam License | — | |
| | `SHA256SUMS.txt` | checksums | — | |
|
|
| `sam3-q8_2f.cmf` embeds, in one file: the vision trunk (ViT with windowed |
| attention), the SAM2-style FPN neck, the CLIP text tower (BPE tokenizer |
| included), the DETR-style detector, the geometry encoder, the mask decoder |
| and the full video tracker (`trk.*` — memory bank + its mask decoder), plus |
| the vendor configs. |
|
|
| ## Usage |
|
|
| Build or install `cortiq` v0.8.34+, then: |
|
|
| ```bash |
| # Text-prompt segmentation of one image -> mask overlay PNG + boxes/scores JSON |
| cortiq segment sam3-q8_2f.cmf image.png --prompt "red square" |
| |
| # Video (or a directory of PNG frames) -> memory-bank tracker: |
| # a folder of overlay PNGs (mask + stable track id) and tracks.json |
| # with per-frame boxes, scores and id stability |
| cortiq segment sam3-q8_2f.cmf frames/ --prompt "red square" --out overlays/ --json tracks.json |
| cortiq segment sam3-q8_2f.cmf clip.mp4 --prompt "red square" --out overlays/ --json tracks.json |
| ``` |
|
|
| Video input accepts a PNG-frame directory or an mp4/mkv/webm/mov/avi file |
| (frames are extracted with `ffmpeg`). The tracker keeps one stable track id |
| across frames. |
|
|
| To re-pack the model from the upstream `facebook/sam3` export: |
|
|
| ```bash |
| cortiq sam3-pack /path/to/sam3-export --quant q8_2f --out sam3-q8_2f.cmf |
| ``` |
|
|
| ## Performance |
|
|
| Measured on the same synthetic 8-frame video (1008x1008 frames): |
|
|
| | Path | Per frame | vs. baseline | |
| |---|---|---| |
| | CPU, `cortiq` v0.8.34 baseline | 611.9 s | — | |
| | **CPU, `cortiq` v0.8.35** | **91.6 s** | **6.7x faster** | |
|
|
| The v0.8.35 CPU speedup comes from vectorized convolution rows (accumulation |
| order preserved, so results stay bit-exact) and a packed-GEMM attention core |
| (`sgemm_public`, AVX2) replacing ~10^9 per-element dot/axpy calls per frame. |
| Breakdown of where the baseline time went: convolutions 64%, attention core |
| 26%. |
|
|
| v0.8.35 also ships an optional **GPU (Vulkan) attention core** (`gpu_wgpu`): |
| `QK^T` and `A*V` run on the device with an automatic CPU fallback whenever |
| the device refuses a call. Measured on an NVIDIA RTX 2000 Ada: image forward |
| 323 s -> 220 s (-31%), video detection -41%, whole video 402.5 s -> 265 s |
| per frame (-34%) — those GPU figures are against the pre-optimization |
| baseline; with the CPU optimizations and the GPU core combined in v0.8.35 |
| the two effects stack. Toggle: `CMF_SAM3_GPU=0` forces the CPU path. |
|
|
| ## Validation |
|
|
| - CPU parity against the HF reference (`transformers`): 6/6 stages pass |
| (text tower, trunk, encoder, decoder, boxes, masks). |
| - Video tracker parity: memory conditioning cos >= 0.9999 vs the HF |
| reference on a 3-frame fixture. |
| - End-to-end on a synthetic 8-frame video: the object is found in every |
| frame (box within <=1 px of ground truth), one stable track id across all |
| frames, mask IoU = 1.000 per frame. |
|
|
| ## Русский-перевод |
|
|
| Нативная однофайловая CMF-упаковка |
| [facebook/sam3](https://huggingface.co/facebook/sam3) для Rust-рантайма |
| [`cortiq`](https://github.com/infosave2007/cmf). |
|
|
| Один файл `sam3-q8_2f.cmf` (1.96 GB) содержит: vision trunk (ViT с оконным |
| вниманием), FPN-шейку, CLIP text tower с BPE-токенизатором, DETR-детектор, |
| геометрический энкодер, mask decoder и полный видео-трекер (memory bank). |
| Квантование: q8_2f для 2-D проекций, f16 для свёрток, f32 для норм, |
| геометрических таблиц и текстовой башни. |
| |
| Требуется `cortiq` **v0.8.34** или новее; **v0.8.35** добавляет |
| оптимизации из раздела Performance. |
| |
| Производительность (одинаковый синтетический видео-тест из 8 кадров): |
| CPU-путь **611.9 → 91.6 с/кадр (×6.7)**; опциональное GPU-ядро внимания |
| на Vulkan (RTX 2000 Ada) даёт −31% на image-forward и −34% на видео в целом |
| (замеры GPU относительно базовой линии до оптимизаций; в v0.8.35 пути |
| объединены). Выключатель: `CMF_SAM3_GPU=0`. |
| |
| ```bash |
| # Изображение |
| cortiq segment sam3-q8_2f.cmf image.png --prompt "красный квадрат" |
|
|
| # Видео (mp4/mkv/webm/папка PNG) — трекер с одним стабильным track id |
| cortiq segment sam3-q8_2f.cmf frames/ --prompt "красный квадрат" --out overlays/ --json tracks.json |
| ``` |
| |
| Проверки: паритет CPU с эталоном HF — 6/6 стадий; паритет видео-трекера |
| cos >= 0.9999; сквозной тест на синтетическом 8-кадровом видео — объект |
| найден в каждом кадре (бокс <=1 px от эталона), один стабильный track id, |
| IoU маски 1.000. |
| |
| ## 中文说明 |
| |
| [facebook/sam3](https://huggingface.co/facebook/sam3) 的单文件 CMF 打包， |
| 用于 Rust 运行时 [`cortiq`](https://github.com/infosave2007/cmf)。 |
| |
| 单个 `sam3-q8_2f.cmf`（1.96 GB）包含视觉主干（ViT + 窗口注意力）、FPN、 |
| CLIP 文本塔（含 BPE 分词器）、DETR 检测器、几何编码器、掩码解码器和完整 |
| 视频追踪器（记忆库）。量化：2-D 投影 q8_2f、卷积 f16、归一化/几何表/文本 |
| 塔 f32。需要 `cortiq` **v0.8.34** 或更新版本；**v0.8.35** 包含 |
| [Performance](#performance) 中的性能优化：CPU 611.9 → 91.6 秒/帧（×6.7）， |
| 可选的 Vulkan GPU 注意力核心再降约 31%（`CMF_SAM3_GPU=0` 强制 CPU）。 |
| |
| ```bash |
| cortiq segment sam3-q8_2f.cmf image.png --prompt "红色方块" |
| cortiq segment sam3-q8_2f.cmf frames/ --prompt "红色方块" --out overlays/ --json tracks.json |
| ``` |
| |
| ## License |
| |
| Weights are distributed under the upstream |
| [Sam License](https://huggingface.co/facebook/sam3/blob/main/LICENSE) from |
| `facebook/sam3`. This repository only re-packages them; see `LICENSE`. |
| |
