--- license: other license_name: sam-license library_name: cortiq base_model: - facebook/sam3 base_model_relation: quantized pipeline_tag: image-segmentation tags: - cmf - cortiq - image-segmentation - sam3 - video-tracking - q8_2f - rust language: - en - ru - zh --- # SAM3 CMF Native, single-file CMF packaging of [facebook/sam3](https://huggingface.co/facebook/sam3) for the Rust [`cortiq`](https://github.com/infosave2007/cmf) runtime. SAM3 is Meta's Segment Anything Model 3: promptable instance segmentation with a CLIP text encoder, a ViT trunk with windowed attention, a DETR-style detector with 200 queries, a geometry encoder (boxes and points), a mask decoder, and a video tracker (memory bank). This repository stores every model section plus the vendor configuration and BPE tokenizer in one `sam3-q8_2f.cmf` file: quantized q8_2f for 2-D projections, f16 for convolutions, f32 for norms, geometry tables and the text tower. Requires `cortiq` **v0.8.34** or newer (the release that ships `sam3-pack`, the `segment` CLI and the video tracker). **v0.8.35** adds the CPU and GPU optimizations described under [Performance](#performance). [English](#files) · [Русский](#русский-перевод) · [中文](#中文说明) ## Files | File | Size | SHA-256 | |---|---|---| | `sam3-q8_2f.cmf` | 1,962,174,217 bytes (1.96 GB) | `a7098fd061e0f884b40453fe684a4919113a2942ba9052c46fa208342fbcd3d1` | | `LICENSE` | upstream Sam License | — | | `SHA256SUMS.txt` | checksums | — | `sam3-q8_2f.cmf` embeds, in one file: the vision trunk (ViT with windowed attention), the SAM2-style FPN neck, the CLIP text tower (BPE tokenizer included), the DETR-style detector, the geometry encoder, the mask decoder and the full video tracker (`trk.*` — memory bank + its mask decoder), plus the vendor configs. ## Usage Build or install `cortiq` v0.8.34+, then: ```bash # Text-prompt segmentation of one image -> mask overlay PNG + boxes/scores JSON cortiq segment sam3-q8_2f.cmf image.png --prompt "red square" # Video (or a directory of PNG frames) -> memory-bank tracker: # a folder of overlay PNGs (mask + stable track id) and tracks.json # with per-frame boxes, scores and id stability cortiq segment sam3-q8_2f.cmf frames/ --prompt "red square" --out overlays/ --json tracks.json cortiq segment sam3-q8_2f.cmf clip.mp4 --prompt "red square" --out overlays/ --json tracks.json ``` Video input accepts a PNG-frame directory or an mp4/mkv/webm/mov/avi file (frames are extracted with `ffmpeg`). The tracker keeps one stable track id across frames. To re-pack the model from the upstream `facebook/sam3` export: ```bash cortiq sam3-pack /path/to/sam3-export --quant q8_2f --out sam3-q8_2f.cmf ``` ## Performance Measured on the same synthetic 8-frame video (1008x1008 frames): | Path | Per frame | vs. baseline | |---|---|---| | CPU, `cortiq` v0.8.34 baseline | 611.9 s | — | | **CPU, `cortiq` v0.8.35** | **91.6 s** | **6.7x faster** | The v0.8.35 CPU speedup comes from vectorized convolution rows (accumulation order preserved, so results stay bit-exact) and a packed-GEMM attention core (`sgemm_public`, AVX2) replacing ~10^9 per-element dot/axpy calls per frame. Breakdown of where the baseline time went: convolutions 64%, attention core 26%. v0.8.35 also ships an optional **GPU (Vulkan) attention core** (`gpu_wgpu`): `QK^T` and `A*V` run on the device with an automatic CPU fallback whenever the device refuses a call. Measured on an NVIDIA RTX 2000 Ada: image forward 323 s -> 220 s (-31%), video detection -41%, whole video 402.5 s -> 265 s per frame (-34%) — those GPU figures are against the pre-optimization baseline; with the CPU optimizations and the GPU core combined in v0.8.35 the two effects stack. Toggle: `CMF_SAM3_GPU=0` forces the CPU path. ## Validation - CPU parity against the HF reference (`transformers`): 6/6 stages pass (text tower, trunk, encoder, decoder, boxes, masks). - Video tracker parity: memory conditioning cos >= 0.9999 vs the HF reference on a 3-frame fixture. - End-to-end on a synthetic 8-frame video: the object is found in every frame (box within <=1 px of ground truth), one stable track id across all frames, mask IoU = 1.000 per frame. ## Русский-перевод Нативная однофайловая CMF-упаковка [facebook/sam3](https://huggingface.co/facebook/sam3) для Rust-рантайма [`cortiq`](https://github.com/infosave2007/cmf). Один файл `sam3-q8_2f.cmf` (1.96 GB) содержит: vision trunk (ViT с оконным вниманием), FPN-шейку, CLIP text tower с BPE-токенизатором, DETR-детектор, геометрический энкодер, mask decoder и полный видео-трекер (memory bank). Квантование: q8_2f для 2-D проекций, f16 для свёрток, f32 для норм, геометрических таблиц и текстовой башни. Требуется `cortiq` **v0.8.34** или новее; **v0.8.35** добавляет оптимизации из раздела Performance. Производительность (одинаковый синтетический видео-тест из 8 кадров): CPU-путь **611.9 → 91.6 с/кадр (×6.7)**; опциональное GPU-ядро внимания на Vulkan (RTX 2000 Ada) даёт −31% на image-forward и −34% на видео в целом (замеры GPU относительно базовой линии до оптимизаций; в v0.8.35 пути объединены). Выключатель: `CMF_SAM3_GPU=0`. ```bash # Изображение cortiq segment sam3-q8_2f.cmf image.png --prompt "красный квадрат" # Видео (mp4/mkv/webm/папка PNG) — трекер с одним стабильным track id cortiq segment sam3-q8_2f.cmf frames/ --prompt "красный квадрат" --out overlays/ --json tracks.json ``` Проверки: паритет CPU с эталоном HF — 6/6 стадий; паритет видео-трекера cos >= 0.9999; сквозной тест на синтетическом 8-кадровом видео — объект найден в каждом кадре (бокс <=1 px от эталона), один стабильный track id, IoU маски 1.000. ## 中文说明 [facebook/sam3](https://huggingface.co/facebook/sam3) 的单文件 CMF 打包， 用于 Rust 运行时 [`cortiq`](https://github.com/infosave2007/cmf)。 单个 `sam3-q8_2f.cmf`（1.96 GB）包含视觉主干（ViT + 窗口注意力）、FPN、 CLIP 文本塔（含 BPE 分词器）、DETR 检测器、几何编码器、掩码解码器和完整 视频追踪器（记忆库）。量化：2-D 投影 q8_2f、卷积 f16、归一化/几何表/文本 塔 f32。需要 `cortiq` **v0.8.34** 或更新版本；**v0.8.35** 包含 [Performance](#performance) 中的性能优化：CPU 611.9 → 91.6 秒/帧（×6.7）， 可选的 Vulkan GPU 注意力核心再降约 31%（`CMF_SAM3_GPU=0` 强制 CPU）。 ```bash cortiq segment sam3-q8_2f.cmf image.png --prompt "红色方块" cortiq segment sam3-q8_2f.cmf frames/ --prompt "红色方块" --out overlays/ --json tracks.json ``` ## License Weights are distributed under the upstream [Sam License](https://huggingface.co/facebook/sam3/blob/main/LICENSE) from `facebook/sam3`. This repository only re-packages them; see `LICENSE`.