Nine open-weight builds: consolidated benchmark figure on the pinned 1,370-item protocol

.gitattributes

@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 assets/org-banner.png filter=lfs diff=lfs merge=lfs -text
 assets/qwen-metrics.png filter=lfs diff=lfs merge=lfs -text
 assets/lineup.png filter=lfs diff=lfs merge=lfs -text
+assets/benchmarks.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -15,7 +15,7 @@ pinned: false
   <a href="https://primitive.com"><img src="https://img.shields.io/badge/web-primitive.com-F98424?style=flat-square" alt="primitive.com"></a>
   <img src="https://img.shields.io/badge/focus-inference%20economics-F98424?style=flat-square" alt="Inference economics">
   <img src="https://img.shields.io/badge/runtimes-vLLM%20·%20ONNX-F98424?style=flat-square" alt="vLLM and ONNX">
-  <img src="https://img.shields.io/badge/open%20weights-6%20models-3DD13D?style=flat-square" alt="Six open-weight models">
+  <img src="https://img.shields.io/badge/open%20weights-9%20builds-3DD13D?style=flat-square" alt="Nine open-weight builds">
 </p>
 
 ---
@@ -53,46 +53,75 @@ already have, not just the newest ones.
 
 ---
 
-## Out now: six open-weight quants
+## Out now: nine open-weight builds
 
 <p align="center">
-  <img src="assets/lineup.png" alt="Across the six public quants: 2.5× to 3.4× smaller than BF16; 1.72× to 2.68× BF16 throughput at concurrency 32; accuracy within 0.5 points of BF16 in either direction, inside the ±0.8 run-to-run band — ties, not wins" width="100%">
+  <img src="assets/benchmarks.png" alt="Every published build on one pinned protocol — 1,370 items across fourteen public benchmarks, split into knowledge, tool calls and abstentions. Qwen3.8-Flash-Next leads overall at 90.3; Ornith-1.5 leads on knowledge at 91.7 but sits near the bottom on calls at 74.4; Laguna-S-2.1 is second from last overall yet best of all on abstentions at 81.0" width="100%">
 </p>
 
 Weights-only mixed precision, `compressed-tensors`, stock `vllm serve`. Every
-figure below is on the model's own card, next to the run that produced it.
-
+figure above is on the model's own card, next to the run that produced it.
+
+**One protocol for everything we publish.** 1,370 items across fourteen public
+benchmarks — a 1,170-item knowledge suite over nine of them and a 200-item
+tool-calling suite over five more, scored as calls and abstentions separately and
+never blended. Same decode settings for every model, same 16k budget, thinking
+on. Tool-calling figures are means of repeated runs, because a single run of that
+suite can move three points on the same weights.
+
+* 🚀 **[Qwen3.8-Flash-Next — mixed NVFP4/FP8](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8)**
+  · A **180 B model on one 96 GB GPU**. 360 GB in BF16, two data-center cards in
+  FP8 — this serves on one, with the 51 B n-gram table offloaded to host RAM and
+  no patched runtime. Our highest score anywhere: **90.3 overall, 92.2 knowledge**.
+  The [plain NVFP4 build](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4)
+  is the same deployment; this one puts the attention and GDN projections at FP8
+  and is **13% faster single-stream** for an accuracy tie.
 * 🧮 **[Qwen3.8-27B — mixed NVFP4/FP8](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8)**
-  · 20.7 GiB, **2.5× smaller than BF16** and 28% smaller than Qwen's own FP8, with
-  accuracy tied to BF16 (88.5 vs 88.6 — and the same weights re-scored 87.5 in a
-  second sitting, which is what a tie looks like). **Loads on an A100**, not just
-  Blackwell, and the in-checkpoint MTP head survives, so speculative decoding
-  still works.
+  · 20.7 GiB, **2.5× smaller than BF16** and 28% smaller than Qwen's own FP8.
+  The **shortest answers of any build we measured** (651 tokens) and the highest
+  completion rate. **Loads on an A100**, not just Blackwell — the BF16 `lm_head`
+  is what keeps that path open — and the MTP head survives, so speculative
+  decoding still works.
+* 🦅 **[Ornith-1.5-35B-A3B — mixed NVFP4/FP8](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8)**
+  · A 35 B MoE vision-language model at 22.6 GiB, **3.0× smaller than BF16**, and
+  **the best knowledge score in the lineup at 91.7**. Tool calling is its weak
+  axis, which is why it has a sibling ↓
+* 🛠️ **[Ornith-1.5-35B-A3B — agentic](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-agentic-NVFP4-FP8)**
+  · Same model, same size, calibrated for tool use. Level with BF16 on tool
+  calling and **2.8× steadier run to run** (sd 1.3 against 3.1). We do *not*
+  claim it is more accurate at tool calling than the general build — that gap
+  does not survive the noise. What reproduces is the consistency.
+* ⚡ **[Nemotron-3.5-Lightning-30B-A3B — mixed INT4/INT8](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8)**
+  · A MoE+Mamba hybrid at 19.2 GiB and 5.01 bits/weight, **3.3× smaller than
+  BF16 at 2.7× its throughput**, and **8.6% smaller and ~7.7% faster than
+  nvidia's own NVFP4**. The strongest tool-calling profile here — 85.4 on calls
+  and 70.5 on abstentions, the best balance of the two in the lineup.
+* 👁️ **[Muse-Glimmer-30B — mixed NVFP4](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4)**
+  · A vision-language model at 20.4 GiB, the smallest of its field, tied with
+  BF16 on knowledge. **The vision tower stays BF16** — a VLM loses its visual
+  grounding before it loses its words.
 * 🪶 **[Laguna-XS-2.1 — mixed NVFP4/MXFP8](https://huggingface.co/primitive-ai/Laguna-XS-2.1-mixed-NVFP4-MXFP8)**
   · 19.3 GiB at 4.96 bits/weight, **3.2× smaller than BF16 and smaller than every
-  official quant** — while running faster than all of them at 3,268 tok/s.
+  official quant** — and **2 points of pooled score ahead of the official NVFP4**,
+  almost all of it in the tool-calling half.
 * 🐘 **[Laguna-S-2.1 — mixed NVFP4/MXFP8](https://huggingface.co/primitive-ai/Laguna-S-2.1-mixed-NVFP4-MXFP8)**
   · 219 GB of source weights down to 64 GiB at 4.67 bits/weight — **it serves
-  from one 96 GiB GPU**, the smallest and fastest of the 4-bit field, with +23%
-  single-stream decode over the NVFP4 alternative at a 3 GiB smaller footprint.
-* 👁️ **[Muse-Glimmer-30B — mixed NVFP4](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4)**
-  · A vision-language model at 20.4 GiB, **2.40× BF16 throughput** and 86.0
-  against BF16's 86.2. **The vision tower stays BF16** — a VLM loses its visual
-  grounding before it loses its words.
-* 🦅 **[Ornith-1.5-35B-A3B — mixed NVFP4/FP8](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8)**
-  · A 35 B MoE vision-language model at 22.6 GiB, **3.0× smaller than BF16** and
-  39% smaller than the official FP8, scoring 91.5 against BF16's 92.0 — the only
-  quant of this model we have measured level with BF16. **The official NVFP4 is
-  faster and 0.75 GiB smaller**; this build trades that for the closest accuracy
-  to BF16 and a BF16 output head that keeps the Ampere path loadable.
-* ⚡ **[Nemotron-3.5-Lightning-30B-A3B — mixed INT4/INT8](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8)**
-  · A MoE+Mamba hybrid at 19.2 GiB and 5.01 bits/weight, **3.3× smaller than
-  BF16 at 2.7× its throughput** — and **8.6% smaller and ~7.7% faster than
-  nvidia's own NVFP4**, from weights-only round-to-nearest with no calibration
-  data. We make no accuracy claim here: identical weights scored 88.0 and 87.4
-  on two runs, so that column is a tie.
-
----
+  from one 96 GiB GPU**, the smallest of the 4-bit field, and **the best
+  abstention score of anything we publish (81.0)**: the one that knows when
+  *not* to call a tool.
+* 🧰 **[Qwen3.8-Flash-Next — quantized PLE tables](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant)**
+  · Not a checkpoint — the 51.2 B-parameter n-gram tables that Flash-Next offloads,
+  quantized from 95.4 GB to **28.8 GB** and served memory-mapped from disk.
+  Validated end to end in a **48 GB container**, so a 64 GB-RAM host can serve
+  the 180 B model.
+
+**Read the halves, not just the pooled score.** `overall` weights tool calling at
+14.6% because that is what item count says — which shrinks a tool-calling-only
+divergence to a seventh of its size. Ornith leads the lineup on knowledge and
+sits near the bottom on calls; Laguna-S is second from last overall and best of
+all on abstentions. Abstention is the weak axis on every model anyone has
+measured, ours included, so a build can look strong on one number and still
+over-call in an agent.
 
 ## Still in private beta
 
@@ -132,18 +161,23 @@ squeezing a large checkpoint onto GPUs you already own.
 
 | repo | what it does | status |
 |---|---|---|
-| `Qwen3.8-27B-mixed-NVFP4-FP8` | Mixed-precision quant of Qwen3.8-27B. vLLM-native, A100 fallback, MTP head preserved for speculative decoding. | **Open weights** |
-| `Laguna-XS-2.1-mixed-NVFP4-MXFP8` | Mixed-precision quant of poolside's Laguna-XS-2.1. Smallest and fastest build of it we've measured. | **Open weights** |
-| `Laguna-S-2.1-mixed-NVFP4-MXFP8` | Mixed-precision quant of poolside's Laguna-S-2.1. 219 GB of weights onto a single 96 GiB card. | **Open weights** |
+| `Qwen3.8-Flash-Next-mixed-NVFP4-FP8` | 180 B MoE on one 96 GB GPU, FP8 attention + GDN projections. Our highest-scoring build. | **Open weights** |
+| `Qwen3.8-Flash-Next-NVFP4` | The same single-GPU deployment, plain NVFP4. | **Open weights** |
+| `Qwen3.8-Flash-Next-PLE-quant` | The 51.2 B n-gram tables quantized to 28.8 GB, mmapped from disk. Not a checkpoint. | **Open weights** |
+| `Qwen3.8-27B-mixed-NVFP4-FP8` | Mixed-precision quant of Qwen3.8-27B. vLLM-native, A100 fallback, MTP head preserved. | **Open weights** |
+| `Ornith-1.5-35B-A3B-mixed-NVFP4-FP8` | Mixed-precision quant of Ornith-1.5-35B-A3B, a 35 B MoE VLM. Best knowledge score here. | **Open weights** |
+| `Ornith-1.5-35B-A3B-agentic-NVFP4-FP8` | Same model calibrated for tool use — 2.8× steadier run to run. | **Open weights** |
+| `Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8` | INT4/INT8 quant of nvidia's MoE+Mamba hybrid. Smaller and faster than the official NVFP4. | **Open weights** |
 | `Muse-Glimmer-30B-mixed-NVFP4` | NVFP4 quant of the Muse-Glimmer-30B VLM, GPTQ-calibrated. Vision tower left at BF16. | **Open weights** |
-| `Ornith-1.5-35B-A3B-mixed-NVFP4-FP8` | Mixed-precision quant of Ornith-1.5-35B-A3B, a 35 B MoE VLM. Vision tower and MTP block left at BF16. | **Open weights** |
-| `Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8` | INT4/INT8 quant of nvidia's MoE+Mamba hybrid. Smaller and faster than the official NVFP4, no calibration data. | **Open weights** |
+| `Laguna-XS-2.1-mixed-NVFP4-MXFP8` | Mixed-precision quant of poolside's Laguna-XS-2.1. Smallest build of it we've measured. | **Open weights** |
+| `Laguna-S-2.1-mixed-NVFP4-MXFP8` | Mixed-precision quant of poolside's Laguna-S-2.1. 219 GB of weights onto a single 96 GiB card. | **Open weights** |
 | `llm-router-lite-80m-onnx` | CPU-only router: reads a chat `messages` array, ranks which LLM to call. ONNX, no torch or transformers. | Private beta |
 
-Published weights carry their upstream licence — Apache-2.0 for the Qwen and
-Muse-Glimmer builds, OpenMDW-1.1 for the Laguna and Nemotron builds, MIT for the
-Ornith build. Derivative models carry upstream attribution and a `NOTICE` or
-`LICENSE` file.
+Published weights carry their upstream licence — Apache-2.0 for the Qwen3.8-27B
+and Muse-Glimmer builds, Qwen Community 1.0 for the Flash-Next builds,
+OpenMDW-1.1 for the Laguna and Nemotron builds, MIT for both Ornith builds, and
+Apache-2.0 for the PLE tables. Derivative models carry upstream attribution and a
+`NOTICE` or `LICENSE` file.
 
 ---