README.md

metadata

title: README
emoji: 🟧
colorFrom: yellow
colorTo: red
sdk: static
pinned: false

---

Serving LLMs well is two problems

Which model should answer this request? and how cheaply can that model run? Most teams solve neither — they pin one frontier model to every route and pay frontier prices on the easy majority of their traffic.

We build the two pieces that fix that.

🔀 Route A small, fast model that reads the prompt before you spend anything on it, then ranks which LLM is actually worth calling. Runs on CPU next to your API server — no GPU, no PyTorch, no extra service to operate.

🗜️ Compress Mixed-precision quantization that shrinks a checkpoint and speeds up serving without quietly trading away accuracy — and that still loads on the GPUs you already have, not just the newest ones.

---

Out now: six open-weight quants

Weights-only mixed precision, compressed-tensors, stock vllm serve. Every figure below is on the model's own card, next to the run that produced it.

• 🧮 Qwen3.8-27B — mixed NVFP4/FP8 · 20.7 GiB, 2.5× smaller than BF16 and 28% smaller than Qwen's own FP8, with accuracy tied to BF16 (88.5 vs 88.6 — and the same weights re-scored 87.5 in a second sitting, which is what a tie looks like). Loads on an A100, not just Blackwell, and the in-checkpoint MTP head survives, so speculative decoding still works.
• 🪶 Laguna-XS-2.1 — mixed NVFP4/MXFP8 · 19.3 GiB at 4.96 bits/weight, 3.2× smaller than BF16 and smaller than every official quant — while running faster than all of them at 3,268 tok/s.
• 🐘 Laguna-S-2.1 — mixed NVFP4/MXFP8 · 219 GB of source weights down to 64 GiB at 4.67 bits/weight — it serves from one 96 GiB GPU, the smallest and fastest of the 4-bit field, with +23% single-stream decode over the NVFP4 alternative at a 3 GiB smaller footprint.
• 👁️ Muse-Glimmer-30B — mixed NVFP4 · A vision-language model at 20.4 GiB, 2.40× BF16 throughput and 86.0 against BF16's 86.2. The vision tower stays BF16 — a VLM loses its visual grounding before it loses its words.
• 🦅 Ornith-1.5-35B-A3B — mixed NVFP4/FP8 · A 35 B MoE vision-language model at 22.6 GiB, 3.0× smaller than BF16 and 39% smaller than the official FP8, scoring 91.5 against BF16's 92.0 — the only quant of this model we have measured level with BF16. The official NVFP4 is faster and 0.75 GiB smaller; this build trades that for the closest accuracy to BF16 and a BF16 output head that keeps the Ampere path loadable.
• ⚡ Nemotron-3.5-Lightning-30B-A3B — mixed INT4/INT8 · A MoE+Mamba hybrid at 19.2 GiB and 5.01 bits/weight, 3.3× smaller than BF16 at 2.7× its throughput — and 8.6% smaller and ~7.7% faster than nvidia's own NVFP4, from weights-only round-to-nearest with no calibration data. We make no accuracy claim here: identical weights scored 88.0 and 87.4 on two runs, so that column is a tie.

---

Still in private beta

The router is not published yet. We're working with a small number of teams running real production traffic — the ones who can tell us where the sharp edges are before everyone else finds them.

A good fit if you're: paying per-token to a frontier API on traffic you suspect doesn't need it, serving an open-weights model at meaningful volume, or squeezing a large checkpoint onto GPUs you already own.

Request access at primitive.com →

---

How we publish

• Numbers or nothing. Every figure on our model cards comes from a held-out suite and is reproducible from files in the repo. Where a metric is relative, we say what it's relative to.
• We document what breaks. Our cards carry the failure modes, not just the wins — the dtype choices that matter, the upstream quirks that bite, and the optimisations we tried and rejected.
• No silent quality loss. A compression or routing win that changes answers isn't a win. When a cheaper option moved outputs too much, we don't ship it — and we say so on the card.
• A tie is a tie. Our accuracy deltas sit inside the run-to-run spread of the suite, and we label them as ties rather than dressing them up as wins.
• Runs on hardware you have. Older-generation GPU fallbacks for the quants, CPU-only for the router. Nothing here assumes you just bought the newest rack.

---

Repositories

repo	what it does	status
Qwen3.8-27B-mixed-NVFP4-FP8	Mixed-precision quant of Qwen3.8-27B. vLLM-native, A100 fallback, MTP head preserved for speculative decoding.	Open weights
Laguna-XS-2.1-mixed-NVFP4-MXFP8	Mixed-precision quant of poolside's Laguna-XS-2.1. Smallest and fastest build of it we've measured.	Open weights
Laguna-S-2.1-mixed-NVFP4-MXFP8	Mixed-precision quant of poolside's Laguna-S-2.1. 219 GB of weights onto a single 96 GiB card.	Open weights
Muse-Glimmer-30B-mixed-NVFP4	NVFP4 quant of the Muse-Glimmer-30B VLM, GPTQ-calibrated. Vision tower left at BF16.	Open weights
Ornith-1.5-35B-A3B-mixed-NVFP4-FP8	Mixed-precision quant of Ornith-1.5-35B-A3B, a 35 B MoE VLM. Vision tower and MTP block left at BF16.	Open weights
Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8	INT4/INT8 quant of nvidia's MoE+Mamba hybrid. Smaller and faster than the official NVFP4, no calibration data.	Open weights
llm-router-lite-80m-onnx	CPU-only router: reads a chat messages array, ranks which LLM to call. ONNX, no torch or transformers.	Private beta

Published weights carry their upstream licence — Apache-2.0 for the Qwen and Muse-Glimmer builds, OpenMDW-1.1 for the Laguna and Nemotron builds, MIT for the Ornith build. Derivative models carry upstream attribution and a NOTICE or LICENSE file.

---

Cutting your inference bill?

We'd like to hear what you're serving and where it hurts.
primitive.com

_{primitive · inference economics for production LLM systems}