Spaces:
Running
Running
Add Primitive organization card
Browse filesBrand banner, positioning, public Qwen3.8-27B metrics, models index. Contains no data from private repos.
- .gitattributes +2 -0
- README.md +110 -4
- assets/org-banner.png +3 -0
- assets/primitive-logo.png +0 -0
- assets/qwen-metrics.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/org-banner.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/qwen-metrics.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -1,10 +1,116 @@
|
|
| 1 |
---
|
| 2 |
title: README
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
title: README
|
| 3 |
+
emoji: 🟧
|
| 4 |
+
colorFrom: yellow
|
| 5 |
+
colorTo: red
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
|
| 10 |
+
<p align="center">
|
| 11 |
+
<img src="assets/org-banner.png" alt="Primitive — inference economics for production LLM systems" width="100%">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
<p align="center">
|
| 15 |
+
<img src="https://img.shields.io/badge/focus-inference%20economics-F98424?style=flat-square" alt="Inference economics">
|
| 16 |
+
<img src="https://img.shields.io/badge/runtimes-vLLM%20·%20ONNX-F98424?style=flat-square" alt="vLLM and ONNX">
|
| 17 |
+
<img src="https://img.shields.io/badge/hardware-Blackwell%20·%20A100%20·%20CPU-F98424?style=flat-square" alt="Blackwell, A100, CPU">
|
| 18 |
+
<img src="https://img.shields.io/badge/weights-Apache--2.0-lightgrey?style=flat-square" alt="Apache 2.0">
|
| 19 |
+
</p>
|
| 20 |
+
|
| 21 |
+
---
|
| 22 |
+
|
| 23 |
+
## Serving LLMs well is two problems
|
| 24 |
+
|
| 25 |
+
**Which model should answer this request?** and **how cheaply can that model
|
| 26 |
+
run?** Most teams solve neither — they pin one frontier model to every route and
|
| 27 |
+
pay frontier prices on the easy majority of their traffic.
|
| 28 |
+
|
| 29 |
+
We build the two pieces that fix that, and we publish the measurements.
|
| 30 |
+
|
| 31 |
+
<table>
|
| 32 |
+
<tr>
|
| 33 |
+
<td width="50%" valign="top">
|
| 34 |
+
|
| 35 |
+
### 🔀 Route
|
| 36 |
+
|
| 37 |
+
A small, fast model that reads the prompt **before** you spend anything on it,
|
| 38 |
+
then ranks which LLM is actually worth calling. Runs on CPU next to your API
|
| 39 |
+
server — no GPU, no PyTorch, no extra service to operate.
|
| 40 |
+
|
| 41 |
+
*`llm-router-lite-80m-onnx` — private beta.*
|
| 42 |
+
|
| 43 |
+
</td>
|
| 44 |
+
<td width="50%" valign="top">
|
| 45 |
+
|
| 46 |
+
### 🗜️ Compress
|
| 47 |
+
|
| 48 |
+
Mixed-precision quantization that shrinks a checkpoint and speeds up serving
|
| 49 |
+
**without quietly trading away accuracy** — and that still loads on the GPUs
|
| 50 |
+
you already have, not just the newest ones.
|
| 51 |
+
|
| 52 |
+
*`Qwen3.8-27B-mixed-NVFP4-FP8` — public, open weights.*
|
| 53 |
+
|
| 54 |
+
</td>
|
| 55 |
+
</tr>
|
| 56 |
+
</table>
|
| 57 |
+
|
| 58 |
+
---
|
| 59 |
+
|
| 60 |
+
## Latest release: Qwen3.8-27B, 2.5× smaller, faster, accuracy intact
|
| 61 |
+
|
| 62 |
+
<p align="center">
|
| 63 |
+
<img src="assets/qwen-metrics.png" alt="Qwen3.8-27B mixed NVFP4/FP8: 21 GiB vs 51.8 GiB BF16, accuracy 88.5 vs 88.6 BF16, 1134 tok/s vs 560 tok/s at concurrency 32" width="100%">
|
| 64 |
+
</p>
|
| 65 |
+
|
| 66 |
+
| config | size | accuracy | tok/s @ conc 32 | tok/s @ conc 1 |
|
| 67 |
+
|---|---|---|---|---|
|
| 68 |
+
| BF16 | 51.8 G | 88.6 | 559.6 | 26.3 |
|
| 69 |
+
| Qwen FP8 | 28.8 G | 87.6 | 884.0 | 45.6 |
|
| 70 |
+
| **primitive mixed NVFP4/FP8** | **~21 G** | **88.5** | **1134.5 (2.0×)** | **66.4 (2.5×)** |
|
| 71 |
+
|
| 72 |
+
Measured on a 1,170-sample mixed suite. Accuracy is statistically tied with BF16
|
| 73 |
+
(run-to-run spread ~±0.8). Serves on stock vLLM on **Blackwell** via native
|
| 74 |
+
FP4/FP8 kernels and on **Ampere/A100** via the Marlin W·A16 fallback — 88.1
|
| 75 |
+
accuracy, 790 tok/s @ concurrency 32 on an A100 40 GB. Speculative decoding
|
| 76 |
+
works on the quant: up to **+101%** with a DSpark drafter.
|
| 77 |
+
|
| 78 |
+
```bash
|
| 79 |
+
vllm serve primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8 \
|
| 80 |
+
--reasoning-parser qwen3 \
|
| 81 |
+
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
+
---
|
| 85 |
+
|
| 86 |
+
## How we publish
|
| 87 |
+
|
| 88 |
+
* **Numbers or nothing.** Every figure on our model cards comes from a held-out
|
| 89 |
+
suite and is reproducible from files in the repo. Where a metric is relative,
|
| 90 |
+
we say what it's relative to.
|
| 91 |
+
* **We document what breaks.** Our cards carry the failure modes, not just the
|
| 92 |
+
wins — which head dtype makes an A100 load succeed, which upstream config
|
| 93 |
+
field crashes vLLM, which optimisation we *rejected* and why.
|
| 94 |
+
* **No silent quality loss.** A compression or routing win that changes answers
|
| 95 |
+
isn't a win. When a cheaper option moved outputs too much, we don't ship it —
|
| 96 |
+
and we say so on the card.
|
| 97 |
+
* **Runs on hardware you have.** Ampere fallbacks for the quants, CPU-only for
|
| 98 |
+
the router. Nothing here assumes you just bought a Blackwell rack.
|
| 99 |
+
|
| 100 |
+
---
|
| 101 |
+
|
| 102 |
+
## Models
|
| 103 |
+
|
| 104 |
+
| repo | what it does | status |
|
| 105 |
+
|---|---|---|
|
| 106 |
+
| [`Qwen3.8-27B-mixed-NVFP4-FP8`](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8) | Mixed NVFP4/FP8 weights-only quant of Qwen3.8-27B. vLLM-native, Blackwell + A100, MTP head preserved for speculative decoding. | Public |
|
| 107 |
+
| `llm-router-lite-80m-onnx` | CPU-only router: reads a chat `messages` array, ranks which LLM to call. ONNX, no torch or transformers. | Private beta |
|
| 108 |
+
|
| 109 |
+
All published weights are Apache-2.0. Derivative models carry upstream
|
| 110 |
+
attribution and a `NOTICE` file.
|
| 111 |
+
|
| 112 |
+
<p align="center">
|
| 113 |
+
<br>
|
| 114 |
+
<img src="assets/primitive-logo.png" alt="Primitive" width="34"><br>
|
| 115 |
+
<sub><b>primitive</b> · inference economics for production LLM systems</sub>
|
| 116 |
+
</p>
|
assets/org-banner.png
ADDED
|
Git LFS Details
|
assets/primitive-logo.png
ADDED
|
assets/qwen-metrics.png
ADDED
|
Git LFS Details
|