ornith-ai/Ornith-1.0-35B-GGUF

Text Generation • 35B • Updated Jul 18 • 3.88M • 1.05k

Note MoE (Qwen3_5MoeForConditionalGeneration) and the fastest thing I run: 74.1 tok/s decode, 1102 prefill, 20.1 GiB at Q4_K_M. That is ~6x the decode of any dense 27-31B here at comparable VRAM. On a bandwidth-bound box the ACTIVE parameter count sets decode speed, not the total. Best interactive model on this machine, and the reason to start here.

unsloth/Qwen3.6-35B-A3B-GGUF

Image-Text-to-Text • 35B • Updated Apr 20 • 1.22M • 1.55k

Note MoE, ~3B active of 35B. UD-Q4_K_M: 62.2 tok/s decode, 1014 prefill, 20.7 GiB. Q8_0: 53.5 decode at 34.7 GiB — the higher quant costs 14% decode for 68% more memory. Capacity is cheap on a 64GiB box; bandwidth is what you pay per token.

unsloth/gpt-oss-120b-GGUF

Text Generation • 117B • Updated Aug 25, 2025 • 107k • 289

Note The case for unified memory: 120B resident on a machine with no discrete GPU, via MXFP4 at 58.9 GiB — no 24GB card holds this at any quant. And it is not slow: routing 4 experts per token gives 57.2 tok/s decode, 648 prefill. Faster than every dense 27B here while holding over 3x the parameters.

unsloth/Qwen3.8-27B-GGUF

27B • Updated 26 minutes ago • 4.32M • 1.9k

Note My daily driver, served at 128k context with q8_0 KV cache. Dense 27B: 349 tok/s prefill but only 11.9 decode at 16.9 GiB. Newest and best per-token quality I run — but the MoE models above decode ~6x faster on this hardware. That is a real trade, not a free win.

Qwen/Qwen3.8-27B

Image-Text-to-Text • 28B • Updated 5 days ago • 1.01M • • 11.4k

Note Upstream weights behind the GGUF above — for the model card, licence and evals rather than for running locally.

unsloth/Qwen3.6-27B-GGUF

Image-Text-to-Text • 27B • Updated Apr 22 • 890k • 943

Note Same dense 27B, four quants, one machine: Q3_K_M 13.2GiB/14.4 tok/s decode; Q4_0 14.9GiB/13.4; Q5_K_M 18.5GiB/11.0; Q6_K 20.9GiB/9.5. Prefill stays ~300-340 throughout. Your quant choice is your decode speed and prefill will not warn you — but note the scale: quant moves decode 34%, architecture moves it 6x.

unsloth/gemma-4-31B-it-GGUF

Image-Text-to-Text • 31B • Updated Jul 17 • 501k • 582

Note Largest dense model I run: 279.7 tok/s prefill, 11.2 decode, 19.6 GiB at UD-Q4_K_XL. The slowest decode I measured — the practical floor for dense-at-this-size here.

unsloth/diffusiongemma-26B-A4B-it-GGUF

Image-Text-to-Text • 25B • Updated Jul 4 • 31.5k • 383

Note A diffusion LM rather than autoregressive. One of two models I could NOT benchmark: this llama.cpp build fails to load it, so the prefill/decode split does not even apply in the usual sense. Kept as an honest gap rather than dropped.

Lucebox/DeepSeek-V4-Flash-0731-ROCmFP3

284B • Updated 8 days ago • 8.71k • 18

Note Strix-specific ROCm-FP quant, and the other model I could not measure: 95 GiB exceeds the 64GiB VRAM allocation, so it would stream from disk. The honest ceiling — unified memory is large, not unlimited.

moonshotai/Kimi-K3

Image-Text-to-Text • 2.8T • Updated 23 days ago • 2.29M • • 10.8k

Note Downloaded, not runnable here at full weights. The reference point for what stays out of reach on one workstation, however much unified memory you have.

Note GPTQ. Post-training quantization that made 4-bit routine.

Note AWQ. Activation-aware — protect the salient weights. Explains why not all 4-bit is equal.

Note LLM.int8(). Outlier features are the whole story of why naive quantization breaks.

Note BitNet b1.58. The lower bound on how far this goes, if you train for it from scratch.

Note FlexGen. Offloading throughput when the model genuinely does not fit — the right mental model for latency-tolerant batch jobs.

Note LLM in a flash. Flash-aware inference for memory-limited devices. Increasingly relevant as local becomes on-device.

Note Phi-3. The clearest argument that data quality, not size, sets what a small local model can do.

Note My own measurements on the hardware described above: prefill vs decode throughput and real VRAM cost per model and quant, with the harness included so you can check them.