Hemmingway-1 NVFP4
NVFP4 build of Altworld/Hemmingway-1 (Qwen3.8-27B fine-tune for everyday writing) for vLLM on NVIDIA Blackwell, which runs NVFP4 natively, made from darrellbest/Hemmingway-1-VL (the same weights in the stock Qwen3.8 layout, vision restored). 27 GB instead of 55.6 GB. All credit for the model goes to Altworld. An uncensored version is at darrellbest/Hemmingway-1-Heretic-NVFP4.
What is quantized
| Part | Precision |
|---|---|
| MLP linear layers, and the attention projections of the full-attention layers | NVFP4, 16-value groups, FP8 scales (W4A4) |
Gated DeltaNet (linear_attn) layers, vision tower, MTP head, embeddings, lm_head, norms |
bf16, unchanged |
Made with llm-compressor 0.13.0 (scheme="NVFP4", calibrated on 64 harmless chat prompts). The
multi-token-prediction weights, which the quantized save drops, were copied back unchanged.
Tested (vLLM 0.30, RTX PRO 6000)
| bf16 | NVFP4 | |
|---|---|---|
| Same top token as bf16 (held-out UltraChat, 24.6K positions) | 100% | 91.7% |
| Perplexity, same text | 5.21 | 5.48 |
| MATH-500 levels 4-5, thinking mode (40) | 32/40 | 32/40 |
get_weather tool call |
correct | correct |
4-bit weights and activations cost more than the GGUF K-quants (Q4_K_M: 95.9% same top token), but reasoning and tool calling hold up. Pick FP8 if it fits.
vllm serve darrellbest/Hemmingway-1-NVFP4
Licence: CC BY-NC 4.0, as Hemmingway-1: non-commercial use, with credit to Hemmingway-1 / Altworld. Commercial use: luka@hemmingway.io.
- Downloads last month
- 50