--- license: apache-2.0 base_model: Nanbeige/Nanbeige4.2-3B base_model_relation: quantized pipeline_tag: text-generation language: [en, zh] tags: [fp8, nvfp4, quantized, compressed-tensors, vllm, looped-transformer, nanbeige] model-index: - name: Nanbeige4.2-3B-FP8-Dynamic results: - task: type: text-generation dataset: name: GSM8K type: gsm8k metrics: - name: GSM8K (strict, thinking, n=100) type: exact_match value: 89.0 verified: false - task: type: text-generation dataset: name: GSM8K type: gsm8k metrics: - name: GSM8K (flexible) type: exact_match value: 93.0 verified: false - task: type: text-generation dataset: name: GPQA type: gpqa metrics: - name: GPQA-diamond (flexible-extract, thinking, 32k budget, n=198) type: exact_match value: 81.3 verified: false - task: type: text-generation dataset: name: IFEval type: ifeval metrics: - name: IFEval prompt-strict (non-thinking, n=250) type: exact_match value: 76.4 verified: false - task: type: text-generation dataset: name: MMLU-Pro type: mmlu_pro metrics: - name: MMLU-Pro (non-thinking, 25/category) type: exact_match value: 62.6 verified: false - task: type: text-generation dataset: name: BBH type: bbh metrics: - name: BBH CoT few-shot (non-thinking) type: exact_match value: 64.9 verified: false - task: type: text-generation dataset: name: MultiHop-RAG type: multihop_rag metrics: - name: MultiHop-RAG generator-only, gold evidence (n=248) type: exact_match value: 74.2 verified: false --- # Nanbeige4.2-3B-FP8-Dynamic FP8 W8A8 (dynamic per-token activations) quantization of [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) — the looped transformer (22 layers × `num_loops: 2`). **4.9 GB (from 7.9), +60% batch-1 decode, no measurable quality loss on any gate I ran.** This is my recommended serving artifact for this model. ## Why FP8 and not 4-bit for this architecture The looped architecture executes every layer twice per token, which **compounds quantization error twice per token**. In my gate battery (below), uniform NVFP4 lost 8 points of GSM8K strict; FP8-dynamic lost nothing — its runtime per-token activation scales adapt to each loop pass's distinct activation distribution. As far as I can tell these are the first published FP4/FP8 quantization results for a looped LLM (the only prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only). ## Quality evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV) Public harnesses first (lm-eval, reproducible), then my closed harnesses (relative signals within this family). Speed has its own section below. | bench | mode | bf16 | **FP8-Dynamic** | |---|---|---|---| | GSM8K strict (n=100) | thinking | not measured | **89** | | GSM8K flexible | thinking | not measured | **93** | | GPQA-diamond (flexible-extract, n=198, 32k budget) | thinking | not measured | **81.3** | | IFEval prompt-strict (n=250) | non-thinking | not measured | **76.4** | | IFEval inst-strict | non-thinking | not measured | **83.5** | | MMLU-Pro (25/category) | non-thinking | not measured | **62.6** | | BBH CoT few-shot | non-thinking | not measured | **64.9** | | MultiHop-RAG (gold evidence, n=248) | non-thinking | not measured | **74.2** | | Blind-judge summarization (closed, 48 articles, same-judge pair) | non-thinking | 4.61 | 4.55 (noise-level Δ) | | Judged faithfulness (closed) | non-thinking | 5.0 | 4.95-5.0 | | Judged fabrication / context leaks (closed) | non-thinking | 0% / 0% | 0-2% / 0% | | Dictation-rewrite taxonomy (closed, 20 cases) | non-thinking | 18/20 | 18/20 | Judge protocol: blind single-judge, per-article anonymized+shuffled candidates, strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity). ## Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV) Decode tok/s, single stream, per-workload best speculative config: | workload | spec decode | bf16 | **FP8-Dynamic** | |---|---|---|---| | freeform / chat / agent | off | 96 | **154** | | summarize / RAG (2k+ ctx prompts) | ngram, 8 tok | 136 | **206** | These are two workload anchors, not an ISL sweep; decode speed shifts with context length, batch size, and backend. Notes from the serving campaign: - The looped arch reads every weight twice per token, so weight-compression speedups roughly double vs a normal 3B (+60% batch-1 here). - ngram speculation is workload-dependent: +40-60% on copy-heavy work (summarization/RAG), −15-20% on freeform. The rows above use each workload's best config. - Attention backend is context-dependent: FlashInfer (default) wins at 2k+ ctx; `VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a further +13% in my follow-up runs. ## Performance under load (GuideLLM, real articles, eagle3 head) Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN, k=3) on real article prompts. Aggregate throughput scales with concurrency; inter-token latency stays essentially flat under load. ![Concurrency scaling](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_concurrency.png) | concurrent streams | aggregate output tok/s | TTFT p50 (ms) | ITL p50 (ms) | |---|---|---|---| | 4 | 501 | 166 | 6.2 | | 8 | 878 | 70 | 6.7 | | 16 | 1017 | 242 | 7.6 | Measured with [GuideLLM](https://github.com/vllm-project/guidellm) on an RTX 5090, vLLM v0.25.1, real multi-kB article prompts, EAGLE-3 v2 (thinking-aware) head. The server saturates to ~1000 tok/s aggregate at concurrency 16 with per-token latency near-flat (6.2 to 7.6 ms), so it batches well for multi-user / parallel-agent workloads. TTFT p50 is queueing-dominated at these request counts, not a decode signal. Spec acceptance held over the run (154k accepted / 354k drafted, 0.44). ## Serving (vLLM) The architecture is not yet in upstream vLLM ([PR #49433](https://github.com/vllm-project/vllm/pull/49433) open). Until it merges, install the bundled out-of-tree plugin (vendored from that PR), then serve normally — the checkpoint format is auto-detected: ```bash pip install --no-deps ./vllm_plugin VLLM_ATTENTION_BACKEND=TRITON_ATTN \ vllm serve --trust-remote-code \ --max-model-len 65536 --kv-cache-dtype fp8 \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \ --speculative-config '{"method":"eagle3","model":"NullSense/Nanbeige4.2-3B-EAGLE3","num_speculative_tokens":3}' ``` Since 2026-07-23 the recommended speculation is my [EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3), thinking-aware since the 2026-07-24 retrain: thinking-mode acceptance 0.41 (0.33 before), and +12-41% single-stream decode over the prior head depending on context, in one config (TRITON_ATTN required with it; FlashInfer + spec drops CUDA graphs to piecewise on SM120). One measured exception: LONG-document summarization (multi-k-char inputs) still favors the ngram config (`{"method":"ngram","num_speculative_tokens":8, "prompt_lookup_max":4,"prompt_lookup_min":2}`): 231 vs 160 output tok/s on 16 real articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload. - Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}` (+ `preserve_thinking: false` for chat) to disable. Sampling defaults (T=0.6/top_p=0.95/top_k=20) ship in `generation_config.json`; use T=1.0 for agentic. - ngram speculation: +40-60% on summarization/RAG, −15-20% on freeform — drop the `--speculative-config` for freeform-heavy serving. - `--kv-cache-dtype nvfp4` is SM100-only; use `fp8` on consumer Blackwell. Download just this artifact: ```bash hf download NullSense/Nanbeige4.2-3B-FP8-Dynamic --local-dir Nanbeige4.2-3B-FP8-Dynamic ``` ## Creation llmcompressor `QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])`, data-free `oneshot()`. transformers ≥5 users: the bundled `modeling_nanbeige.py` includes two one-line compat patches vs the original repo (rope_scaling `"type"` key, `_tied_weights_keys` list→dict) — required for any transformers-5 loading (llmcompressor, finetuning); vLLM serving does not use them. ## Limitations - vLLM-only until PR #49433 merges (transformers inference works but is unoptimized for the looped arch). - English+Chinese model; all my evals are English-only. - Evals are n=100 (GSM8K) / n≈44 (judged summarization) — directional, not leaderboard-grade. ## Which artifact should I choose? ![Quality across benchmarks](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_quality_benches.png) ![Quality vs size](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_quality_vs_size.png) ![Decode speed per workload](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_speed_workloads.png) | artifact | size | tok/s (freeform / summ) | pick when | |---|---|---|---| | [FP8-Dynamic](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) (this repo) | 4.9 GB | 154 / 206 | default: no measured quality loss on any gate. Recommended. | | [NVFP4-FP8-LoopShield](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield) | 4.5 GB | 159 / 202 | smallest artifact that keeps reasoning at FP8 parity; best JSON reliability. Recommended for tight VRAM. | | [NVFP4A16](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16) | 3.6 GB | 175 / 220 | fastest; summarization/extraction only (reasoning drops 8 GSM8K points). Not for math/agentic. | | [EAGLE3 draft](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3) | +1.5 GB | +12-41% decode | add-on speculator for any of the above; thinking-aware retrain (2026-07-24), thinking-mode acceptance 0.41, lossless. Serve with TRITON_ATTN. | All three serve identically (same plugin, same flags); only the checkpoint differs. Rule of thumb: this model + 64k fp8-KV context fits in ~7 GB VRAM at FP8, ~6 GB at NVFP4A16; any 8 GB card runs the full family. ## Full quality matrix vs the bf16 original | metric | bf16 original | FP8 (this repo) | |---|---|---| | rewrite taxonomy (/20, non-think) | 18 | 18 | | blind-judge summarization (same judging pass) | 4.61 | 4.55 (noise-level delta) | | JSON parse rate (/48) | 44 | 42 | | GSM8K, IFEval, MMLU-Pro, BBH, MultiHop-RAG | not measured on bf16 | measured (tables above) | Where bf16 was measured, FP8 shows no regression; that is why FP8 serves as the reference for the rest of the family instead of re-running every gate on bf16. The vendor's published numbers are a different suite entirely (agentic/frontier benches, thinking mode at a 131k token budget) and don't map onto these rows — see Benchmark provenance below. ## Links & provenance - Base model: [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) - Family: [FP8-Dynamic](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) · [NVFP4-FP8-LoopShield](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield) · [NVFP4A16](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16) · [EAGLE3 draft](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3) - Arch serving support: [vLLM PR #49433](https://github.com/vllm-project/vllm/pull/49433) (until merged, the bundled `vllm_plugin/` registers it out-of-tree) - Upstream tooling reports I filed from this work: [llm-compressor#2952](https://github.com/vllm-project/llm-compressor/issues/2952) (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · [llm-compressor#2953](https://github.com/vllm-project/llm-compressor/issues/2953) (AWQ lacks arch mappings) - Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor) 0.12.0 (compressed-tensors format) ## Benchmark provenance (ordered: public-harness first, then my closed harnesses) **Public, reproducible** (lm-eval-harness `local-chat-completions`, exact configs in each row's annotation): 1. [GSM8K](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/gsm8k) — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192) 1. [GPQA-diamond](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/gpqa) — graduate-level science QA (gpqa_diamond_cot_zeroshot, thinking mode, 32k generation budget, full n=198, seed 1234, mean generation ~12.5k tokens, zero budget truncation; flexible-extract because strict-match's answer regex does not fit thinking output; served with the lossless EAGLE-3 head, which does not affect outputs) 2. [IFEval](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/ifeval) — verifiable instruction following (non-thinking, n=250) 3. [MMLU-Pro](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu_pro) — 10-choice knowledge/reasoning (non-thinking, 25/category) 4. [BBH](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/bbh) — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask) 5. [MultiHop-RAG](https://huggingface.co/datasets/yixuantt/MultiHopRAG) — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public) **Closed/personal harnesses** (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores): - **Blind-judge summarization** — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality? - **Rewrite taxonomy** — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing. - **JSON parse rate** — structured-output emission over the summarization set. Tests: format discipline under quantization. ## Citation ```bibtex @misc{peciukonis2026nanbeige42fp8, author = {Pe{\v{c}}iukonis, Matas (NullSense)}, title = {Nanbeige4.2-3B quantization family: loop-aware FP8 and NVFP4 quants of a looped transformer}, year = {2026}, howpublished = {Hugging Face}, url = {https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic}, note = {First published classic-bench (GSM8K/IFEval/MMLU-Pro/BBH) numbers for this model; no measured regression vs bf16 on any gate run.} } ```