README.md

---
license: apache-2.0
base_model: Nanbeige/Nanbeige4.2-3B
base_model_relation: quantized
pipeline_tag: text-generation
language: [en, zh]
tags: [fp8, nvfp4, quantized, compressed-tensors, vllm, looped-transformer, nanbeige]
model-index:
- name: Nanbeige4.2-3B-FP8-Dynamic
  results:
  - task:
      type: text-generation
    dataset:
      name: GSM8K
      type: gsm8k
    metrics:
    - name: GSM8K (strict, thinking, n=100)
      type: exact_match
      value: 89.0
      verified: false
  - task:
      type: text-generation
    dataset:
      name: GSM8K
      type: gsm8k
    metrics:
    - name: GSM8K (flexible)
      type: exact_match
      value: 93.0
      verified: false
  - task:
      type: text-generation
    dataset:
      name: GPQA
      type: gpqa
    metrics:
    - name: GPQA-diamond (flexible-extract, thinking, 32k budget, n=198)
      type: exact_match
      value: 81.3
      verified: false
  - task:
      type: text-generation
    dataset:
      name: IFEval
      type: ifeval
    metrics:
    - name: IFEval prompt-strict (non-thinking, n=250)
      type: exact_match
      value: 76.4
      verified: false
  - task:
      type: text-generation
    dataset:
      name: MMLU-Pro
      type: mmlu_pro
    metrics:
    - name: MMLU-Pro (non-thinking, 25/category)
      type: exact_match
      value: 62.6
      verified: false
  - task:
      type: text-generation
    dataset:
      name: BBH
      type: bbh
    metrics:
    - name: BBH CoT few-shot (non-thinking)
      type: exact_match
      value: 64.9
      verified: false
  - task:
      type: text-generation
    dataset:
      name: MultiHop-RAG
      type: multihop_rag
    metrics:
    - name: MultiHop-RAG generator-only, gold evidence (n=248)
      type: exact_match
      value: 74.2
      verified: false
---

# Nanbeige4.2-3B-FP8-Dynamic

FP8 W8A8 (dynamic per-token activations) quantization of
[Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) — the looped
transformer (22 layers × `num_loops: 2`). **4.9 GB (from 7.9), +60% batch-1 decode, no
measurable quality loss on any gate I ran.** This is my recommended serving artifact
for this model.

## Why FP8 and not 4-bit for this architecture

The looped architecture executes every layer twice per token, which **compounds
quantization error twice per token**. In my gate battery (below), uniform NVFP4 lost
8 points of GSM8K strict; FP8-dynamic lost nothing — its runtime per-token activation
scales adapt to each loop pass's distinct activation distribution. As far as I can tell
these are the first published FP4/FP8 quantization results for a looped LLM (the only
prior looped-LLM PTQ study, LoopQ arXiv:2605.16343, tested INT formats only).

## Quality evals (RTX 5090, vLLM v0.25.1, 64k ctx, fp8 KV)

Public harnesses first (lm-eval, reproducible), then my closed harnesses
(relative signals within this family). Speed has its own section below.

| bench | mode | bf16 | **FP8-Dynamic** |
|---|---|---|---|
| GSM8K strict (n=100) | thinking | not measured | **89** |
| GSM8K flexible | thinking | not measured | **93** |
| GPQA-diamond (flexible-extract, n=198, 32k budget) | thinking | not measured | **81.3** |
| IFEval prompt-strict (n=250) | non-thinking | not measured | **76.4** |
| IFEval inst-strict | non-thinking | not measured | **83.5** |
| MMLU-Pro (25/category) | non-thinking | not measured | **62.6** |
| BBH CoT few-shot | non-thinking | not measured | **64.9** |
| MultiHop-RAG (gold evidence, n=248) | non-thinking | not measured | **74.2** |
| Blind-judge summarization (closed, 48 articles, same-judge pair) | non-thinking | 4.61 | 4.55 (noise-level Δ) |
| Judged faithfulness (closed) | non-thinking | 5.0 | 4.95-5.0 |
| Judged fabrication / context leaks (closed) | non-thinking | 0% / 0% | 0-2% / 0% |
| Dictation-rewrite taxonomy (closed, 20 cases) | non-thinking | 18/20 | 18/20 |

Judge protocol: blind single-judge, per-article anonymized+shuffled candidates,
strict rubric (faithfulness/coverage/conciseness/coherence/concern-validity).

## Speed (RTX 5090, batch-1, vLLM v0.25.1, 64k ctx, fp8 KV)

Decode tok/s, single stream, per-workload best speculative config:

| workload | spec decode | bf16 | **FP8-Dynamic** |
|---|---|---|---|
| freeform / chat / agent | off | 96 | **154** |
| summarize / RAG (2k+ ctx prompts) | ngram, 8 tok | 136 | **206** |

These are two workload anchors, not an ISL sweep; decode speed shifts with
context length, batch size, and backend. Notes from the serving campaign:

- The looped arch reads every weight twice per token, so weight-compression
  speedups roughly double vs a normal 3B (+60% batch-1 here).
- ngram speculation is workload-dependent: +40-60% on copy-heavy work
  (summarization/RAG), −15-20% on freeform. The rows above use each
  workload's best config.
- Attention backend is context-dependent: FlashInfer (default) wins at 2k+ ctx;
  `VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a
  further +13% in my follow-up runs.


## Performance under load (GuideLLM, real articles, eagle3 head)

Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN,
k=3) on real article prompts. Aggregate throughput scales with concurrency;
inter-token latency stays essentially flat under load.

![Concurrency scaling](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_concurrency.png)

| concurrent streams | aggregate output tok/s | TTFT p50 (ms) | ITL p50 (ms) |
|---|---|---|---|
| 4 | 501 | 166 | 6.2 |
| 8 | 878 | 70 | 6.7 |
| 16 | 1017 | 242 | 7.6 |

Measured with [GuideLLM](https://github.com/vllm-project/guidellm) on an RTX 5090,
vLLM v0.25.1, real multi-kB article prompts, EAGLE-3 v2 (thinking-aware) head. The
server saturates to ~1000 tok/s aggregate at concurrency 16 with per-token latency
near-flat (6.2 to 7.6 ms), so it batches well for multi-user / parallel-agent
workloads. TTFT p50 is queueing-dominated at these request counts, not a decode
signal. Spec acceptance held over the run (154k accepted / 354k drafted, 0.44).

## Serving (vLLM)

The architecture is not yet in upstream vLLM
([PR #49433](https://github.com/vllm-project/vllm/pull/49433) open). Until it merges,
install the bundled out-of-tree plugin (vendored from that PR), then serve normally —
the checkpoint format is auto-detected:

```bash
pip install --no-deps ./vllm_plugin
VLLM_ATTENTION_BACKEND=TRITON_ATTN \
vllm serve <this-repo> --trust-remote-code \
  --max-model-len 65536 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"eagle3","model":"NullSense/Nanbeige4.2-3B-EAGLE3","num_speculative_tokens":3}'
```

Since 2026-07-23 the recommended speculation is my
[EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3),
thinking-aware since the 2026-07-24 retrain: thinking-mode acceptance 0.41 (0.33
before), and +12-41% single-stream decode over the prior head depending on context,
in one config (TRITON_ATTN required with it; FlashInfer + spec drops CUDA graphs to
piecewise on SM120). One measured exception: LONG-document summarization (multi-k-char inputs)
still favors the ngram config (`{"method":"ngram","num_speculative_tokens":8,
"prompt_lookup_max":4,"prompt_lookup_min":2}`): 231 vs 160 output tok/s on 16 real
articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload.

- Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}`
  (+ `preserve_thinking: false` for chat) to disable. Sampling defaults
  (T=0.6/top_p=0.95/top_k=20) ship in `generation_config.json`; use T=1.0 for agentic.
- ngram speculation: +40-60% on summarization/RAG, −15-20% on freeform — drop the
  `--speculative-config` for freeform-heavy serving.
- `--kv-cache-dtype nvfp4` is SM100-only; use `fp8` on consumer Blackwell.

Download just this artifact:

```bash
hf download NullSense/Nanbeige4.2-3B-FP8-Dynamic --local-dir Nanbeige4.2-3B-FP8-Dynamic
```

## Creation

llmcompressor `QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC",
ignore=["lm_head"])`, data-free `oneshot()`. transformers ≥5 users: the bundled
`modeling_nanbeige.py` includes two one-line compat patches vs the original repo
(rope_scaling `"type"` key, `_tied_weights_keys` list→dict) — required for any
transformers-5 loading (llmcompressor, finetuning); vLLM serving does not use them.

## Limitations

- vLLM-only until PR #49433 merges (transformers inference works but is unoptimized
  for the looped arch).
- English+Chinese model; all my evals are English-only.
- Evals are n=100 (GSM8K) / n≈44 (judged summarization) — directional, not leaderboard-grade.


## Which artifact should I choose?

![Quality across benchmarks](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_quality_benches.png)

![Quality vs size](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_quality_vs_size.png)

![Decode speed per workload](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_speed_workloads.png)

| artifact | size | tok/s (freeform / summ) | pick when |
|---|---|---|---|
| [FP8-Dynamic](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) (this repo) | 4.9 GB | 154 / 206 | default: no measured quality loss on any gate. Recommended. |
| [NVFP4-FP8-LoopShield](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield) | 4.5 GB | 159 / 202 | smallest artifact that keeps reasoning at FP8 parity; best JSON reliability. Recommended for tight VRAM. |
| [NVFP4A16](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16) | 3.6 GB | 175 / 220 | fastest; summarization/extraction only (reasoning drops 8 GSM8K points). Not for math/agentic. |
| [EAGLE3 draft](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3) | +1.5 GB | +12-41% decode | add-on speculator for any of the above; thinking-aware retrain (2026-07-24), thinking-mode acceptance 0.41, lossless. Serve with TRITON_ATTN. |

All three serve identically (same plugin, same flags); only the checkpoint differs.
Rule of thumb: this model + 64k fp8-KV context fits in ~7 GB VRAM at FP8, ~6 GB at
NVFP4A16; any 8 GB card runs the full family.

## Full quality matrix vs the bf16 original

| metric | bf16 original | FP8 (this repo) |
|---|---|---|
| rewrite taxonomy (/20, non-think) | 18 | 18 |
| blind-judge summarization (same judging pass) | 4.61 | 4.55 (noise-level delta) |
| JSON parse rate (/48) | 44 | 42 |
| GSM8K, IFEval, MMLU-Pro, BBH, MultiHop-RAG | not measured on bf16 | measured (tables above) |

Where bf16 was measured, FP8 shows no regression; that is why FP8 serves as the
reference for the rest of the family instead of re-running every gate on bf16.
The vendor's published numbers are a different suite entirely (agentic/frontier
benches, thinking mode at a 131k token budget) and don't map onto these rows —
see Benchmark provenance below.

## Links & provenance

- Base model: [Nanbeige/Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B)
- Family: [FP8-Dynamic](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic) · [NVFP4-FP8-LoopShield](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4-FP8-LoopShield) · [NVFP4A16](https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16) · [EAGLE3 draft](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3)
- Arch serving support: [vLLM PR #49433](https://github.com/vllm-project/vllm/pull/49433) (until merged, the bundled `vllm_plugin/` registers it out-of-tree)
- Upstream tooling reports I filed from this work: [llm-compressor#2952](https://github.com/vllm-project/llm-compressor/issues/2952) (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · [llm-compressor#2953](https://github.com/vllm-project/llm-compressor/issues/2953) (AWQ lacks arch mappings)
- Quantized with [llm-compressor](https://github.com/vllm-project/llm-compressor) 0.12.0 (compressed-tensors format)

## Benchmark provenance (ordered: public-harness first, then my closed harnesses)

**Public, reproducible** (lm-eval-harness `local-chat-completions`, exact configs in each row's annotation):
1. [GSM8K](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/gsm8k) — grade-school math, the reasoning gate (thinking mode, n=100, max_tokens 8192)
1. [GPQA-diamond](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/gpqa) — graduate-level science QA (gpqa_diamond_cot_zeroshot, thinking mode, 32k generation budget, full n=198, seed 1234, mean generation ~12.5k tokens, zero budget truncation; flexible-extract because strict-match's answer regex does not fit thinking output; served with the lossless EAGLE-3 head, which does not affect outputs)
2. [IFEval](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/ifeval) — verifiable instruction following (non-thinking, n=250)
3. [MMLU-Pro](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu_pro) — 10-choice knowledge/reasoning (non-thinking, 25/category)
4. [BBH](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/bbh) — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
5. [MultiHop-RAG](https://huggingface.co/datasets/yixuantt/MultiHopRAG) — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)

**Closed/personal harnesses** (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):
- **Blind-judge summarization** — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
- **Rewrite taxonomy** — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
- **JSON parse rate** — structured-output emission over the summarization set. Tests: format discipline under quantization.

## Citation

```bibtex
@misc{peciukonis2026nanbeige42fp8,
  author       = {Pe{\v{c}}iukonis, Matas (NullSense)},
  title        = {Nanbeige4.2-3B quantization family: loop-aware FP8 and NVFP4 quants of a looped transformer},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic},
  note         = {First published classic-bench (GSM8K/IFEval/MMLU-Pro/BBH) numbers for this model; no measured regression vs bf16 on any gate run.}
}
```