GetCortex27/Nanbeige4.3-33B-FP8

Nanbeige4.3-33B — FP8 (W8A8, dynamic) by GetCortex27

An FP8 quantisation of Nanbeige/Nanbeige4.3-33B (Apache 2.0, 33B dense, 512K native context, agentic-coding model, thinking on by default) produced with llm-compressor and published with our own fidelity and throughput measurements. We serve this checkpoint in production; the numbers below are what we measured on it, not the base model's self-reported benchmarks.

Recipe

QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])

• Weights: FP8 E4M3, per-channel scales. Activations: FP8, dynamic per-token. No calibration data (data-free).
• llm-compressor 0.15.0, transformers 5.21.0. Format: compressed-tensors — loads natively in vLLM and SGLang.
• Size: 34 GB safetensors (from 66 GB BF16). Fits one 96 GB GPU with 516K tokens of fp8 KV cache at 32K context.
• use_cache is set to true in config.json and generation_config.json (the upstream repo ships false, which makes plain transformers.generate recompute the prefix every step).

Fidelity vs BF16

Teacher-forced next-token distributions on 16 × 512-token chunks (6,000 scored positions) of a fixed mixed corpus — Python, English prose, a claims-adjudication JSON record, and a chat-templated agentic turn containing an XML tool call. Reference = BF16 top-64 log-probs; KL computed on that support, renormalised. Script: kld_eval.py in our repo (to be published).

Metric	Value
Mean KL(BF16 ‖ FP8), top-64	0.0139
p95 KL	0.0505
Top-1 agreement	96.6%
Mean |Δ log p| of the actual next token	0.092 nats

The base model has been reported as unusually sensitive to 4-bit quantisation (plain Q4_K_M ≈ 90% top-1 agreement). At 8-bit the drift is small; we did not find a need for calibration-based FP8.

Caveat: the comparison ran the FP8 weights dequantised in transformers, so it measures weight error. The serving-path smoke tests below ran through SGLang's FP8 kernels.

Throughput (SGLang 0.5.23, one RTX PRO 6000 Blackwell 96 GB, no speculative decoding, thinking off)

Launch: python -m sglang.launch_server --model-path GetCortex27/Nanbeige4.3-33B-FP8 --attention-backend flashinfer --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 2048 --max-running-requests 96 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 32768

Concurrency	Aggregate out tok/s	Per-stream tok/s	TTFT p95
1	29	29	0.07 s
4	104	26	0.11 s
16	355	22	0.16 s
32	607	19	0.11 s
64	1,178	18	0.21 s
96	1,669	17	0.19 s

~137-token prompts, ~165-token outputs, 48–192 requests per level, zero errors in 448 requests. BF16 on the same card: 22 tok/s single-stream, 785 aggregate at 64 concurrent.

Serving notes

• Reasoning parser qwen3, tool-call parser qwen3_coder (the base model recommends an XML tool-call format; this parser handles it). With thinking on and tools present, a weather-tool prompt returned a structured tool_calls entry with the reasoning separated into reasoning_content and counted in usage.
• Thinking is on by default via the chat template; disable per request with chat_template_kwargs: {"enable_thinking": false}.
• Recommended sampling (from the base model card): temperature 1.0 / top_p 0.95 for agentic use; 0.6 / 0.95 for chat.
• vLLM: vllm serve GetCortex27/Nanbeige4.3-33B-FP8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder.

Licence

Apache 2.0, inherited from the base model. Quantisation and measurements by GetCortex27 (Minimorsel Enterprises Pvt. Ltd.). No fine-tuning; weights are a numerical transform of the originals.