AGIcafet Research · interactive report · full pipeline & results

ThaiLLM-30B-NVFP4

NVFP4 (4-bit) quantization of ThaiLLM/ThaiLLM-30B — the Thai/English continued-pretrain of Qwen3-30B-A3B (MoE, 31B total / ~3.3B active) — produced with NVIDIA TensorRT Model Optimizer 0.43.0 and validated with a rigorous paired BF16-vs-NVFP4 evaluation on an NVIDIA DGX Spark (GB10).

• 18.1 GB (from 61.1 GB BF16 — 3.4× smaller); 16.9 GiB GPU-resident
• 2.3–2.5× faster decode on DGX Spark (27 → 63 tok/s single-stream, 69 → 175 tok/s at 4 streams), TTFT 2.0–2.7× lower
• Thai capability statistically unchanged (paired McNemar tests over identical questions)
• W4A4 NVFP4 (E2M1 + FP8 block scales, group 16): attention and all 128 experts/layer quantized; router gates + lm_head kept BF16; KV cache not quantized
• Calibration: 256 Thai Wikipedia + 256 English news documents (512×512 tokens) — half-Thai calibration per multilingual-quantization best practice
• ⚠️ Like the base model, this is a base (non-instruct) model — fine-tune before use as an assistant

Serving

# vLLM (Blackwell GPU, e.g. DGX Spark with nvcr.io/nvidia/vllm:26.05.post1-py3)
vllm serve NatdhanaiPython/ThaiLLM-30B-NVFP4 \
  --quantization modelopt \
  --gpu-memory-utilization 0.70 \
  --attention-backend flashinfer
# vLLM auto-selects native FLASHINFER_CUTLASS NVFP4 MoE kernels on SM121/SM100.
# For extra KV headroom in production add: --kv-cache-dtype fp8

Benchmarks (paired BF16 vs NVFP4, identical questions, seed 0)

Benchmark (n)	BF16	NVFP4	Δ (pt)	Paired p	Verdict
ThaiExam (letter MC, 565)	0.6195	0.6142	−0.53	0.79	not significant
Belebele-TH (900)	0.7700	0.7656	−0.44	—	noise
XCOPA-TH (500)	0.6400	0.6340	−0.60	—	noise
XNLI-TH (2,490)	0.4707	0.4578	−1.29	—	≤1.3σ
Thai MC pooled (3,890)	—	—	−1.00	0.13	not significant
MMLU 5-shot @50/subj (2,850)	0.8168	0.8028	−1.40	0.0017	significant
HellaSwag (10,042)	0.6003	0.5941	−0.62	<0.001	significant, small
ARC-Challenge (1,172)	0.5614	0.5597	−0.17	—	noise
WinoGrande (1,267)	0.7395	0.7285	−1.10	—	≤0.9σ
All MC pooled (19,786)	—	—	−0.81	<1e-4	significant, small

Fidelity / LM quality	BF16	NVFP4
Thai Wikipedia bits/byte ↓	0.2680	0.2822
WikiText-2 bits/byte ↓	0.5660	0.5819
Thai top-1 next-token agreement	—	92.0 % (32,424 positions)
English top-1 agreement	—	88.1 % (13,848 positions)

Interpretation. Pooled over 19,786 paired questions (ThaiExam counted once, via the letter-based template) the quantization cost is real but under one accuracy point, and it concentrates on English reasoning tails — no Thai task degrades significantly. A Thai-fluent 12-domain qualitative review found no systematic degradation (orthography intact, no new repetition behavior); the one caution is verbatim quotation (e.g., legal statutes), where 4-bit weights can paraphrase — use retrieval grounding or BF16 for citation-critical work.

DGX Spark performance (vllm bench serve, median of 3, no speculative decoding)

Workload	BF16	NVFP4	Speedup
1K in / 128 out, 1 stream	27.1 tok/s	63.1 tok/s	2.33×
1K in / 128 out, 4 streams	59.6 tok/s	145.4 tok/s	2.44×
128 in / 1K out, 4 streams	69.4 tok/s	174.8 tok/s	2.52×
TTFT p50 (1K prompt)	326 ms	140 ms	2.32×
Model load	365 s	109 s	3.34×

Thai context vs other local models (same protocol)

Model	Belebele-TH	ThaiExam	Thai bits/byte ↓
ThaiLLM-30B NVFP4 (this repo)	0.766	0.614	0.286
ThaiLLM-30B BF16	0.770	0.619	0.273
Typhoon2.5-30B-A3B (instruct)	0.856	0.604	0.480
SEA-LION v4.5-27B (instruct)	0.843	0.619	0.591
Qwen3.6-27B NVFP4 (instruct)	0.876	0.658	0.296

How it was made

# TensorRT Model Optimizer 0.43.0, examples/llm_ptq/hf_ptq.py
python hf_ptq.py --pyt_ckpt_path ThaiLLM/ThaiLLM-30B \
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 0 \
  --dataset thai_en_calib.jsonl \
  --export_path ThaiLLM-30B-NVFP4

Full reproducible pipeline (calibration builder, fairness-protocol eval scripts, custom letter-scored ThaiExam lm-eval tasks, paired statistics, complete technical report): github.com/spped2000/thaillm-nvfp4-dgx-spark

Roadmap — planned next phases

1. Draft-model speculative decoding: this base has no MTP head, but a same-tokenizer drafter (e.g. Qwen3-0.6B) is expected to add ~1.3–1.6× decode speed losslessly (63 → ~80–100 tok/s on DGX Spark).
2. Thai instruction SFT on the BF16 base → re-quantize with the same recipe → re-run the eval gate.
3. EAGLE-3 head training post-SFT (~1.5–2× additional).
4. FP8 KV cache in production (--kv-cache-dtype fp8) — disabled in this study only for measurement isolation.

Limitations

• Base model: requires instruction fine-tuning before assistant use (the recommended production flow is SFT the BF16 base → re-quantize with this recipe → re-run the eval gate).
• Evaluated with automatic metrics + a 12-domain expert review; no large-scale human evaluation.
• Serving requires NVFP4-capable hardware (NVIDIA Blackwell) for the measured speedups.

Attribution

Base model by the ThaiLLM project (Apache-2.0). Quantization, evaluation, and release by AGIcafet, 15 July 2026.