README.md

---
license: apache-2.0
language:
- th
- en
base_model: ThaiLLM/ThaiLLM-30B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- nvfp4
- fp4
- modelopt
- quantized
- vllm
- thai
- qwen3_moe
- dgx-spark
- agicafet
---

<p align="center">
  <b>AGIcafet Research</b> · <a href="https://thaillm.agicafet.com">interactive report</a> · <a href="https://github.com/spped2000/thaillm-nvfp4-dgx-spark">full pipeline &amp; results</a>
</p>

# ThaiLLM-30B-NVFP4

**NVFP4 (4-bit) quantization of [ThaiLLM/ThaiLLM-30B](https://huggingface.co/ThaiLLM/ThaiLLM-30B)** — the Thai/English continued-pretrain of Qwen3-30B-A3B (MoE, 31B total / ~3.3B active) — produced with NVIDIA TensorRT Model Optimizer 0.43.0 and validated with a rigorous paired BF16-vs-NVFP4 evaluation on an NVIDIA DGX Spark (GB10).

- **18.1 GB** (from 61.1 GB BF16 — 3.4× smaller); 16.9 GiB GPU-resident
- **2.3–2.5× faster decode** on DGX Spark (27 → 63 tok/s single-stream, 69 → 175 tok/s at 4 streams), TTFT 2.0–2.7× lower
- **Thai capability statistically unchanged** (paired McNemar tests over identical questions)
- W4A4 NVFP4 (E2M1 + FP8 block scales, group 16): attention **and** all 128 experts/layer quantized; router gates + `lm_head` kept BF16; **KV cache not quantized**
- Calibration: **256 Thai Wikipedia + 256 English news documents** (512×512 tokens) — half-Thai calibration per multilingual-quantization best practice
- ⚠️ Like the base model, this is a **base (non-instruct) model** — fine-tune before use as an assistant

## Serving

```bash
# vLLM (Blackwell GPU, e.g. DGX Spark with nvcr.io/nvidia/vllm:26.05.post1-py3)
vllm serve NatdhanaiPython/ThaiLLM-30B-NVFP4 \
  --quantization modelopt \
  --gpu-memory-utilization 0.70 \
  --attention-backend flashinfer
# vLLM auto-selects native FLASHINFER_CUTLASS NVFP4 MoE kernels on SM121/SM100.
# For extra KV headroom in production add: --kv-cache-dtype fp8
```

## Benchmarks (paired BF16 vs NVFP4, identical questions, seed 0)

| Benchmark (n) | BF16 | NVFP4 | Δ (pt) | Paired p | Verdict |
|---|---|---|---|---|---|
| **ThaiExam** (letter MC, 565) | 0.6195 | 0.6142 | −0.53 | 0.79 | not significant |
| **Belebele-TH** (900) | 0.7700 | 0.7656 | −0.44 | — | noise |
| **XCOPA-TH** (500) | 0.6400 | 0.6340 | −0.60 | — | noise |
| **XNLI-TH** (2,490) | 0.4707 | 0.4578 | −1.29 | — | ≤1.3σ |
| Thai MC pooled (3,890) | — | — | −1.00 | 0.13 | **not significant** |
| MMLU 5-shot @50/subj (2,850) | 0.8168 | 0.8028 | −1.40 | 0.0017 | significant |
| HellaSwag (10,042) | 0.6003 | 0.5941 | −0.62 | <0.001 | significant, small |
| ARC-Challenge (1,172) | 0.5614 | 0.5597 | −0.17 | — | noise |
| WinoGrande (1,267) | 0.7395 | 0.7285 | −1.10 | — | ≤0.9σ |
| **All MC pooled (19,786)** | — | — | **−0.81** | <1e-4 | significant, small |

| Fidelity / LM quality | BF16 | NVFP4 |
|---|---|---|
| Thai Wikipedia bits/byte ↓ | 0.2680 | 0.2822 |
| WikiText-2 bits/byte ↓ | 0.5660 | 0.5819 |
| Thai top-1 next-token agreement | — | 92.0 % (32,424 positions) |
| English top-1 agreement | — | 88.1 % (13,848 positions) |

**Interpretation.** Pooled over 19,786 paired questions (ThaiExam counted once, via the letter-based template) the quantization cost is real but under one accuracy point, and it concentrates on English reasoning tails — **no Thai task degrades significantly**. A Thai-fluent 12-domain qualitative review found no systematic degradation (orthography intact, no new repetition behavior); the one caution is verbatim quotation (e.g., legal statutes), where 4-bit weights can paraphrase — use retrieval grounding or BF16 for citation-critical work.

### DGX Spark performance (vllm bench serve, median of 3, no speculative decoding)

| Workload | BF16 | NVFP4 | Speedup |
|---|---|---|---|
| 1K in / 128 out, 1 stream | 27.1 tok/s | 63.1 tok/s | 2.33× |
| 1K in / 128 out, 4 streams | 59.6 tok/s | 145.4 tok/s | 2.44× |
| 128 in / 1K out, 4 streams | 69.4 tok/s | 174.8 tok/s | 2.52× |
| TTFT p50 (1K prompt) | 326 ms | 140 ms | 2.32× |
| Model load | 365 s | 109 s | 3.34× |

### Thai context vs other local models (same protocol)

| Model | Belebele-TH | ThaiExam | Thai bits/byte ↓ |
|---|---|---|---|
| **ThaiLLM-30B NVFP4 (this repo)** | 0.766 | 0.614 | 0.286 |
| ThaiLLM-30B BF16 | 0.770 | 0.619 | 0.273 |
| Typhoon2.5-30B-A3B (instruct) | 0.856 | 0.604 | 0.480 |
| SEA-LION v4.5-27B (instruct) | 0.843 | 0.619 | 0.591 |
| Qwen3.6-27B NVFP4 (instruct) | 0.876 | 0.658 | 0.296 |

## How it was made

```bash
# TensorRT Model Optimizer 0.43.0, examples/llm_ptq/hf_ptq.py
python hf_ptq.py --pyt_ckpt_path ThaiLLM/ThaiLLM-30B \
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 0 \
  --dataset thai_en_calib.jsonl \
  --export_path ThaiLLM-30B-NVFP4
```

Full reproducible pipeline (calibration builder, fairness-protocol eval scripts, custom letter-scored ThaiExam lm-eval tasks, paired statistics, complete technical report): **[github.com/spped2000/thaillm-nvfp4-dgx-spark](https://github.com/spped2000/thaillm-nvfp4-dgx-spark)**

## Roadmap — planned next phases

1. **Draft-model speculative decoding**: this base has no MTP head, but a same-tokenizer drafter (e.g. Qwen3-0.6B) is expected to add ~1.3–1.6× decode speed losslessly (63 → ~80–100 tok/s on DGX Spark).
2. **Thai instruction SFT** on the BF16 base → re-quantize with the same recipe → re-run the eval gate.
3. **EAGLE-3 head training** post-SFT (~1.5–2× additional).
4. **FP8 KV cache** in production (`--kv-cache-dtype fp8`) — disabled in this study only for measurement isolation.

## Limitations

- Base model: requires instruction fine-tuning before assistant use (the recommended production flow is SFT the BF16 base → re-quantize with this recipe → re-run the eval gate).
- Evaluated with automatic metrics + a 12-domain expert review; no large-scale human evaluation.
- Serving requires NVFP4-capable hardware (NVIDIA Blackwell) for the measured speedups.

## Attribution

Base model by the [ThaiLLM](https://huggingface.co/ThaiLLM) project (Apache-2.0). Quantization, evaluation, and release by **[AGIcafet](https://agicafet.com)**, 15 July 2026.