docs/quantization.md

# Quantization

## Summary

| Format | Status | Weights | Practical VRAM | Notes |
|---|---|---:|---:|---|
| bfloat16 (native) | Shipped | 19.8 GB | ~22 GB | The published checkpoint |
| 4-bit NF4 (bitsandbytes) | **Validated** | 5.6 GB | **8.1 GB measured** | Every measured result here used this |
| 8-bit (bitsandbytes) | Untested here | 10.1 GB | ~12 GB | Should work; not validated |
| GGUF | **Not produced** | — | — | See below |
| AWQ | **Not produced** | — | — | Script provided, not run |
| GPTQ | **Not produced** | — | — | Script provided, not run |

No quantized artefact has been published for Piko-9b. Nothing in this section claims a
quantization works unless it was run.

## 4-bit NF4 — the validated path

This is what every number in this repository was measured with.

```python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForMultimodalLM.from_pretrained(
    "Dexy2/Piko-9b",
    quantization_config=quantization,
    device_map={"": 0},          # never "auto"
    dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained("Dexy2/Piko-9b")
```

Measured on an RTX 5070 Ti (17.1 GB), weights on NVMe:

| | |
|---|---|
| Cold load | 100–102 s |
| Resident VRAM | 8.1 GB |
| Text generation | 0.9–2.9 s per short answer |
| Image + text | ~4 s per answer |
| 14,429-token prompt | 3.5 s, needle retrieved correctly |
| Custom suite | 65/70 |

**The vision path survives 4-bit.** OCR scored 10/10 and document understanding 10/10 under NF4,
so the projector and vision tower are not visibly damaged by weight quantization. That is a
measurement on this suite, not a general guarantee.

## The rule that matters more than the format

Whatever quantization you choose, **the whole model must stay on one device.**

Piko-9b keeps a recurrent `float32` state across its 24 linear-attention layers. `device_map="auto"`
on an undersized GPU offloads layers to CPU, corrupts that state, and the model emits a single
repeated character with no error raised. Choosing a stronger quantization is the correct response
to insufficient VRAM; enabling offload is not.

```python
device_map={"": 0}     # correct
device_map="auto"      # silently breaks unless the model fits entirely
```

## 8-bit

```python
quantization = BitsAndBytesConfig(load_in_8bit=True)
```

Expected ~10.1 GB of weights. **Not validated here** — no result in this repository was produced
in 8-bit. If you use it, run `evaluation/run_smoke_eval.py` first; its first check exists to catch
a degenerate model.

## GGUF — why there isn't one

`llama.cpp` support depends on the architecture being implemented there. Piko-9b is `qwen3_5`: a
hybrid stack of gated linear attention plus periodic full attention, with mRoPE and a Qwen3.5
vision tower. Converting it means the converter must handle:

* linear-attention layers (`in_proj_qkv`, `in_proj_a/b/z`, `A_log`, `dt_bias`, `conv1d`)
* the `full_attention_interval` layer pattern
* mRoPE with `mrope_section [11, 11, 10]` and `partial_rotary_factor` 0.25
* a separate multimodal projector for the vision tower

A GGUF of an *earlier, text-only* checkpoint in this project's history exists locally
(`Ornith-1.0-9B.BF16.gguf` plus an mmproj file), which shows the family can be converted. That is
**not** this checkpoint and says nothing about whether the published composition converts cleanly.

No GGUF is published, and none should be assumed to work until someone runs
`scripts/validate_quantized_model.py` against it.

## AWQ and GPTQ

Scripts are provided but **were not run**:

* [`scripts/quantize_awq.py`](../scripts/quantize_awq.py)
* [`scripts/quantize_gptq.py`](../scripts/quantize_gptq.py)

Both are activation-aware methods that need a calibration pass. Two specific risks for this
architecture:

1. **Linear-attention projections.** `A_log` and `dt_bias` are not ordinary linear weights.
   Quantizing them like `q_proj` can destabilise the recurrent state. Both scripts exclude them by
   default.
2. **The vision tower.** Most AWQ/GPTQ tooling calibrates on text only. A text-calibrated
   quantizer applied to the vision tower can degrade image handling in a way no text benchmark
   will show. Both scripts leave the vision tower in bf16 by default.

## Validating any quantized build

Never publish a quantized artefact without this:

```bash
python scripts/validate_quantized_model.py \
  --quantized ./piko-9b-awq \
  --reference Dexy2/Piko-9b \
  --output reports/quantization_validation.json
```

It checks loading, degenerate output, text quality against the reference, **image input**, memory,
and throughput. A quantization that passes text checks but breaks the vision path is the most
likely failure mode here, and the easiest to miss.

## Choosing

| You have | Use |
|---|---|
| ≥ 22 GB VRAM | bfloat16, `device_map={"": 0}` |
| 12–22 GB | 4-bit NF4 (validated) or 8-bit (untested) |
| 8–12 GB | 4-bit NF4 |
| < 8 GB | Not supported. Do not use offload |