FAST_INT4_PIVOT.md

# Fast INT4 Pivot

The original sub-1-bit SVD path is not the right route to FP8-like perplexity. The 90 percent SVD artifact is real, but the repo evaluation notes mark it as broken. The practical pivot is a quality-first low-bit path:

- Use packed row-wise group INT4 weights with per-group scales.
- Evaluate real perplexity through `scripts/eval_quantized.py`.
- Add or export to a real packed INT4 inference backend before claiming throughput gains.

## Why this pivot

Ternary and failed SVD checkpoints are storage experiments. They do not currently provide a fast inference kernel, and their quality is too far from FP8. Groupwise INT4 keeps much more information while still cutting BF16 weight bandwidth by about 3.8x at group size 128.

This does not preserve the original sub-1-bit target. It preserves the project goal that matters more for a usable release: similar quality to FP8 with a path to higher throughput.

## Build

Smoke test the first two matrices:

```powershell
.\.venv\Scripts\python.exe scripts\quantize_groupwise_int4.py `
  --model-dir models\gemma-4-E2B `
  --output quantized\gemma_groupwise_int4_smoke.pt `
  --group-size 128 `
  --max-layers 2
```

Build the full checkpoint when there is enough disk space:

```powershell
.\.venv\Scripts\python.exe scripts\quantize_groupwise_int4.py `
  --model-dir models\gemma-4-E2B `
  --output quantized\gemma_groupwise_int4_g128.pt `
  --group-size 128
```

Expected full artifact size is multi-GB. Check disk first.

## Evaluate

Run reconstruction comparison:

```powershell
.\.venv\Scripts\python.exe scripts\eval_reconstruction.py --model-dir models\gemma-4-E2B
```

Run real perplexity:

```powershell
.\.venv\Scripts\python.exe scripts\eval_quantized.py `
  --quantized-pt quantized\gemma_groupwise_int4_g128.pt `
  --model-dir models\gemma-4-E2B `
  --wikitext data\wiki.test.txt `
  --device cuda `
  --max-length 512 `
  --stride 512
```

Use `--device cpu` only for debugging. It will be slow.

## Throughput

The new checkpoint is packed and kernel-ready, but the current evaluator reconstructs weights into dense tensors for correctness. That proves quality, not speed.

To beat FP8 throughput, the next implementation step is one of:

1. Export this format to an existing INT4 runtime such as a GGUF Q4 family format.
2. Add a CUDA or Triton kernel that multiplies activations by `packed_int4` plus per-group scales without dense dequantization.
3. Route through a maintained INT4 weight-only backend and keep this repo as the quantization/evaluation harness.

Do not claim higher throughput until one of those backends is benchmarked against an FP8 baseline on the target GPU.

## Error-budget residual iteration

The next sub-4-bit candidate is an INT2 base plus a 1-bit residual and a sparse error-budget side channel. The side channel stores the largest remaining residual corrections per group as index plus signed INT4 correction codes.

Build a smoke artifact:

```powershell
.\.venv\Scripts\python.exe scripts\quantize_error_budget_residual.py `
  --model-dir models\gemma-4-E2B `
  --output quantized\gemma_int2_error_budget_g128_k8_smoke.pt `
  --group-size 128 `
  --outliers-per-group 8 `
  --max-layers 4
```

Scan reconstruction tradeoffs:

```powershell
.\.venv\Scripts\python.exe scripts\scan_error_budget_residual.py `
  --model-dir models\gemma-4-E2B `
  --group-size 128 `
  --outliers-per-group 8 `
  --max-layers 4
```

Current local 4-layer scan:

| Method | BPW | Weighted RMSE | Compression vs BF16 |
|--------|-----|---------------|---------------------|
| INT2 base | 2.1250 | 0.010457 | 7.53x |
| INT2 + 1-bit residual | 3.2500 | 0.005689 | 4.92x |
| INT2 + 1-bit residual + g128/k8 side channel | 4.0625 | 0.004022 | 3.94x |
| Groupwise INT4 g128 | 4.1250 | 0.002821 | 3.88x |

The side channel is doing useful work: it roughly halves the weighted MSE of the plain 1-bit residual while staying slightly smaller than g128 INT4. It does not yet beat INT4 reconstruction quality, so it is not the final "beats existing 4-bit" result.

The best below-INT4 g256 variant tested so far was g256/k18:

| Method | BPW | Weighted RMSE | Compression vs BF16 |
|--------|-----|---------------|---------------------|
| INT2 + 1-bit residual + g256/k18 side channel | 4.0313 | 0.004073 | 3.97x |
| Groupwise INT4 g256 | 4.0625 | 0.003100 | 3.94x |

That is smaller than g256 INT4 and better than the binary residual, but still behind INT4 reconstruction quality.

## Mixed budget iteration

Uniform sub-4-bit residual formats still trail INT4. A better next step is mixed budgeting: compute several candidate formats per matrix, then greedily allocate extra bits where they buy the largest weighted error reduction.

Run the mixed-budget scan:

```powershell
.\.venv\Scripts\python.exe scripts\scan_mixed_budget.py `
  --model-dir models\gemma-4-E2B `
  --group-size 128 `
  --outlier-options 4,8 `
  --max-layers 8 `
  --output eval_results\mixed_budget_scan_8layers_g128.json
```

Run a stricter 4.0 BPW target:

```powershell
.\.venv\Scripts\python.exe scripts\scan_mixed_budget.py `
  --model-dir models\gemma-4-E2B `
  --group-size 128 `
  --outlier-options 4,8 `
  --max-layers 8 `
  --target-bpw 4.0 `
  --output eval_results\mixed_budget_scan_8layers_g128_target4p0.json
```

Current 8-matrix scan:

| Method | BPW | Weighted RMSE | Compression vs BF16 |
|--------|-----|---------------|---------------------|
| Uniform groupwise INT4 g128 | 4.1250 | 0.002982 | 3.88x |
| Mixed budget, near-INT4 target | 4.1030 | 0.002982 | 3.90x |
| Mixed budget, 4.0 BPW target | 3.9959 | 0.003163 | 4.00x |

The near-INT4 target found cheap savings by downgrading low-impact matrices while preserving reconstruction almost exactly. The 4.0 BPW target selected one large `int2_error_budget_k4` matrix, one tiny `int2_base` matrix, and INT4 for the rest. This is the most promising direction so far, but it is still a reconstruction result. A real claim needs either activation-weighted calibration or short perplexity evaluation of a material mixed artifact.

Full-surface scan:

```powershell
.\.venv\Scripts\python.exe scripts\scan_mixed_budget.py `
  --model-dir models\gemma-4-E2B `
  --group-size 128 `
  --outlier-options 4,8 `
  --target-bpw 4.0 `
  --output eval_results\mixed_budget_scan_full_g128_target4p0.json
```

Build the selected mixed checkpoint:

```powershell
.\.venv\Scripts\python.exe scripts\quantize_mixed_budget.py `
  --scan-json eval_results\mixed_budget_scan_full_g128_target4p0.json `
  --model-dir models\gemma-4-E2B `
  --output quantized\gemma_mixed_budget_full_g128_target4p0.pt
```

Current full-surface reconstruction result:

| Method | Layers | BPW | Weighted RMSE | Compression vs BF16 | Size |
|--------|--------|-----|---------------|---------------------|------|
| Uniform groupwise INT4 g128 | 316 | 4.1250 | 0.002849 | 3.88x | not built locally |
| Mixed budget full g128 target 4.0 | 316 | 3.9990 | 0.002947 | 4.00x | 948 MB |

The full mixed artifact uses `301` groupwise INT4 matrices, `14` INT2+binary-residual matrices, and `1` INT2+error-budget-k4 matrix. This is a real full-model storage artifact and the best current quality-per-byte result in the repo.

Live Colab CUDA perplexity, 2026-06-29:

| Run | Runtime dtype | BPW | WikiText tokens | Chunks | PPL |
|-----|---------------|-----|-----------------|--------|-----|
| Unquantized `google/gemma-4-E2B` base | BF16 | 16.00 | 292282 | 571 | 108.4542 |
| Mixed budget full g128 target 4.0 | BF16 dense eval after applying quantized weights | 3.9990 | 292282 | 571 | 107.5656 |

This supports a narrow claim: BF16-baseline-equivalent perplexity on this exact Gemma4/WikiText/Colab runner at about 4.00 BPW. It is not an FP16 result, not an FP8 comparison, and not a throughput result because the evaluator reconstructs/applies weights into a normal dense model for correctness.

## Current local smoke result

On the first two Gemma language-model matrices with group size 128:

- Average BPW: 4.1250
- Compression vs BF16: 3.88x
- Weighted MSE: 0.000007
- Weighted RMSE: 0.002638
- Weighted mean absolute error: 0.002103

This is only a smoke result, not full-model perplexity.