SMQ-Quantize-0.1 / README.md
CompressedGemma's picture
Update README.md
b6e4f32 verified
|
Raw History Blame Contribute Delete
51 kB
---
license: apache-2.0
---
# Developer note
SMQ alone is not going to produce a good model; it must be paired with the accompanying QAT.
The QAT stage is now a **closed-form algebraic solver** β€” no optimizer, no gradients, no backpropagation. On a 27B block it drives the held-out block-output relative error from ~12.7% (plain SMQ) down to ~3.4% after real Q2_K re-quantization, in a few minutes on a single 12 GiB GPU, with every output row guarded so it can never end up worse than plain SMQ.
# SMQ: Quadratic Q2_K Quantization and Algebraic Quantization-Aware Adaptation
SMQ is a GGUF quantization pipeline built around a closed-form quadratic factorization of the native llama.cpp `Q2_K` representation. Its companion training path, SMQ-QAT, adapts decoder-block weights against the **actual packed-and-decoded SMQ quantizer** rather than a proxy quantizer.
The project has two related but distinct parts:
1. **SMQ quantizer** (`smq_gguf_convert_live.py`) converts eligible GGUF tensors to `Q2_K` using a deterministic, blockwise quadratic fit.
2. **SMQ-QAT** (`smq_qat.py`, driven by `train_smq_qat.py` or its Python `train()` function) performs sequential, blockwise, **algebraic** quantization-aware adaptation. It solves each projection's Q2_K scales/offsets in closed form from output-error Gram matrices, refines the 2-bit codes by output-aware coordinate descent, and guards every output row on both fitting and held-out tokens.
The converter can be used by itself. QAT is an optional adaptation stage before conversion.
> **Terminology:** β€œsource weights” below means the values decoded from the input GGUF. If the input is `Q8_0`, for example, the QAT teacher is the dequantized `Q8_0` modelβ€”not the original pre-quantization FP16/BF16 checkpoint unless that checkpoint is what was supplied as input.
---
## Contents
- [Overview](#overview)
- [Q2_K representation](#q2_k-representation)
- [SMQ quadratic factorization](#smq-quadratic-factorization)
- [Importance matrices](#importance-matrices)
- [Tensor eligibility and coverage](#tensor-eligibility-and-coverage)
- [Installing dependencies](#installing-dependencies)
- [Using the converter](#using-the-converter)
- [SMQ-QAT: the algebraic solver](#smq-qat-the-algebraic-solver)
- [The output-error objective and Gram matrices](#the-output-error-objective-and-gram-matrices)
- [Closed-form scale/offset solve](#closed-form-scaleoffset-solve)
- [Metric-coupled lattice snapping](#metric-coupled-lattice-snapping)
- [Prior annealing (why high-dimensional projections need it)](#prior-annealing-why-high-dimensional-projections-need-it)
- [Output-aware code coordinate descent](#output-aware-code-coordinate-descent)
- [Per-row guards and exact repacking](#per-row-guards-and-exact-repacking)
- [Block-output residual absorption](#block-output-residual-absorption)
- [Block-level revert guard](#block-level-revert-guard)
- [Sequential block adaptation](#sequential-block-adaptation)
- [QAT training example](#qat-training-example)
- [Preflight and export utilities](#preflight-and-export-utilities)
- [Reading the metrics](#reading-the-metrics)
- [Reported experimental observations](#reported-experimental-observations)
- [Memory and performance notes](#memory-and-performance-notes)
- [Validation recommendations](#validation-recommendations)
- [Implementation notes and limitations](#implementation-notes-and-limitations)
---
## Overview
Conventional round-to-nearest (RTN) quantization chooses the representable code nearest to each value under a selected scale and offset grid. SMQ starts from an RTN-derived code assignment, then uses a closed-form least-squares projection to fit the scale and offset parameters to those codes. The fitted parameters are snapped back onto the parameter lattice used by `Q2_K`, and codes are recalculated against that final grid.
The important distinction is that SMQ does not introduce a new GGML tensor type. It writes the standard `Q2_K` 84-byte block layout. The quantization algorithm used to choose the codes and parameters is different; the packed result remains compatible with the existing `Q2_K` representation.
SMQ-QAT adds a second idea: rather than asking only how well the fixed source weights reconstruct, it **re-solves the Q2_K parameters to minimize the block's output error** on calibration activations. Because Q2_K decodes each 16-weight sub-block as `w = s*q - o` with fixed 2-bit codes `q`, a linear layer's output is *linear* in `(s, o)` for fixed codes. That makes the output-error fit a least-squares problem solvable in closed form from Gram matrices β€” no iterative optimizer and no backward pass are required.
### Pipeline at a glance
```text
Input GGUF
β”‚
β”œβ”€β”€ Optional importance matrix
β”‚
β”œβ”€β”€ [Optional] SMQ-QAT (algebraic, no optimizer)
β”‚ β”œβ”€β”€ Build corpus token sequences; hold out a tail split
β”‚ β”œβ”€β”€ Process decoder blocks sequentially
β”‚ β”œβ”€β”€ Fake-quantize projections with real SMQ Q2_K
β”‚ β”œβ”€β”€ Solve (s, o) in closed form from output-error Gram matrices
β”‚ β”œβ”€β”€ Snap to the Q2_K lattice in the output metric
β”‚ β”œβ”€β”€ Refine 2-bit codes by guarded coordinate descent
β”‚ β”œβ”€β”€ Guard every row on fit + held-out tokens; revert block if worse
β”‚ └── Export an adapted, mostly-F16 staging GGUF
β”‚
└── SMQ converter
β”œβ”€β”€ Find every eligible tensor (unless filtered)
β”œβ”€β”€ Factorize 256-weight blocks
β”œβ”€β”€ Pack native Q2_K data
└── Write a complete GGUF
```
---
## Q2_K representation
The converter targets GGML's native `Q2_K` format. Each quantization super-block contains **256 weights in 84 bytes**:
| Component | Size | Meaning |
|---|---:|---|
| `scales` | 16 bytes | One packed scale/minimum pair per 16-value sub-block; each is a 4-bit value |
| `qs` | 64 bytes | 256 two-bit codes |
| `d` | 2 bytes | FP16 super-block scale |
| `dmin` | 2 bytes | FP16 super-block minimum/offset scale |
| **Total** | **84 bytes** | **256 weights** |
The nominal storage cost is therefore:
\[
84 / 256 = 0.328125\text{ bytes/weight} = 2.625\text{ bits/weight}.
\]
This is the storage cost of the quantized blocks, not a guarantee that an entire model file averages exactly 2.625 bits per parameter. GGUF metadata, non-eligible tensors, and any tensors deliberately passed through at their original type affect the whole-file average.
Each 256-value super-block is divided into sixteen 16-value sub-blocks. For sub-block index \(j\), the decoded value is:
\[
\hat{x}_i = d\,ls_j\,q_i - d_{min}\,lm_j,
\]
where:
- \(q_i\in\{0,1,2,3\}\) is the two-bit code;
- \(ls_j\in\{0,\ldots,15\}\) is the four-bit scale multiplier;
- \(lm_j\in\{0,\ldots,15\}\) is the four-bit minimum multiplier;
- \(d\) and \(d_{min}\) are FP16 super-block parameters.
Writing \(s_j = d\,ls_j\) and \(o_j = d_{min}\,lm_j\), the sub-block decode is simply \(w = s_j q - o_j\). This identity is what makes the algebraic QAT fit possible: for fixed codes, the layer output is linear in the per-sub-block scales and offsets.
The converter explicitly packs the scale/minimum nibbles, code planes, and FP16 parameters into the GGML `Q2_K` byte layout. Its self-test checks that packing and its independent decoder reproduce the expected values exactly.
---
## SMQ quadratic factorization
SMQ processes a tensor as a stream of independent 256-weight Q2_K super-blocks. Within each super-block, the parameter fit is performed over its sixteen 16-value sub-blocks. The implementation is closed-form; it does not run iterative coordinate descent or sweep candidate objective values.
### 1. Construct an RTN starting grid
For each 16-value sub-block \(x\), the converter computes a nonnegative minimum offset and a three-level step:
\[
\ell_{min}=\max(-\min(x),0),
\qquad
s=\frac{\max(\max(x)+\ell_{min},0)}{3}.
\]
The maximum step and maximum offset across the sixteen sub-blocks define the super-block parameters:
\[
d=\operatorname{FP16}\!\left(\frac{\max_j s_j}{15}\right),
\qquad
d_{min}=\operatorname{FP16}\!\left(\frac{\max_j \ell_{min,j}}{15}\right).
\]
The local four-bit multipliers are rounded and clipped to `[0, 15]`:
\[
ls_j=\operatorname{clip}\!\left(\operatorname{round}(s_j/d),0,15\right),
\qquad
lm_j=\operatorname{clip}\!\left(\operatorname{round}(\ell_{min,j}/d_{min}),0,15\right).
\]
This produces the initial RTN grid and initial two-bit codes:
\[
q_0=\operatorname{clip}\!\left(\operatorname{round}
\left(\frac{x+d_{min}lm}{dls}\right),0,3\right).
\]
The implementation uses safe denominators for zero-scale cases and stores `d` and `dmin` on the FP16 lattice required by Q2_K.
### 2. Fit scale and offset for the fixed codes
With the initial codes fixed, SMQ fits each 16-value sub-block to the model \(x_i \approx s q_i-o\) by minimizing the quadratic objective
\[
E(s,o)=\sum_i w_i(x_i-sq_i+o)^2.
\]
For uniform weights, \(w_i=1\). When an importance matrix is available, its values supply the weights. Define the weighted moments:
\[
S_w=\sum_i w_i,\quad S_x=\sum_i w_ix_i,\quad
S_q=\sum_i w_iq_i,\quad
S_{qq}=\sum_i w_iq_i^2,\quad
S_{xq}=\sum_i w_ix_iq_i.
\]
The 2-by-2 normal equations have determinant \(\Delta=S_wS_{qq}-S_q^2\). For a non-degenerate sub-block, the closed-form solution is
\[
s=\frac{S_wS_{xq}-S_qS_x}{\Delta},
\qquad
o=\frac{S_qS_{xq}-S_{qq}S_x}{\Delta}.
\]
The Q2_K minimum term cannot represent a negative offset. If the fitted \(o\) is negative, SMQ clamps it to zero and refits the scale on that boundary (\(s=S_{xq}/S_{qq},\ o=0\)). If the system is degenerate, the implementation falls back to the RTN step and minimum for that sub-block.
### 3. Snap the fitted parameters to the Q2_K lattice
The continuous fitted values are not written directly. SMQ converts the maximum fitted scale and offset across the sixteen sub-blocks into FP16 super-block parameters:
\[
d_2=\operatorname{FP16}\!\left(\frac{\max_j s_j}{15}\right),
\qquad
d_{min,2}=\operatorname{FP16}\!\left(\frac{\max_j o_j}{15}\right).
\]
It then recomputes the four-bit multipliers and re-derives the two-bit codes against that final grid. The final codes are selected for the parameters that can actually be stored, not for the unsnapped floating-point fit.
### 4. Apply the per-super-block RTN guard
By default, the converter compares the candidate quadratic reconstruction with the initial RTN reconstruction for each 256-weight super-block, comparing importance-weighted squared error over the entire super-block and keeping the lower-error candidate. This guard makes the **packed choice** no worse than the RTN candidate under the same per-super-block objective. It is not a guarantee about downstream model quality, perplexity, or benchmark accuracy.
The `--no-rtn-metrics` option disables the RTN comparison and guard as well as its associated metrics. In that mode the decomposition is emitted without the RTN fallback safeguard; it is not merely a β€œhide the RTN statistics” option.
### 5. Pack the native block
The selected codes and quantization parameters are packed into 84 bytes per 256 weights using the GGML Q2_K ordering. The output is a normal GGUF with `general.file_type` marked as mostly-Q2_K and the GGML quantization version set by the converter.
---
## Importance matrices
An importance matrix changes the objective used by the local quadratic fit. For each sub-block, SMQ minimizes \(\sum_i w_i(x_i-sq_i+o)^2\) instead of the unweighted sum of squared errors. The same weights are used in the per-super-block RTN guard and in the converter's weighted reconstruction statistics.
The supplied reader supports two forms:
- **Legacy `.dat` imatrix:** per-tensor entries containing per-element importance values.
- **GGUF imatrix:** `general.type = imatrix`, with per-tensor `<tensor>.in_sum2` vectors and optional `<tensor>.counts` values. The input-channel sums are divided by their token count when a positive count is present.
The reader supports either a per-element weight array or a per-input-channel vector. A channel vector is tiled across the output rows as required by the Q2_K block layout. If a tensor is missing from the matrix, that tensor is quantized with uniform weights. The converter reports matrix hits and misses.
An imatrix is optional. Without one, the quadratic fit and the guard use uniform weights. In QAT, the imatrix is also passed to the fake-quant refresh so the final re-quantization matches what the converter will do with the same matrix.
---
## Tensor eligibility and coverage
The converter's default is **all eligible tensors**, not a hand-picked set of named projections. A tensor is eligible when:
1. It has at least two GGML dimensions.
2. No dimension after the first is equal to one (degenerate elementwise tensors are excluded).
3. `ne[0]` is divisible by 256.
4. Its element count is divisible by 256.
The first dimension, `ne[0]`, is the contiguous row width used to align Q2_K super-blocks. The test is based on GGML dimensions, not on the apparent orientation of a NumPy matrix.
When no tensor filters or maximum-tensor limit are supplied, every eligible tensor is selected. This can include eligible embedding and output matrices; the converter does not exclude them merely because of their names. Tensors that fail eligibility are copied through in their original GGML type and byte representation. Typical pass-through tensors include one-dimensional norms and biases, which must remain floating point for the corresponding GGML multiply/add operations.
Selection controls:
- `--tensor NAME`: select an exact tensor name or a substring; repeatable.
- `--tensor-pattern REGEX`: select tensor names matching a regular expression.
- `--max-tensors N`: stop after the first `N` selected eligible tensors; `0` means no limit.
- `--list-tensors`: list eligible tensors and report tensors that will be skipped for shape/eligibility reasons.
If selection filters are supplied, eligible tensors not selected are copied unchanged.
---
## Installing dependencies
### Converter
The converter requires NumPy and `gguf-py` from the llama.cpp checkout used by the runtime:
```bash
pip install numpy
pip install -e /path/to/llama.cpp/gguf-py
```
The converter can use PyTorch CPU reductions as an optional accelerator. If PyTorch is not importable, the NumPy reduction paths are used.
### QAT
The QAT module requires PyTorch, NumPy, GGUF-Py, Transformers, and Safetensors, plus the tokenizer package appropriate for the model tokenizer:
```bash
pip install torch numpy transformers safetensors
pip install -e /path/to/llama.cpp/gguf-py
# BPE tokenizers (for example, Qwen/Mistral families)
pip install tokenizers
# SentencePiece tokenizers (for example, Llama)
pip install sentencepiece
```
CUDA is used when available and selected. The algebraic fit runs its Gram statistics and coordinate descent on the training device and uses BF16 autocast only for the block forwards that collect activations and evaluate block output. There is **no optimizer state and no backward pass**, so VRAM use is dominated by the weights, the decoded fake-quant cache, and the per-group Gram matrices.
---
## Using the converter
Assuming the file is named `smq_gguf_convert_live.py`:
### Quantize every eligible tensor
```bash
python smq_gguf_convert_live.py model-f16.gguf \
--output-gguf model-smq-q2k.gguf
```
### Quantize using an importance matrix
```bash
python smq_gguf_convert_live.py model-f16.gguf \
--imatrix model.imatrix.gguf \
--output-gguf model-smq-q2k.gguf \
--out reports/model-smq
```
This writes a complete GGUF and benchmark reports at `reports/model-smq.csv` and `reports/model-smq.json`.
### Inspect tensor eligibility first
```bash
python smq_gguf_convert_live.py model.gguf --list-tensors
```
### Quantize selected tensors only
```bash
# One tensor or substring
python smq_gguf_convert_live.py model.gguf \
--tensor blk.12.ffn_up.weight \
--output-gguf selected-q2k.gguf
# Regular-expression selection
python smq_gguf_convert_live.py model.gguf \
--tensor-pattern 'blk\\.(0|1|2)\\.' \
--output-gguf early-blocks-q2k.gguf
```
Unselected tensors are copied unchanged.
### Save per-tensor reconstructions
```bash
python smq_gguf_convert_live.py model.gguf \
--output-gguf model-smq-q2k.gguf \
--save-recon reconstructions/
```
The converter writes per-tensor float32 `.npy` reconstructions for analysis. See the metric caveat under [Implementation notes and limitations](#implementation-notes-and-limitations) when interpreting the current guarded-output reporting path.
### Useful converter flags
| Flag | Default | Purpose |
|---|---:|---|
| `--output-gguf`, `--out-gguf` | none | Write a complete Q2_K GGUF |
| `--list-tensors` | off | List eligible and skipped tensors |
| `--tensor` | none | Restrict selection by exact name or substring; repeatable |
| `--tensor-pattern` | none | Restrict selection by regular expression |
| `--imatrix` | none | Apply per-element or per-channel importance weights |
| `--max-tensors` | `0` | Maximum selected eligible tensors; zero means all |
| `--chunk-blocks` | `32768` | Number of 256-weight blocks processed per chunk |
| `--progress-interval` | `5` seconds | Heartbeat interval for long stages |
| `--save-recon` | none | Save float32 reconstructions as `.npy` files |
| `--out` | `smq_gguf_bench` | CSV/JSON report prefix |
| `--quiet` | off | Reduce progress output |
| `--no-rtn-metrics` | off | Disable RTN comparison and guard; emits unguarded decomposition |
| `--no-metrics` | off | Skip residual metric accumulation |
| `--self-test` | off | Run packing/factorization self-tests |
Run `python smq_gguf_convert_live.py --help` for the CLI help generated by the installed file.
---
## SMQ-QAT: the algebraic solver
SMQ-QAT makes each decoder block learn to operate with SMQ's actual Q2_K representation, but it does so **algebraically**. There is no AdamW, no learning rate, no gradient, and no optimizer state. Each projection's Q2_K scales and offsets are solved in closed form from output-error normal equations, snapped to the storable lattice in the output metric, and the 2-bit codes are refined by output-aware coordinate descent. Every candidate is guarded per output row on both the fitting tokens and a held-out split.
### What gets fake-quantized and trained
For each decoder block, the trainer identifies eligible projection matrices that the converter will emit as Q2_K and attaches a fake-quant parametrization to each corresponding linear weight. The parametrization computes the actual SMQ Q2_K quantize/decode round trip; it does not substitute a different low-bit quantizer or an approximate codebook.
The forward weight is
\[
W_{eff}=W+\big(Q(W)-W\big)_{detach},
\]
where `Q(W)` is the decoded weight reconstructed from the packed Q2_K payload. During the algebraic fit the parametrization is deferred (`_defer=True`) so forwards use the master weight directly while Gram statistics are collected; the fitted decoded lattice is installed at the end of the block.
The quantizer itself is a NumPy engine running on CPU. The full-precision tensor is transferred to CPU for quantization, and the decoded tensor is returned to the training device or kept in a pinned CPU cache when offloading is enabled.
### What is not adapted
The algebraic fit adapts decoder-block **projection matrices only**. One-dimensional block auxiliary parameters (normalization weights, biases) are **not** co-adapted; they pass through unchanged. The trainer does not train the model-level token embedding matrix or the output/LM-head matrix.
This is separate from converter coverage:
- The final converter can still quantize `token_embd.weight` and `output.weight` if they meet the generic eligibility rule.
- The QAT trainer uses the embedding matrix to create the initial corpus activations, but does not optimize it.
- The output head is not part of the block-output target and is not adapted.
Therefore β€œall eligible matrices are quantized” and β€œall eligible matrices receive QAT adaptation” are not the same claim.
### Block targets and the held-out split
For each block, the trainer computes target outputs using that block's source weights and the current incoming activation buffer. The target is the block output captured **before the final model norm**, computed in FP32 without autocast so the distillation signal is exact.
A tail of the corpus is held out for guarding and evaluation: `n_eval = min(8, max(1, n_seq // 4))` sequences. The remaining sequences are the fitting split. A block needs at least two sequences (one to fit, one to hold out); with only one sequence the block is left at plain SMQ and a message is logged. The held-out split is what makes the per-row and per-block guards meaningful rather than self-referential.
The source GGUF may itself contain quantized weights (for example, Q8_0). In that case the block teacher is the source GGUF decoded into the partial HF modelβ€”not an unseen original FP16 checkpoint.
---
## The output-error objective and Gram matrices
For a projection with weight rows \(w_r\in\mathbb{R}^d\), student input \(X_q\in\mathbb{R}^{N\times d}\), and target output \(Y\in\mathbb{R}^{N\times m}\), the per-row output error is
\[
e_r(w_r)=\sum_{t=1}^{N}\big(X_{q,t}\,w_r - Y_{t,r}\big)^2
= w_r^\top C\,w_r - 2\,w_r^\top g_r + yy_r,
\]
where the **Gram matrices** are accumulated by streaming over tokens (the activations themselves are never stored for the fit):
\[
C = X_q^\top X_q\in\mathbb{R}^{d\times d},\qquad
G = X_q^\top Y\in\mathbb{R}^{d\times m},\qquad
yy_r=\sum_t Y_{t,r}^2.
\]
The solve cost therefore does not depend on the token count. Held-out error is evaluated explicitly on the held-out activations \(X_v, Y_v\) via `_row_err_val`.
### Input groups and execution order
Projections that consume the very same input tensor (for example `q/k/v`, or `gate/up`) share one Gram matrix and are solved together as a **group**. Groups are discovered by registering forward pre-hooks and ordering modules by execution. A later group sees the already-quantized outputs of the earlier ones: its student input \(X_q\) is captured with the earlier groups' fitted weights installed, while its target \(Y=X_{fp}W_{fp}^\top\) comes from the pure full-precision block. This is what lets downstream projections compensate for upstream quantization error.
---
## Closed-form scale/offset solve
For fixed codes \(q\), each row's weights are \(w_{r,j,a}=s_{r,j}q_{r,j,a}-o_{r,j}\), linear in the \(2J\) unknowns \(\theta_r=(s_{r,1..J},o_{r,1..J})\) where \(J=d/16\). Substituting into \(e_r\) gives a quadratic form \(\theta_r^\top A_r\theta_r-2b_r^\top\theta_r+\text{const}\), where \(A_r\) is built from \(C\) contracted with the codes and \(b_r\) from \(G\). `_ls_fit_gram` solves the resulting \(2J\times 2J\) normal equations per row:
- The system is **Jacobi-scaled** to a unit diagonal first; the proximal `ridge` is relative to that unit diagonal and pulls undetermined directions toward the current scales (so dead input channels never zero a weight).
- Rows are processed in chunks sized from free GPU memory; a chunk that still runs out of memory is retried with half the rows.
- The intermediate contraction tensor is built in slices of sub-blocks and written directly into \(A_r\), so even a 17,408-wide projection (`ffn_down`, \(J=1088\), \(2J=2176\)) fits without materializing a full-size temporary.
- Normal equations run at `float32_matmul_precision="highest"`; Gram statistics and coordinate descent may use TF32.
The continuous solution \((s_{ls},o_{ls})\) is then snapped to the storable Q2_K lattice (next section).
---
## Metric-coupled lattice snapping
A naive snap rounds each sub-block scale independently in weight space. That throws away the output metric and lets a single low-energy, noisy sub-block dictate the super-block scale, crushing the 4-bit resolution of the important sub-blocks. The solver instead snaps **in the Gram metric** using each sub-block's \(2\times2\) block \((A_{ss},A_{so},A_{oo})\):
1. **Schur-complement scale weights.** The effective curvature for snapping \(s_j\) (with \(o_j\) free to adjust) is \(w_s = A_{ss}-A_{so}^2/A_{oo}\). A small grid of super-block scale multipliers around \(\max(s)/15\) is searched under these weights.
2. **Conditional offset compensation.** When \(s_j\) is snapped to \(s_{\text{snap},j}\), the rounding residual \(\delta s_j=s_{\text{snap},j}-s_{ls,j}\) shifts the sub-block along \(q_j\). The optimal offset shifts by \(-(A_{so}/A_{oo})\,\delta s\) to cancel that mean shift, so the target offset is adjusted to \(O_{adj}=O_{ls}-(A_{so}/A_{oo})(S_{\text{snap}}-S_{ls})\) **before** snapping \((d_{min},lm)\) under \(w_o=A_{oo}\).
3. **Converter-canonical super-scales.** The snapped super-block scale is normalized to \(d=\operatorname{FP16}(\max_j s_{\text{snap},j}/15)\) so that \(\max_j ls_j=15\) (and likewise for \(d_{min}\)). This is exactly the grid the converter's `factorize_q2k` reconstructs from min/max, which is what makes the fitted lattice survive the final re-quantization losslessly (see [Per-row guards and exact repacking](#per-row-guards-and-exact-repacking)).
---
## Prior annealing (why high-dimensional projections need it)
The empirical Gram matrix \(C=X_q^\top X_q\) is estimated from a finite calibration set. For wide, low-rank activations β€” especially the MLP projections `gate/up/down`, whose input dimension can exceed the token count β€” \(C\) has a large nullspace and many tiny eigenvalues. Solving with a tiny ridge lets the fit chase sample noise in those directions: training error collapses while held-out error explodes, and the held-out guard then rejects every row.
The fix is an **empirical-Bayes / Tikhonov prior toward the true teacher weights**, applied to the Gram matrices themselves:
\[
C_\gamma = C + \gamma\,\operatorname{diag}(C),\qquad
G_\gamma = G + \gamma\,\operatorname{diag}(C)\,W_{fp}^\top.
\]
This is equivalent to augmenting the calibration tokens with synthetic orthogonal tokens of energy \(\gamma C_{ii}\) whose teacher output is exact. In high-energy directions the real data dominates; in the nullspace the fit stays anchored to \(W_{fp}\), weighted by channel importance.
The prior strength \(\gamma\) is **annealed from strong to weak** across the rounds:
- \(d\le 8192\): \(\gamma\in(2.0,\ 0.75,\ 0.15,\ 0.0)\) over `ls_iters` rounds.
- \(d>8192\) (e.g. `ffn_down`): \(\gamma\in(3.0,\ 1.0)\) over a reduced round count.
Crucially, **every candidate is judged on the true, unregularized** training error (`_row_err_gram` with the original \(C,G\)) **and the held-out error** (`_row_err_val`). The prior only shapes the proposal; the per-row guard decides acceptance. So each row automatically lands at its own bias–variance optimum: rows that benefit from drifting away from \(W_{fp}\) (e.g. an attention output projection compensating upstream error) take the weak-prior steps, while rows that would overfit take the strong-prior steps. The prior is added to and removed from \(C\) and \(G\) in place, so it costs no extra VRAM.
---
## Output-aware code coordinate descent
After each scale solve, the 2-bit codes are refined by `_cd_codes`, which minimizes the same output-error quadratic directly over \(q\in\{0,1,2,3\}\):
- For a row with weights \(w\) and fixed scales/offsets, moving weight \(i\) to lattice value \(c\) changes the error by \(2(c-w_i)h_i + C_{ii}(c-w_i)^2\) where \(h=Cw-g\). The minimizing lattice point is nearest to \(t_i=w_i-h_i/C_{ii}\).
- Weights are visited 16 at a time (one sub-block). Inside a sub-block the gradient \(h\) is updated exactly column by column; the rest of the row receives one GEMM per sub-block. This keeps the update output-aware rather than weight-space nearest.
- `cd_damp` shrinks every step (denominator \(C_{ii}(1+\text{damp})\)) as a trust region; `cd_ridge` adds a proximal term toward the starting weights, lifting small eigenvalues the way GPTQ's `percdamp` does.
- Each sweep is **guarded per row** on fit + held-out error; a sweep that improves no row stops early. The first round uses `cd_sweeps` sweeps, later rounds use one.
When `cd_sweeps=0`, the solver falls back to weight-space nearest-code re-derivation (`_rederive_codes`), still guarded per row.
For very wide projections (\(d>8192\)) the number of scale rounds is capped at two (`eff_ls_iters = min(ls_iters, 2)`) because each round solves a \(2176\times2176\) Cholesky system per row; two rounds plus the final alternate-ridge phase capture essentially all of the achievable reduction at a fraction of the cost.
---
## Per-row guards and exact repacking
The best candidate is tracked **per output row**, including its codes and all Q2_K lattice fields \((q,s,o,d,d_{min},ls,lm)\):
- A row only takes a new candidate if its output error improves on **both** the fitting tokens and the held-out tokens (`e_c <= e_best` and `v_c <= v_best`).
- Rejected rows restart the next round from their own best state, so a rejected candidate can never poison a later solve.
- After the normal rounds, an **alternate-ridge phase** (`ls_ridge_alt`, default `3e-2`) runs one more scale solve with a stronger prior. Because it runs after the fit has reached its best state and is row-guarded, it can only improve on that state.
- The winning rows are repacked with `_q2k_pack_from_parts` (no re-quantization) and verified bit-exactly against the independent test decoder; a mismatch raises rather than silently emitting a wrong payload.
### Converter round-trip invariance
The staging GGUF stores F16 weights that the converter later re-quantizes with `factorize_q2k`. That function recovers a Q2_K lattice tensor bit-exactly when each active sub-block spans both boundary codes (\(\min q=0,\ \max q=3\)) and each super-block uses the canonical super-scales \(d=\operatorname{FP16}(\max_j s_j/15)\), \(d_{min}=\operatorname{FP16}(\max_j o_j/15)\). The metric-coupled snap produces canonical super-scales, and the code re-derivation enforces the boundary codes on the extreme continuous ratios. The trainer's final `manager.refresh()` therefore measures **exactly** what the F16 export followed by the converter will emit, and the block-level guard (below) is evaluated on that real re-quantized result.
---
## Block-output residual absorption
A decoder block computes \(\text{Out}=h+\text{Attn}(h)+\text{FFN}(h+\text{Attn}(h))\). By the time the final projection group runs (typically `ffn_down` / `mlp.down_proj`, which writes linearly into the residual stream), the block already carries the accumulated quantization error of the attention sub-block and the activation shift from quantized `gate/up`.
Fitting that final projection against its isolated teacher output \(X_{fp}W_{fp}^\top\) would ignore this upstream error. Instead, when the last group is a single projection that adds directly into the block output, the solver sets its target to the **true block-output residual**:
\[
Y^\star = \text{targets} - \big(\text{BlockOut}_{stu} - X_q W_{stu}^\top\big)
= \text{targets} - \text{residual}_{stu}.
\]
Because \(\text{BlockOut}_{stu}(W)=\text{residual}_{stu}+X_qW^\top\), minimizing \(\|X_qW^\top-Y^\star\|^2\) is **identically** minimizing the end-to-end block-output error \(\|\text{BlockOut}_{stu}(W)-\text{targets}\|^2\), row by row. The final projection thus absorbs the upstream quantization residual in closed form.
Whether a projection qualifies is verified at runtime, not assumed: a one-token probe adds `+1.0` to the module's output and checks that the decoder-layer output changes by exactly `+1.0` (identity residual, no post-projection norm). If the check fails, the standard teacher target is used. This keeps the absorption correct across architectures without hardcoding module names.
---
## Block-level revert guard
After the algebraic fit, the trainer re-quantizes the fitted weights (`manager.refresh()`) and evaluates the held-out block output. If the fitted-and-re-quantized relative error is worse than the plain-SMQ baseline by more than `1e-6`, the **entire block is reverted** to plain SMQ and re-evaluated. Combined with the per-row guards, this means a block can never end up worse than plain SMQ on the held-out split.
---
## Sequential block adaptation
The QAT flow is:
1. Read the GGUF architecture metadata and discover the decoder block structure from tensor names and metadata.
2. Build corpus token sequences using the tokenizer embedded in the GGUF, or a supplied tokenizer path.
3. For the first trained block, embed the corpus with the source embedding table. For later blocks, use the activation buffer produced by the previous block.
4. Prepare a one-decoder-layer HF model containing the current block. Tensor-name/shape mappings and supported conversion inversions are checked before training.
5. Compute source-weight block-output targets (FP32, pre-norm) and record the held-out tail split.
6. Attach fake-quant modules to eligible linear projections.
7. Run the algebraic fit: group discovery β†’ Gram statistics β†’ closed-form scale solve β†’ metric-coupled snap β†’ prior-annealed code coordinate descent β†’ per-row guards β†’ alternate-ridge phase β†’ residual absorption on the final projection.
8. Re-quantize, evaluate the held-out block output, and revert the block if it regressed.
9. Save adapted block tensors in `safetensors` under the QAT work directory (`trained/blk{b}.safetensors`).
10. Refresh Q2_K caches and propagate the adapted, fake-quantized block output to the next decoder block (`propagate="qat"`).
11. Export a complete mostly-F16 staging GGUF. Trained 2-D projections are written as F16; tensors with no trained counterpart pass through unchanged.
12. Run the SMQ converter on that staging GGUF to produce the final Q2_K model.
The trainer discovers block tensors from the GGUF and uses explicit architecture/name mappings for the HF bridge. The supplied mapping includes `llama`, `qwen2`, `qwen3`, `mistral`, `qwen3next`, and `qwen35`; unsupported architectures or unmapped tensor names fail validation rather than being silently guessed. NextN/MTP draft blocks are excluded from training because they are not part of the main forward path.
### Resume
Each block's fitted weights are saved to `trained/blk{b}.safetensors`. On a rerun, an existing file is loaded and the block is skipped. **To retrain a block with new settings, delete its `trained/blk{b}.safetensors` first.**
### Corpus construction
The trainer reads `.txt`, `.text`, `.md`, and `.jsonl` inputs. For JSONL it uses a `text` field, falling back to `content`; malformed JSON lines are treated as plain text. A directory is read recursively in sorted path order.
After tokenization, it selects `num_seqs` evenly spaced windows of length `seq_len` from the available token stream, then shuffles their order using the supplied seed. `max_tokens=0` means use the full token stream; a positive value truncates the stream before windows are selected.
Activation buffers are stored as BF16, in RAM for smaller corpora or memory-mapped files when the sequence count exceeds the configured threshold (64 by default). Targets are stored as FP32.
---
## QAT training example
The training CLI is `train_smq_qat.py`:
```bash
python train_smq_qat.py \
--gguf model-source-Q8_0.gguf \
--corpus calibration_data.txt \
--output model-adapted-f16.gguf \
--imatrix model.imatrix.gguf \
--num-seqs 32 --seq-len 512 \
--micro-batch 2 \
--ls-iters 4 --cd-sweeps 2 \
--device cuda:0
```
Then quantize the staging GGUF with the converter:
```bash
python smq_gguf_convert_live.py model-adapted-f16.gguf \
--imatrix model.imatrix.gguf \
--output-gguf model-smq-q2k.gguf \
--out reports/model-smq-q2k
```
The Python API is equivalent:
```python
from pathlib import Path
from smq_qat import train
train(
gguf_path=Path("model-source-Q8_0.gguf"),
corpus=Path("calibration_data.txt"),
output=Path("model-adapted-f16.gguf"),
num_seqs=32,
seq_len=512,
micro_batch=2,
ls_iters=4,
ls_ridge=1.0e-3,
ls_ridge_alt=3.0e-2,
cd_sweeps=2,
cd_damp=0.01,
cd_ridge=0.0,
seed=1234,
imatrix=Path("model.imatrix.gguf"),
device="cuda:0",
propagate="qat",
)
```
### Algebraic fit parameters
| Parameter / flag | Default | Meaning |
|---|---:|---|
| `ls_iters` / `--ls-iters` | `4` | Scale-solve β†’ snap β†’ code-update rounds per projection (capped at 2 for `d>8192`) |
| `ls_ridge` / `--ls-ridge` | `1e-3` | Proximal ridge of the scale solve toward current scales, relative to a unit diagonal |
| `ls_ridge_alt` / `--ls-ridge-alt` | `3e-2` | Extra final scale-solve phase with a stronger prior; row-guarded, so it cannot worsen the fit (`0` disables) |
| `ls_row_chunk` / `--ls-row-chunk` | `0` | Rows per batched scale solve (`0` = auto from free VRAM) |
| `cd_sweeps` / `--cd-sweeps` | `2` | Output-aware coordinate-descent sweeps over the 2-bit codes after each scale solve (`0` = weight-space re-derivation fallback) |
| `cd_damp` / `--cd-damp` | `0.01` | Relative step damping of the code update (trust region) |
| `cd_ridge` / `--cd-ridge` | `0.0` | Proximal ridge of the code update toward its starting weights (GPTQ-style `percdamp`) |
### Other useful parameters
| Parameter / flag | Default | Meaning |
|---|---:|---|
| `gguf_path` / `--gguf` | required | Source GGUF used to construct block teachers |
| `corpus` / `--corpus` | required | Text file, JSONL file, or directory of supported text files |
| `output` / `--output` | required | Complete adapted staging GGUF output path |
| `blocks` / `--blocks LO-HI` | all | Optional block range; `HI` exclusive |
| `num_seqs` / `--num-seqs` | `32` | Number of corpus windows |
| `seq_len` / `--seq-len` | `512` | Tokens per window |
| `max_tokens` / `--max-tokens` | `0` | Optional token-stream limit; zero means all |
| `micro_batch` / `--micro-batch` | `4` | Sequences per forward micro-batch while collecting Gram statistics |
| `seed` / `--seed` | `1234` (CLI) | Corpus-window shuffle seed |
| `imatrix` / `--imatrix` | none | Optional `.dat` or GGUF importance matrix |
| `tokenizer` / `--tokenizer` | none | Optional external tokenizer path; otherwise GGUF tokenizer metadata is used |
| `device` / `--device` | `cuda` | Training device, e.g. `cuda:0` or `cpu` |
| `propagate` / `--propagate` | `qat` | Activation propagation mode (`qat` or `full`) |
| `train_types` / `--train-types` | all | Comma-separated tensor types to train |
| `offload_q` / `--offload-q` | `auto` | Decoded fake-quant cache placement (`auto`/`on`/`off`) |
| `offload_threshold` | `64` | Sequence count above which activations are memory-mapped |
| `quant_workers` / `--quant-workers` | auto | Tensors quantized concurrently during a refresh |
| `work_dir` / `--work-dir` | `<output>.qat_work` | Work directory (holds `trained/` and temp buffers) |
| `keep_block_dirs` / `--keep-block-dirs` | off | Keep per-block HF partial directories |
| `validate` | true | Run structural/transform/HF-mapping preflight before training |
| `verbose` / `--verbose` | off | Log every trainable tensor and per-iteration row counts |
| `dry_run` / `--dry-run` | off | Print the plan and exit |
### Legacy optimizer arguments
The CLI and `train()` still accept the old optimizer arguments (`--lr`, `--steps`, `--weight-decay`, `--weight-rmse-lambda`, `--cosine-lambda`, `--grad-accum`, `--requant-every`, `--requant-drift`, `--warmup-frac`, `--norm-lr-mult`, `--cache-quant-error`, `--fresh-penalty`, `--adaptive-step`, etc.) so existing callers do not break, but they are **ignored** by the algebraic fit. There is no optimizer to configure.
---
## Preflight and export utilities
The `smq_qat.py` command-line entry point is for validation and export; full training is driven by `train_smq_qat.py`.
### Validate architecture and tensor mappings
```bash
python smq_qat.py model.gguf
python smq_qat.py model.gguf --blocks 0 8 # upper bound exclusive
python smq_qat.py model.gguf --no-deep # skip the Transformers name/shape comparison
```
The preflight checks tensor mappings and dimensions, verifies GGUF-to-HF transforms against their source values, and can compare supplied parameters with the actual Transformers model structure.
### Export already-trained blocks
```bash
python smq_qat.py source.gguf \
--export-trained /path/to/output.qat_work/trained \
--output adapted-f16.gguf
```
An optional `--tmp-dir` controls where the GGUF export spool file is written.
---
## Reading the metrics
### Converter metrics
The converter's per-tensor CSV/JSON reports include quantization statistics such as relative RMSE and its comparison with the RTN baseline. With an imatrix, these statistics are importance-weighted. They describe weight reconstruction, not model-level perplexity or downstream task performance.
### QAT weight RMSE
The trainer logs absolute and relative RMSE (`||W-Q(W)|| / ||W||`) for each trained projection and an aggregate relative RMSE, measured **after the final re-quantization** β€” i.e. the trained master weights against the decoded Q2_K representation the converter will actually emit. A lower weight RMSE means the adapted weights are more accurately represented by Q2_K; it does not by itself establish better PPL or generalization.
### QAT output RMSE and cosine
The block-output diagnostic compares the trained, re-quantized block's output with its source-weight target, captured before the final model norm. Unlike earlier optimizer-based versions, this diagnostic is evaluated on the **held-out tail split**, not on training sequences. The trainer logs:
- `BASELINE, plain SMQ before training`
- `ALGEBRAIC, fitted + re-quantized`
- `REVERTED, plain SMQ` (only if the block-level guard fired)
- `FINAL, trained + re-quantized`
Each line reports `out_rmse`, relative error, and cosine similarity on the held-out sequences. Per-projection lines additionally report fit-split and held-out relative error before/after, and the number of rows improved.
Use an independent corpus and evaluate the complete exported Q2_K model for final quality comparisons; block-output error is a necessary but not sufficient condition for end-to-end quality.
---
## Reported experimental observations
The following are experimental observations from development, not universal guarantees or independently reproduced benchmark claims.
### Block 0: algebraic fit on a 27B model
A single-block run on `ThinkingCap-Qwen3.8-27B-abliterated-Q8_0.gguf` (block 0, 32 sequences Γ— 512 tokens, imatrix supplied, `ls_iters=4`, `cd_sweeps=2`, `ls_ridge_alt=3e-2`) completed in **~266 s on one RTX 3060 (12 GiB)** with **every output row improved on every tensor**:
| Tensor | Rows improved | Train rel error | Held-out rel error | Fit time |
|---|---:|---:|---:|---:|
| `linear_attn.in_proj_qkv.weight` | 10240 / 10240 | 8.06% β†’ 4.26% | 7.94% β†’ 5.31% | 25.1 s |
| `linear_attn.in_proj_z.weight` | 6144 / 6144 | 10.76% β†’ 5.85% | 10.48% β†’ 7.21% | 15.4 s |
| `linear_attn.in_proj_b.weight` | 48 / 48 | 7.96% β†’ 4.00% | 7.81% β†’ 4.92% | 2.0 s |
| `linear_attn.in_proj_a.weight` | 48 / 48 | 8.82% β†’ 4.59% | 8.46% β†’ 5.55% | 2.0 s |
| `linear_attn.out_proj.weight` | 5120 / 5120 | 16.73% β†’ 1.63% | 16.72% β†’ 2.26% | 18.6 s |
| `mlp.gate_proj.weight` | 17408 / 17408 | 22.52% β†’ 13.13% | 24.72% β†’ 18.82% | 43.5 s |
| `mlp.up_proj.weight` | 17408 / 17408 | 23.25% β†’ 13.48% | 25.43% β†’ 19.24% | 43.3 s |
| `mlp.down_proj.weight` | 5120 / 5120 | 22.93% β†’ 12.10% | 30.29% β†’ 26.28% | 104.3 s |
End-to-end held-out block output, **after real Q2_K re-quantization**:
| Measurement | Plain SMQ baseline | After algebraic QAT |
|---|---:|---:|
| Block-output RMSE (`out_rmse`) | 2.609e-02 | 7.016e-03 |
| Relative block-output error | 12.68% | 3.41% |
| Cosine similarity | 0.99351 | 0.99950 |
| Aggregate relative weight RMSE (post-requant) | 29.36% | 1.81% |
This is a **~3.7Γ— reduction in held-out block-output RMSE** with no optimizer and no backward pass. The `mlp.down_proj` row uses residual absorption, so its before/after numbers reflect the full block-output residual rather than an isolated layer target. These are single-block diagnostics on a held-out calibration split, not a whole-model perplexity or benchmark claim.
### Whole-model observations
During development, an approximately 8.6 GB Q2 model from a reported ~45 GB Q8_0 source was reported, along with perplexity around 8 and a higher ARC-Challenge score than the compared baseline. The exact evaluation corpus, ARC scoring protocol, baseline artifacts, and reproducibility procedure were not included in the supplied source files; treat these as reported experimental observations rather than a benchmark claim.
---
## Memory and performance notes
- **No optimizer state.** Per trainable parameter the fit needs roughly the student weight (4 B) + decoded cache (4 B) + codes (1 B) + decode/snap temporaries (~4 B) β‰ˆ **13 B/param**, plus one \(d\times d\) Gram matrix and a \(d\times m\) cross-Gram per input group. There are no momentum/variance buffers and no activation graph for backprop.
- **Chunked, OOM-resilient solves.** Scale-solve row chunks and coordinate-descent row chunks are sized from free VRAM and halved on `OutOfMemoryError`. The scale-solve contraction tensor is built in sub-block slices so wide projections do not materialize a full-size temporary.
- **Precision.** Gram statistics and coordinate descent may use TF32 (`"high"`); the normal-equation Cholesky solves run at `"highest"`. Because the codes are 2-bit integers, TF32 in the statistics does not degrade the lattice fit.
- **Wide projections.** For `d>8192` the scale-round count is capped at two and the prior schedule is shortened, keeping the \(2176\times2176\) per-row Cholesky cost bounded.
- **Fragmentation.** On tight VRAM, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` reduces fragmentation across the many differently-sized allocations of a block.
- **Offloading.** The decoded fake-quant cache can be kept in pinned CPU RAM (`--offload-q on` or `auto`) when the memory plan indicates it will not fit on the GPU.
---
## Validation recommendations
For a fair evaluation of quantizer changes or QAT hyperparameters:
1. **Keep the source model fixed.** Record its exact GGUF file, tensor types, and checksum.
2. **Keep imatrix provenance fixed.** Record the imatrix file and whether each tensor hit or missed it.
3. **Use the same corpus windows and seed** when comparing settings.
4. **Separate training and evaluation text.** The block diagnostic uses a held-out calibration tail, but final quality should be measured on independent data with the complete exported model.
5. **Measure the actual emitted Q2_K file.** Decode the final packed Q2_K tensors and compute reconstruction error against the corresponding adapted staging weights.
6. **Evaluate end-to-end.** Record PPL, KLD, token-probability changes, and benchmark accuracy with identical prompts, tokenization, scoring rules, and evaluation subsets.
7. **Ablate the fit.** Compare `ls_ridge`, `ls_ridge_alt`, `cd_sweeps`, `cd_damp`, and `cd_ridge` while holding corpus, seed, and source checkpoint constant.
8. **Keep a copy of the adapted staging GGUF.** It makes it possible to distinguish QAT effects from the final converter effects.
For multiple-choice benchmarks such as ARC-Challenge, report both accuracy and the exact scoring method. Perplexity and multiple-choice accuracy measure different behavior and can move in different directions.
---
## Implementation notes and limitations
### QAT does not adapt every matrix that the converter quantizes
The converter's selection is generic and can include eligible embeddings and output matrices. The QAT module's trainable projection list is built from decoder-block tensors. The token embedding is used to construct the first activation buffer but is not optimized; the output/LM head is not part of the block-output objective. One-dimensional norms/biases are not co-adapted by the algebraic fit. Those matrices can still be quantized by the final converter if eligible.
### Source precision defines the teacher
β€œFull-precision target” in the QAT code means the target produced by the source GGUF weights after the GGUF-to-HF conversion. It does not restore precision discarded before the input GGUF was created.
### Block diagnostics are not a held-out model benchmark
The block-output diagnostic uses a held-out tail of the calibration corpus, which is a genuine validation split for the fit, but it is still a single-block, calibration-domain measurement. Evaluate the complete exported model on independent data before drawing conclusions about generalization.
### Guarded payload versus reported candidate metrics
In the supplied converter snapshot, the per-super-block guard chooses between the quadratic candidate and RTN for the **packed parameters**. However, the current metric accumulator uses the quadratic candidate's residual moments, and the optional reconstruction buffer is built from the quadratic candidate residual. If any super-block falls back to RTN, these candidate metrics/reconstruction files may not exactly equal the mixed guarded payload that was packed. For an exact report on the emitted GGUF, decode the written Q2_K tensor and compare it directly with the source tensor.
### Disabling the RTN guard
`--no-rtn-metrics` disables both the RTN comparison and its per-block safeguard. Use it only when an unguarded quadratic decomposition is intended.
### Supported architectures
The QAT HF bridge uses an explicit architecture-to-Transformers model mapping plus GGUF metadata and tensor-name transforms. Unsupported architectures, unknown tensor suffixes, or shape mismatches fail preflight rather than being guessed. Converter-only GGUF quantization does not require the HF bridge. Residual absorption is verified per block by a runtime probe rather than assumed from architecture names.
### File-size comparisons
The nominal Q2_K block payload is 2.625 bits per weight. Whole-file size depends on the number and size of eligible tensors, pass-through tensors, metadata, and any selected tensor filters. Always compare actual file sizes and list-tensor reports rather than assuming every model parameter is stored at exactly 2.625 bits.
---
## Project status
SMQ and SMQ-QAT are experimental implementations. The converter includes packing/factorization self-tests and validates GGUF tensor shapes when writing. The QAT module includes preflight checks for GGUF/HF mappings and conversion transforms, per-row and per-block held-out guards, and bit-exact repack verification. Model-level PPL and benchmark claims should be reported with independent evaluation data and reproducible settings.