Algebraic QAT for SMQ Q2_K
smq_qat.py quantizes a GGUF model block by block. It used to do this with
AdamW and a straight-through estimator. It now does it with closed-form
least squares: no optimizer, no gradients, no learning rate, no steps.
The idea in one paragraph: Q2_K stores each 16-weight sub-block as
w = s*q - o with 2-bit codes q. For fixed codes, a linear layer's output
is linear in the scales s and offsets o, so choosing the best (s, o)
for the layer's output is a least-squares problem with an exact solution. We
solve it, snap the result to the Q2_K lattice, and guard every output row so
nothing can get worse.
Status. The Gram-matrix algebra was checked numerically against the direct least-squares formulation (agreement ~1e-15). The full pipeline has been run on the first blocks of a 27B hybrid linear-attention model (results below). It has not been run across a whole model, and it has not been tested on architectures other than the one in the logs.
1. What changed compared with the AdamW trainer
| Before | Now |
|---|---|
| AdamW on fp weights, STE through the real SMQ quantizer | Closed-form solve per projection, no gradients |
lr, steps, grad_accum, requant_*, warmup_frac, cosine_lambda, weight_rmse_lambda, cache_quant_error, norm_lr_multiplier |
Still accepted by train() so launchers don't break, but ignored |
RMSNorm weights co-adapted (train_norms) |
Not trained. Norm weights are left untouched |
| Optimizer state on the GPU (~22 B/param) | No optimizer state (~13 B/param plus Gram matrices) |
_SpaceStop (space key ends a block) |
Removed, nothing to stop |
New arguments of train():
| Argument | Default | Meaning |
|---|---|---|
ls_iters |
2 |
Rounds of solve → snap → re-derive codes. Round 0 keeps SMQ's own codes; round ≥ 1 re-derives codes for the snapped scales and refits |
ls_ridge |
1e-3 |
Proximal ridge toward the current scales, relative to a unit diagonal (see §4.4) |
ls_row_chunk |
0 |
Rows per batched solve. 0 = chosen automatically from free GPU memory |
verbose=True prints per-iteration row counts and training error.
The unchanged parts: the GGUF→HF bridge, corpus handling, full-precision
targets, activation propagation (propagate="qat"|"full"), checkpoint
resume, and GGUF export.
2. Pipeline per block
1. compute full-precision block targets (unchanged)
2. manager.refresh() real SMQ quantizer on fp16(W_fp); keeps packed payloads
3. BASELINE eval plain SMQ block-output error on held-out sequences
4. algebraic fit (this document)
5. ALGEBRAIC eval block-output error with the fitted lattice weights
6. save trained fp weights = the decoded lattice points, written as F16 later
7. manager.refresh() again real SMQ quantizer applied to the fitted weights
8. FINAL eval what the converter will actually produce
9. propagate activations to the next block (fake-quantized when propagate="qat")
Step 7 is the honest end-to-end check. The exported weight is a lattice point, and the converter re-quantizes the GGUF it is given. If the quantizer reproduces the lattice point, FINAL ≈ ALGEBRAIC. A large gap would mean the fitted solution does not survive the converter.
Step 4 in detail
- Discover order and groups. A probe forward pass with pre-hooks records the order in which the attached projections run and which of them receive the same input tensor (q/k/v, gate/up, qkv/z/a/b, ...). Projections with one shared input form a group and share one Gram matrix. A projection never called in the forward pass keeps its baseline quantization.
- For each group, in execution order, stream the training sequences and
accumulate the Gram statistics (§4.2):
- the teacher pass runs the block in pure fp, capturing the group's
input
X_fpand computingY = X_fp · W_fpᵀ; - the student pass runs the block with all earlier groups already
replaced by their fitted weights, capturing
X_q. (For the first groupX_q = X_fpand the second pass is skipped); - both passes abort as soon as the group's input is captured (a pre-hook raises an exception), so nothing past that layer is computed.
- the teacher pass runs the block in pure fp, capturing the group's
input
- Fit each projection of the group (solve, snap, optional re-derive, guard, §4).
- The fitted weights become the student weights for the next group.
Held-out sequences (the last min(8, max(1, N//4)) of the corpus) are
collected in the same pass and used only for the row guard and for logging.
3. The model
A row w ∈ ℝ^d of a weight matrix (d = 16J) is split into J sub-blocks of
16 weights:
w[16j + a] = s[j] * q[j,a] - o[j] q ∈ {0,1,2,3}, s, o ≥ 0
On the Q2_K lattice, per 256-weight super-block (16 sub-blocks):
s[j] = D * ls[j] ls[j] ∈ {0..15} D = fp16(max_j s / 15)
o[j] = Dmin * lm[j] lm[j] ∈ {0..15} Dmin = fp16(max_j o / 15)
The packed 84-byte block layout is scales[16] | qs[64] | d(fp16) | dmin(fp16).
_q2k_unpack mirrors the bit-exact decoder, and _fit_module refuses to
run if its unpacked weights disagree with the bit-exact decoder
(max|diff| > 1e-4 · max|W|).
4. The math
All equations are per output row r of one projection; rows are independent.
4.1 Objective
X_q : input of the projection in the block with EARLIER groups quantized
X_fp : input of the projection in the pure fp block
y_r = X_fp · w_fp,r (bias excluded: it is identical in both)
min over θ_r = (s_r, o_r): || X_q · w_r(θ) - y_r ||²
The quantized layer is trained to reproduce what the fp layer would have output, on the inputs the quantized block actually receives. Because groups run in execution order, later projections compensate for error introduced by earlier ones.
4.2 Linearity in (s, o) and the Gram form
For fixed codes, let U_j = X_{q,j} · q_j and V_j = X_{q,j} · 1, where
X_{q,j} is the slice of inputs belonging to sub-block j. Then
X_q · w_r = Σ_j ( s_j U_j - o_j V_j ) = A_r θ_r , A_r = [ U_1..U_J , -V_1..-V_J ]
which is ordinary least squares: (A_rᵀ A_r) θ_r = A_rᵀ y_r.
The expensive object is A_r (N × 2J). We never build it. Instead, accumulate
by streaming over tokens, in fp32, with TF32 disabled:
C = X_qᵀ X_q (d × d)
G = X_qᵀ Y (d × m)
yy = Σ_tokens Y² (m)
Writing C_{jk} for the 16×16 block of C for sub-blocks (j,k), and 1
for the all-ones 16-vector:
UtU[j,k] = q_jᵀ C_{jk} q_k
UtV[j,k] = q_jᵀ C_{jk} 1
VtV[j,k] = 1ᵀ C_{jk} 1 (shared by all rows)
A_rᵀ A_r = [ UtU -UtV ] b_r = A_rᵀ y_r = [ q_jᵀ G_{j,r} ]
[ -UtVᵀ VtV ] [ -1ᵀ G_{j,r} ]
Consequences:
- Activations are never stored. Memory is
O(d² + d·m); the solve does not depend on the number of tokens. - Cost per row is about
2·256·J²flops to assemble and(2J)³/3to solve. - Row error is available without activations:
e_r(w) = wᵀ C w - 2 wᵀ G[:,r] + yy[r] ( = || X_q w - y_r ||² )
4.3 Why training error is computed this way but held-out error is not
Held-out error is computed directly, v_r(w) = || X_v w - Y_v[:,r] ||², from
stored held-out inputs X_v and targets Y_v (about 8k tokens). That is
cheaper than another d×d Gram matrix for large d.
4.4 Regularized solve
The solver minimizes
|| A θ - y ||² + λ Σ_i (AᵀA)_ii (θ_i - θ0_i)²
⇔ (AᵀA + λ D²) θ = b + λ D² θ0 , D = diag(AᵀA)^{1/2}
with θ0 the current scales (the SMQ baseline in round 0, the previous
round's snapped scales afterwards) and λ = ls_ridge. In practice it solves
the Jacobi-scaled system (D⁻¹ AᵀA D⁻¹ + λI) θ̃ = D⁻¹ b + λ D θ0 with a
batched Cholesky factorization, then θ = D⁻¹ θ̃.
Why a prior instead of plain ridge: directions the data does not determine (for example input channels that are always zero on the fitting tokens) stay at their current value instead of being pulled to zero. Without this, a plain ridge would zero weights that matter on other inputs.
If Cholesky fails for a row, λ is multiplied by 100 for that row, up to
three times; if it still fails the row keeps θ0.
4.5 Snap to the lattice
The solve is unconstrained. Afterwards:
θ ← max(θ, 0)
ls[j] = clamp( round(s[j] / D), 0, 15 ), D = fp16( max_j s / 15 )
lm[j] = clamp( round(o[j] / Dmin), 0, 15 ), Dmin = fp16( max_j o / 15 )
This is a heuristic projection onto the lattice, not the constrained optimum. The guard (§4.7) is what makes that safe.
4.6 Code re-derivation (rounds ≥ 1)
For the snapped scales, codes are re-derived against the fp16-rounded fp
weights w_src:
q_new = clamp( round( (w_src + o') / s' ), 0, 3 ), q_new = 0 where s' = 0
This is nearest rounding in weight space. It is not output-aware.
4.7 Row guard
A candidate row replaces the best-so-far row only if it improves both the fitting-token error and the held-out error:
accept row r ⇔ e_r(cand) ≤ e_r(best) AND v_r(cand) ≤ v_r(best)
So a row can never end up worse than the plain SMQ baseline on either set.
The rows N/M field in the logs counts rows that took an update.
4.8 Full iteration
(q⁰, θ⁰) = real SMQ quantization of fp16(W_fp) # via packed payload
best = decode(q⁰, θ⁰)
for t in 0 .. ls_iters - 1:
θ_ls = LS_solve(q^t ; prior θ^t) # §4.2–4.4
θ' = snap(max(θ_ls, 0)) # §4.5
cand = decode(q^t, θ') → guard against best # §4.7
q^{t+1}, θ^{t+1} = rederive(θ', w_src), θ' # §4.6 (not guarded)
The exported tensor is best: the decoded lattice point, stored fp32 in the
trained-weights file and written as F16 to the output GGUF.
5. Code map
| Piece | Where |
|---|---|
| Real quantizer + packed payload | quantize_pack, decode_packed, quantize_dequantize_torch(..., return_packed=True) |
| Cache the payload | QatWeight(keep_packed=True), QatManager(keep_packed=True) |
| Unpack / decode / snap / re-derive | _q2k_unpack, _q2k_decode, _snap_scales, _rederive_codes |
| Row errors | _row_err_gram (train, from Gram), _row_err_val (held-out, direct) |
| Normal equations + solve | _ls_fit_gram |
| Early-abort input capture | _StopForward, _capture_input, _set_weights |
| Grouping | BlockQatTrainer._discover_groups |
| Gram statistics | BlockQatTrainer._collect_group_stats |
| Per-projection fit | BlockQatTrainer._fit_module |
| Block driver | BlockQatTrainer._algebraic_fit, _train |
The teacher and student passes share the same model object. Weights are
swapped by pointing each parameter's .data at either the fp tensor or the
fitted tensor, and the fake-quant parametrization is bypassed
(qw._defer = True) while fitting.
6. Reading the logs
For each block you get three block-output lines, all on held-out sequences:
BASELINE plain SMQ before training
ALGEBRAIC fitted lattice, before the converter's re-quantization
FINAL real quantizer applied to the fitted weights (what you will ship)
Observed on a 27B hybrid linear-attention model (256 sequences × 1024 tokens, 12 GB GPU):
| Block | BASELINE | ALGEBRAIC | FINAL | cos (FINAL) |
|---|---|---|---|---|
| 0 | 12.69 % | 3.02 % | 3.26 % | 0.99954 |
| 2 | 5.81 % | 4.11 % | 4.16 % | 0.99922 |
Absolute errors are not comparable across blocks (different activations and targets); what matters is FINAL against BASELINE for the same block and whether FINAL stays close to ALGEBRAIC.
Per-projection lines look like:
blk.2: mlp.down_proj.weight out-rel train 21.15%->17.53% held-out 20.04%->17.57% rows 5096/5120
out-relis the relative output error of that single projection against the fp teacher, before and after.- Train and held-out moving together means the fit is not overfitting.
rowsbelow the total means the guard rejected some rows.
The line weight RMSE after final re-quantization (trained W vs Q(W)) measures
how far the exported weights move when the converter runs the quantizer on
them again. Small values (≈1–3 % relative) mean the lattice points survive.
It is not comparable with the ≈29 % at the start of the block, which is the
distance from the original fp weights.
7. Resources
- Time. About 280–300 s of fitting per block on the logged hardware, plus
≈13 s for the initial quantization and the target pass.
mlp.down_projdominates the fit (≈60 s) becauseJis large and the batched Cholesky scales as(2J)³. - Memory. Per parameter ≈ 13 B (student weight, decoded cache, codes,
temporaries), plus per input group a
d × dGram matrix and ad × mcross-Gram per projection. The largest object is the Gram offfn_down(about 1.2 GB atd = 17408). - What the fit does to fit on a 12 GB card: decoded baseline copies are moved to host RAM during the fit; during the last group the finished weights are parked in host RAM; the solve picks its row batch from free GPU memory and retries with half the rows on out-of-memory.
The CUDA "allocation failed" warning
[W...] CUDACachingAllocator.cpp: memory allocation failed with OOM ...
appearing during mlp.down_proj is harmless. It is PyTorch's allocator
warning before the solve's retry path halves the row batch. If the run
actually fails, you get a Python exception. In that case try, in this order:
--micro-batch 1, a smaller --seq-len, then train_types=<subset>.
8. Known limitations
- Codes are not output-aware. The solve only moves scales and offsets
around codes that were chosen by weight-space rounding. Most of the
remaining error comes from here. A GPTQ-style or coordinate-descent update
of the codes against
Cis the natural next step; the Gram matrices it needs are already computed. - Norm weights are not co-adapted, unlike the old trainer.
- Sequential, not joint. Each group is fitted against the teacher output of its own layer, not against the block output. Error from earlier groups is compensated only through the student inputs.
- The snap is a heuristic projection, not a constrained optimum, so the unconstrained least-squares improvement is only partly retained. The guard prevents regressions but does not recover that loss.
- Weight-space code re-derivation (round ≥ 1) can propose codes that are
worse for the output; the guard filters them, but it can mean round 2
gains little.
verbose=Trueshows the gain per round; usels_iters=1to cut fitting time if it is negligible. - Layout assumption. Quantization groups are formed in the HF tensor layout, as in the previous trainer. Any architecture-specific column reordering between the HF and GGUF layouts is not modeled.
- Dense architectures only (no mixture-of-experts handling).
- Overfitting protection is limited to the guard. With few sequences
relative to
2Junknowns per row (larged), increase--num-seqsorls_ridge.
9. Tuning guide
| Symptom | Try |
|---|---|
| FINAL much worse than ALGEBRAIC | The converter's quantizer is not reproducing the lattice points; inspect the weight-RMSE line and the per-tensor row counts |
| Many rows rejected by the guard | Raise ls_ridge (more shrinkage toward the baseline) or use more sequences |
| Almost no rows rejected and gains plateau | Lower ls_ridge (e.g. 1e-4) or raise ls_iters to 3 |
| Round 2 gains almost nothing | ls_iters=1 to save time |
| Out of memory | --micro-batch 1, smaller --seq-len, train_types=<subset> |
| FINAL error creeps up with depth | Compare BASELINE→FINAL per block; consider the code-refinement step (§8.1) |