Algebraic QAT for SMQ Q2_K

smq_qat.py quantizes a GGUF model block by block. It used to do this with AdamW and a straight-through estimator. It now does it with closed-form least squares: no optimizer, no gradients, no learning rate, no steps.

The idea in one paragraph: Q2_K stores each 16-weight sub-block as w = s*q - o with 2-bit codes q. For fixed codes, a linear layer's output is linear in the scales s and offsets o, so choosing the best (s, o) for the layer's output is a least-squares problem with an exact solution. We solve it, snap the result to the Q2_K lattice, and guard every output row so nothing can get worse.

Status. The Gram-matrix algebra was checked numerically against the direct least-squares formulation (agreement ~1e-15). The full pipeline has been run on the first blocks of a 27B hybrid linear-attention model (results below). It has not been run across a whole model, and it has not been tested on architectures other than the one in the logs.


1. What changed compared with the AdamW trainer

Before Now
AdamW on fp weights, STE through the real SMQ quantizer Closed-form solve per projection, no gradients
lr, steps, grad_accum, requant_*, warmup_frac, cosine_lambda, weight_rmse_lambda, cache_quant_error, norm_lr_multiplier Still accepted by train() so launchers don't break, but ignored
RMSNorm weights co-adapted (train_norms) Not trained. Norm weights are left untouched
Optimizer state on the GPU (~22 B/param) No optimizer state (~13 B/param plus Gram matrices)
_SpaceStop (space key ends a block) Removed, nothing to stop

New arguments of train():

Argument Default Meaning
ls_iters 2 Rounds of solve → snap → re-derive codes. Round 0 keeps SMQ's own codes; round ≥ 1 re-derives codes for the snapped scales and refits
ls_ridge 1e-3 Proximal ridge toward the current scales, relative to a unit diagonal (see §4.4)
ls_row_chunk 0 Rows per batched solve. 0 = chosen automatically from free GPU memory

verbose=True prints per-iteration row counts and training error.

The unchanged parts: the GGUF→HF bridge, corpus handling, full-precision targets, activation propagation (propagate="qat"|"full"), checkpoint resume, and GGUF export.


2. Pipeline per block

1. compute full-precision block targets (unchanged)
2. manager.refresh()          real SMQ quantizer on fp16(W_fp); keeps packed payloads
3. BASELINE eval              plain SMQ block-output error on held-out sequences
4. algebraic fit              (this document)
5. ALGEBRAIC eval             block-output error with the fitted lattice weights
6. save trained fp weights    = the decoded lattice points, written as F16 later
7. manager.refresh() again    real SMQ quantizer applied to the fitted weights
8. FINAL eval                 what the converter will actually produce
9. propagate activations to the next block (fake-quantized when propagate="qat")

Step 7 is the honest end-to-end check. The exported weight is a lattice point, and the converter re-quantizes the GGUF it is given. If the quantizer reproduces the lattice point, FINAL ≈ ALGEBRAIC. A large gap would mean the fitted solution does not survive the converter.

Step 4 in detail

  1. Discover order and groups. A probe forward pass with pre-hooks records the order in which the attached projections run and which of them receive the same input tensor (q/k/v, gate/up, qkv/z/a/b, ...). Projections with one shared input form a group and share one Gram matrix. A projection never called in the forward pass keeps its baseline quantization.
  2. For each group, in execution order, stream the training sequences and accumulate the Gram statistics (§4.2):
    • the teacher pass runs the block in pure fp, capturing the group's input X_fp and computing Y = X_fp · W_fpᵀ;
    • the student pass runs the block with all earlier groups already replaced by their fitted weights, capturing X_q. (For the first group X_q = X_fp and the second pass is skipped);
    • both passes abort as soon as the group's input is captured (a pre-hook raises an exception), so nothing past that layer is computed.
  3. Fit each projection of the group (solve, snap, optional re-derive, guard, §4).
  4. The fitted weights become the student weights for the next group.

Held-out sequences (the last min(8, max(1, N//4)) of the corpus) are collected in the same pass and used only for the row guard and for logging.


3. The model

A row w ∈ ℝ^d of a weight matrix (d = 16J) is split into J sub-blocks of 16 weights:

w[16j + a] = s[j] * q[j,a] - o[j]          q ∈ {0,1,2,3},  s, o ≥ 0

On the Q2_K lattice, per 256-weight super-block (16 sub-blocks):

s[j] = D    * ls[j]          ls[j] ∈ {0..15}      D    = fp16(max_j s / 15)
o[j] = Dmin * lm[j]          lm[j] ∈ {0..15}      Dmin = fp16(max_j o / 15)

The packed 84-byte block layout is scales[16] | qs[64] | d(fp16) | dmin(fp16). _q2k_unpack mirrors the bit-exact decoder, and _fit_module refuses to run if its unpacked weights disagree with the bit-exact decoder (max|diff| > 1e-4 · max|W|).


4. The math

All equations are per output row r of one projection; rows are independent.

4.1 Objective

X_q  : input of the projection in the block with EARLIER groups quantized
X_fp : input of the projection in the pure fp block
y_r  = X_fp · w_fp,r                 (bias excluded: it is identical in both)

min over θ_r = (s_r, o_r):   || X_q · w_r(θ) - y_r ||²

The quantized layer is trained to reproduce what the fp layer would have output, on the inputs the quantized block actually receives. Because groups run in execution order, later projections compensate for error introduced by earlier ones.

4.2 Linearity in (s, o) and the Gram form

For fixed codes, let U_j = X_{q,j} · q_j and V_j = X_{q,j} · 1, where X_{q,j} is the slice of inputs belonging to sub-block j. Then

X_q · w_r = Σ_j ( s_j U_j - o_j V_j ) = A_r θ_r ,     A_r = [ U_1..U_J , -V_1..-V_J ]

which is ordinary least squares: (A_rᵀ A_r) θ_r = A_rᵀ y_r.

The expensive object is A_r (N × 2J). We never build it. Instead, accumulate by streaming over tokens, in fp32, with TF32 disabled:

C  = X_qᵀ X_q      (d × d)
G  = X_qᵀ Y        (d × m)
yy = Σ_tokens Y²   (m)

Writing C_{jk} for the 16×16 block of C for sub-blocks (j,k), and 1 for the all-ones 16-vector:

UtU[j,k] = q_jᵀ C_{jk} q_k
UtV[j,k] = q_jᵀ C_{jk} 1
VtV[j,k] = 1ᵀ  C_{jk} 1                 (shared by all rows)

A_rᵀ A_r = [  UtU    -UtV ]        b_r = A_rᵀ y_r = [  q_jᵀ G_{j,r}  ]
           [ -UtVᵀ    VtV ]                         [ -1ᵀ  G_{j,r}   ]

Consequences:

  • Activations are never stored. Memory is O(d² + d·m); the solve does not depend on the number of tokens.
  • Cost per row is about 2·256·J² flops to assemble and (2J)³/3 to solve.
  • Row error is available without activations:
e_r(w) = wᵀ C w - 2 wᵀ G[:,r] + yy[r]          ( = || X_q w - y_r ||² )

4.3 Why training error is computed this way but held-out error is not

Held-out error is computed directly, v_r(w) = || X_v w - Y_v[:,r] ||², from stored held-out inputs X_v and targets Y_v (about 8k tokens). That is cheaper than another d×d Gram matrix for large d.

4.4 Regularized solve

The solver minimizes

|| A θ - y ||²  +  λ Σ_i (AᵀA)_ii (θ_i - θ0_i)²
        ⇔   (AᵀA + λ D²) θ  =  b + λ D² θ0 ,        D = diag(AᵀA)^{1/2}

with θ0 the current scales (the SMQ baseline in round 0, the previous round's snapped scales afterwards) and λ = ls_ridge. In practice it solves the Jacobi-scaled system (D⁻¹ AᵀA D⁻¹ + λI) θ̃ = D⁻¹ b + λ D θ0 with a batched Cholesky factorization, then θ = D⁻¹ θ̃.

Why a prior instead of plain ridge: directions the data does not determine (for example input channels that are always zero on the fitting tokens) stay at their current value instead of being pulled to zero. Without this, a plain ridge would zero weights that matter on other inputs.

If Cholesky fails for a row, λ is multiplied by 100 for that row, up to three times; if it still fails the row keeps θ0.

4.5 Snap to the lattice

The solve is unconstrained. Afterwards:

θ  ←  max(θ, 0)
ls[j] = clamp( round(s[j] / D),  0, 15 ),   D    = fp16( max_j s / 15 )
lm[j] = clamp( round(o[j] / Dmin), 0, 15 ), Dmin = fp16( max_j o / 15 )

This is a heuristic projection onto the lattice, not the constrained optimum. The guard (§4.7) is what makes that safe.

4.6 Code re-derivation (rounds ≥ 1)

For the snapped scales, codes are re-derived against the fp16-rounded fp weights w_src:

q_new = clamp( round( (w_src + o') / s' ), 0, 3 ),     q_new = 0 where s' = 0

This is nearest rounding in weight space. It is not output-aware.

4.7 Row guard

A candidate row replaces the best-so-far row only if it improves both the fitting-token error and the held-out error:

accept row r  ⇔  e_r(cand) ≤ e_r(best)   AND   v_r(cand) ≤ v_r(best)

So a row can never end up worse than the plain SMQ baseline on either set. The rows N/M field in the logs counts rows that took an update.

4.8 Full iteration

(q⁰, θ⁰) = real SMQ quantization of fp16(W_fp)            # via packed payload
best     = decode(q⁰, θ⁰)

for t in 0 .. ls_iters - 1:
    θ_ls  = LS_solve(q^t ; prior θ^t)                     # §4.2–4.4
    θ'    = snap(max(θ_ls, 0))                            # §4.5
    cand  = decode(q^t, θ')   →  guard against best       # §4.7
    q^{t+1}, θ^{t+1} = rederive(θ', w_src),  θ'           # §4.6  (not guarded)

The exported tensor is best: the decoded lattice point, stored fp32 in the trained-weights file and written as F16 to the output GGUF.


5. Code map

Piece Where
Real quantizer + packed payload quantize_pack, decode_packed, quantize_dequantize_torch(..., return_packed=True)
Cache the payload QatWeight(keep_packed=True), QatManager(keep_packed=True)
Unpack / decode / snap / re-derive _q2k_unpack, _q2k_decode, _snap_scales, _rederive_codes
Row errors _row_err_gram (train, from Gram), _row_err_val (held-out, direct)
Normal equations + solve _ls_fit_gram
Early-abort input capture _StopForward, _capture_input, _set_weights
Grouping BlockQatTrainer._discover_groups
Gram statistics BlockQatTrainer._collect_group_stats
Per-projection fit BlockQatTrainer._fit_module
Block driver BlockQatTrainer._algebraic_fit, _train

The teacher and student passes share the same model object. Weights are swapped by pointing each parameter's .data at either the fp tensor or the fitted tensor, and the fake-quant parametrization is bypassed (qw._defer = True) while fitting.


6. Reading the logs

For each block you get three block-output lines, all on held-out sequences:

BASELINE   plain SMQ before training
ALGEBRAIC  fitted lattice, before the converter's re-quantization
FINAL      real quantizer applied to the fitted weights (what you will ship)

Observed on a 27B hybrid linear-attention model (256 sequences × 1024 tokens, 12 GB GPU):

Block BASELINE ALGEBRAIC FINAL cos (FINAL)
0 12.69 % 3.02 % 3.26 % 0.99954
2 5.81 % 4.11 % 4.16 % 0.99922

Absolute errors are not comparable across blocks (different activations and targets); what matters is FINAL against BASELINE for the same block and whether FINAL stays close to ALGEBRAIC.

Per-projection lines look like:

blk.2: mlp.down_proj.weight   out-rel train 21.15%->17.53%  held-out 20.04%->17.57%  rows 5096/5120
  • out-rel is the relative output error of that single projection against the fp teacher, before and after.
  • Train and held-out moving together means the fit is not overfitting.
  • rows below the total means the guard rejected some rows.

The line weight RMSE after final re-quantization (trained W vs Q(W)) measures how far the exported weights move when the converter runs the quantizer on them again. Small values (≈1–3 % relative) mean the lattice points survive. It is not comparable with the ≈29 % at the start of the block, which is the distance from the original fp weights.


7. Resources

  • Time. About 280–300 s of fitting per block on the logged hardware, plus ≈13 s for the initial quantization and the target pass. mlp.down_proj dominates the fit (≈60 s) because J is large and the batched Cholesky scales as (2J)³.
  • Memory. Per parameter ≈ 13 B (student weight, decoded cache, codes, temporaries), plus per input group a d × d Gram matrix and a d × m cross-Gram per projection. The largest object is the Gram of ffn_down (about 1.2 GB at d = 17408).
  • What the fit does to fit on a 12 GB card: decoded baseline copies are moved to host RAM during the fit; during the last group the finished weights are parked in host RAM; the solve picks its row batch from free GPU memory and retries with half the rows on out-of-memory.

The CUDA "allocation failed" warning

[W...] CUDACachingAllocator.cpp: memory allocation failed with OOM ...

appearing during mlp.down_proj is harmless. It is PyTorch's allocator warning before the solve's retry path halves the row batch. If the run actually fails, you get a Python exception. In that case try, in this order: --micro-batch 1, a smaller --seq-len, then train_types=<subset>.


8. Known limitations

  1. Codes are not output-aware. The solve only moves scales and offsets around codes that were chosen by weight-space rounding. Most of the remaining error comes from here. A GPTQ-style or coordinate-descent update of the codes against C is the natural next step; the Gram matrices it needs are already computed.
  2. Norm weights are not co-adapted, unlike the old trainer.
  3. Sequential, not joint. Each group is fitted against the teacher output of its own layer, not against the block output. Error from earlier groups is compensated only through the student inputs.
  4. The snap is a heuristic projection, not a constrained optimum, so the unconstrained least-squares improvement is only partly retained. The guard prevents regressions but does not recover that loss.
  5. Weight-space code re-derivation (round ≥ 1) can propose codes that are worse for the output; the guard filters them, but it can mean round 2 gains little. verbose=True shows the gain per round; use ls_iters=1 to cut fitting time if it is negligible.
  6. Layout assumption. Quantization groups are formed in the HF tensor layout, as in the previous trainer. Any architecture-specific column reordering between the HF and GGUF layouts is not modeled.
  7. Dense architectures only (no mixture-of-experts handling).
  8. Overfitting protection is limited to the guard. With few sequences relative to 2J unknowns per row (large d), increase --num-seqs or ls_ridge.

9. Tuning guide

Symptom Try
FINAL much worse than ALGEBRAIC The converter's quantizer is not reproducing the lattice points; inspect the weight-RMSE line and the per-tensor row counts
Many rows rejected by the guard Raise ls_ridge (more shrinkage toward the baseline) or use more sequences
Almost no rows rejected and gains plateau Lower ls_ridge (e.g. 1e-4) or raise ls_iters to 3
Round 2 gains almost nothing ls_iters=1 to save time
Out of memory --micro-batch 1, smaller --seq-len, train_types=<subset>
FINAL error creeps up with depth Compare BASELINE→FINAL per block; consider the code-refinement step (§8.1)
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support