--- license: apache-2.0 --- # Algebraic QAT for SMQ Q2_K `smq_qat.py` quantizes a GGUF model block by block. It used to do this with AdamW and a straight-through estimator. It now does it with **closed-form least squares**: no optimizer, no gradients, no learning rate, no steps. The idea in one paragraph: Q2_K stores each 16-weight sub-block as `w = s*q - o` with 2-bit codes `q`. For fixed codes, a linear layer's output is *linear* in the scales `s` and offsets `o`, so choosing the best `(s, o)` for the layer's output is a least-squares problem with an exact solution. We solve it, snap the result to the Q2_K lattice, and guard every output row so nothing can get worse. > **Status.** The Gram-matrix algebra was checked numerically against the > direct least-squares formulation (agreement ~1e-15). The full pipeline has > been run on the first blocks of a 27B hybrid linear-attention model > (results below). It has not been run across a whole model, and it has not > been tested on architectures other than the one in the logs. --- ## 1. What changed compared with the AdamW trainer | Before | Now | |---|---| | AdamW on fp weights, STE through the real SMQ quantizer | Closed-form solve per projection, no gradients | | `lr`, `steps`, `grad_accum`, `requant_*`, `warmup_frac`, `cosine_lambda`, `weight_rmse_lambda`, `cache_quant_error`, `norm_lr_multiplier` | Still accepted by `train()` so launchers don't break, but **ignored** | | RMSNorm weights co-adapted (`train_norms`) | **Not trained.** Norm weights are left untouched | | Optimizer state on the GPU (~22 B/param) | No optimizer state (~13 B/param plus Gram matrices) | | `_SpaceStop` (space key ends a block) | Removed, nothing to stop | New arguments of `train()`: | Argument | Default | Meaning | |---|---|---| | `ls_iters` | `2` | Rounds of solve → snap → re-derive codes. Round 0 keeps SMQ's own codes; round ≥ 1 re-derives codes for the snapped scales and refits | | `ls_ridge` | `1e-3` | Proximal ridge toward the current scales, relative to a unit diagonal (see §4.4) | | `ls_row_chunk` | `0` | Rows per batched solve. `0` = chosen automatically from free GPU memory | `verbose=True` prints per-iteration row counts and training error. The unchanged parts: the GGUF→HF bridge, corpus handling, full-precision targets, activation propagation (`propagate="qat"|"full"`), checkpoint resume, and GGUF export. --- ## 2. Pipeline per block ``` 1. compute full-precision block targets (unchanged) 2. manager.refresh() real SMQ quantizer on fp16(W_fp); keeps packed payloads 3. BASELINE eval plain SMQ block-output error on held-out sequences 4. algebraic fit (this document) 5. ALGEBRAIC eval block-output error with the fitted lattice weights 6. save trained fp weights = the decoded lattice points, written as F16 later 7. manager.refresh() again real SMQ quantizer applied to the fitted weights 8. FINAL eval what the converter will actually produce 9. propagate activations to the next block (fake-quantized when propagate="qat") ``` Step 7 is the honest end-to-end check. The exported weight is a lattice point, and the converter re-quantizes the GGUF it is given. If the quantizer reproduces the lattice point, FINAL ≈ ALGEBRAIC. A large gap would mean the fitted solution does not survive the converter. ### Step 4 in detail 1. **Discover order and groups.** A probe forward pass with pre-hooks records the order in which the attached projections run and which of them receive the *same input tensor* (q/k/v, gate/up, qkv/z/a/b, ...). Projections with one shared input form a **group** and share one Gram matrix. A projection never called in the forward pass keeps its baseline quantization. 2. **For each group, in execution order**, stream the training sequences and accumulate the Gram statistics (§4.2): * the *teacher* pass runs the block in pure fp, capturing the group's input `X_fp` and computing `Y = X_fp · W_fpᵀ`; * the *student* pass runs the block with all earlier groups already replaced by their fitted weights, capturing `X_q`. (For the first group `X_q = X_fp` and the second pass is skipped); * both passes abort as soon as the group's input is captured (a pre-hook raises an exception), so nothing past that layer is computed. 3. **Fit each projection of the group** (solve, snap, optional re-derive, guard, §4). 4. The fitted weights become the student weights for the next group. Held-out sequences (the last `min(8, max(1, N//4))` of the corpus) are collected in the same pass and used only for the row guard and for logging. --- ## 3. The model A row `w ∈ ℝ^d` of a weight matrix (`d = 16J`) is split into `J` sub-blocks of 16 weights: ``` w[16j + a] = s[j] * q[j,a] - o[j] q ∈ {0,1,2,3}, s, o ≥ 0 ``` On the Q2_K lattice, per 256-weight super-block (16 sub-blocks): ``` s[j] = D * ls[j] ls[j] ∈ {0..15} D = fp16(max_j s / 15) o[j] = Dmin * lm[j] lm[j] ∈ {0..15} Dmin = fp16(max_j o / 15) ``` The packed 84-byte block layout is `scales[16] | qs[64] | d(fp16) | dmin(fp16)`. `_q2k_unpack` mirrors the bit-exact decoder, and `_fit_module` **refuses to run** if its unpacked weights disagree with the bit-exact decoder (`max|diff| > 1e-4 · max|W|`). --- ## 4. The math All equations are per output row `r` of one projection; rows are independent. ### 4.1 Objective ``` X_q : input of the projection in the block with EARLIER groups quantized X_fp : input of the projection in the pure fp block y_r = X_fp · w_fp,r (bias excluded: it is identical in both) min over θ_r = (s_r, o_r): || X_q · w_r(θ) - y_r ||² ``` The quantized layer is trained to reproduce what the fp layer would have output, on the inputs the quantized block actually receives. Because groups run in execution order, later projections compensate for error introduced by earlier ones. ### 4.2 Linearity in (s, o) and the Gram form For fixed codes, let `U_j = X_{q,j} · q_j` and `V_j = X_{q,j} · 1`, where `X_{q,j}` is the slice of inputs belonging to sub-block `j`. Then ``` X_q · w_r = Σ_j ( s_j U_j - o_j V_j ) = A_r θ_r , A_r = [ U_1..U_J , -V_1..-V_J ] ``` which is ordinary least squares: `(A_rᵀ A_r) θ_r = A_rᵀ y_r`. The expensive object is `A_r` (N × 2J). We never build it. Instead, accumulate by streaming over tokens, in fp32, with TF32 disabled: ``` C = X_qᵀ X_q (d × d) G = X_qᵀ Y (d × m) yy = Σ_tokens Y² (m) ``` Writing `C_{jk}` for the 16×16 block of `C` for sub-blocks `(j,k)`, and `1` for the all-ones 16-vector: ``` UtU[j,k] = q_jᵀ C_{jk} q_k UtV[j,k] = q_jᵀ C_{jk} 1 VtV[j,k] = 1ᵀ C_{jk} 1 (shared by all rows) A_rᵀ A_r = [ UtU -UtV ] b_r = A_rᵀ y_r = [ q_jᵀ G_{j,r} ] [ -UtVᵀ VtV ] [ -1ᵀ G_{j,r} ] ``` Consequences: * **Activations are never stored.** Memory is `O(d² + d·m)`; the solve does not depend on the number of tokens. * Cost per row is about `2·256·J²` flops to assemble and `(2J)³/3` to solve. * Row error is available without activations: ``` e_r(w) = wᵀ C w - 2 wᵀ G[:,r] + yy[r] ( = || X_q w - y_r ||² ) ``` ### 4.3 Why training error is computed this way but held-out error is not Held-out error is computed directly, `v_r(w) = || X_v w - Y_v[:,r] ||²`, from stored held-out inputs `X_v` and targets `Y_v` (about 8k tokens). That is cheaper than another `d×d` Gram matrix for large `d`. ### 4.4 Regularized solve The solver minimizes ``` || A θ - y ||² + λ Σ_i (AᵀA)_ii (θ_i - θ0_i)² ⇔ (AᵀA + λ D²) θ = b + λ D² θ0 , D = diag(AᵀA)^{1/2} ``` with `θ0` the *current* scales (the SMQ baseline in round 0, the previous round's snapped scales afterwards) and `λ = ls_ridge`. In practice it solves the Jacobi-scaled system `(D⁻¹ AᵀA D⁻¹ + λI) θ̃ = D⁻¹ b + λ D θ0` with a batched Cholesky factorization, then `θ = D⁻¹ θ̃`. Why a prior instead of plain ridge: directions the data does not determine (for example input channels that are always zero on the fitting tokens) stay at their current value instead of being pulled to zero. Without this, a plain ridge would zero weights that matter on other inputs. If Cholesky fails for a row, `λ` is multiplied by 100 for that row, up to three times; if it still fails the row keeps `θ0`. ### 4.5 Snap to the lattice The solve is unconstrained. Afterwards: ``` θ ← max(θ, 0) ls[j] = clamp( round(s[j] / D), 0, 15 ), D = fp16( max_j s / 15 ) lm[j] = clamp( round(o[j] / Dmin), 0, 15 ), Dmin = fp16( max_j o / 15 ) ``` This is a heuristic projection onto the lattice, not the constrained optimum. The guard (§4.7) is what makes that safe. ### 4.6 Code re-derivation (rounds ≥ 1) For the snapped scales, codes are re-derived against the fp16-rounded fp weights `w_src`: ``` q_new = clamp( round( (w_src + o') / s' ), 0, 3 ), q_new = 0 where s' = 0 ``` This is nearest rounding in **weight space**. It is not output-aware. ### 4.7 Row guard A candidate row replaces the best-so-far row only if it improves both the fitting-token error and the held-out error: ``` accept row r ⇔ e_r(cand) ≤ e_r(best) AND v_r(cand) ≤ v_r(best) ``` So a row can never end up worse than the plain SMQ baseline on either set. The `rows N/M` field in the logs counts rows that took an update. ### 4.8 Full iteration ``` (q⁰, θ⁰) = real SMQ quantization of fp16(W_fp) # via packed payload best = decode(q⁰, θ⁰) for t in 0 .. ls_iters - 1: θ_ls = LS_solve(q^t ; prior θ^t) # §4.2–4.4 θ' = snap(max(θ_ls, 0)) # §4.5 cand = decode(q^t, θ') → guard against best # §4.7 q^{t+1}, θ^{t+1} = rederive(θ', w_src), θ' # §4.6 (not guarded) ``` The exported tensor is `best`: the decoded lattice point, stored fp32 in the trained-weights file and written as F16 to the output GGUF. --- ## 5. Code map | Piece | Where | |---|---| | Real quantizer + packed payload | `quantize_pack`, `decode_packed`, `quantize_dequantize_torch(..., return_packed=True)` | | Cache the payload | `QatWeight(keep_packed=True)`, `QatManager(keep_packed=True)` | | Unpack / decode / snap / re-derive | `_q2k_unpack`, `_q2k_decode`, `_snap_scales`, `_rederive_codes` | | Row errors | `_row_err_gram` (train, from Gram), `_row_err_val` (held-out, direct) | | Normal equations + solve | `_ls_fit_gram` | | Early-abort input capture | `_StopForward`, `_capture_input`, `_set_weights` | | Grouping | `BlockQatTrainer._discover_groups` | | Gram statistics | `BlockQatTrainer._collect_group_stats` | | Per-projection fit | `BlockQatTrainer._fit_module` | | Block driver | `BlockQatTrainer._algebraic_fit`, `_train` | The teacher and student passes share the same model object. Weights are swapped by pointing each parameter's `.data` at either the fp tensor or the fitted tensor, and the fake-quant parametrization is bypassed (`qw._defer = True`) while fitting. --- ## 6. Reading the logs For each block you get three block-output lines, all on held-out sequences: ``` BASELINE plain SMQ before training ALGEBRAIC fitted lattice, before the converter's re-quantization FINAL real quantizer applied to the fitted weights (what you will ship) ``` Observed on a 27B hybrid linear-attention model (256 sequences × 1024 tokens, 12 GB GPU): | Block | BASELINE | ALGEBRAIC | FINAL | cos (FINAL) | |---|---|---|---|---| | 0 | 12.69 % | 3.02 % | 3.26 % | 0.99954 | | 2 | 5.81 % | 4.11 % | 4.16 % | 0.99922 | Absolute errors are not comparable across blocks (different activations and targets); what matters is FINAL against BASELINE for the same block and whether FINAL stays close to ALGEBRAIC. Per-projection lines look like: ``` blk.2: mlp.down_proj.weight out-rel train 21.15%->17.53% held-out 20.04%->17.57% rows 5096/5120 ``` * `out-rel` is the relative output error of that single projection against the fp teacher, before and after. * Train and held-out moving together means the fit is not overfitting. * `rows` below the total means the guard rejected some rows. The line `weight RMSE after final re-quantization (trained W vs Q(W))` measures how far the exported weights move when the converter runs the quantizer on them again. Small values (≈1–3 % relative) mean the lattice points survive. It is not comparable with the ≈29 % at the start of the block, which is the distance from the original fp weights. --- ## 7. Resources * **Time.** About 280–300 s of fitting per block on the logged hardware, plus ≈13 s for the initial quantization and the target pass. `mlp.down_proj` dominates the fit (≈60 s) because `J` is large and the batched Cholesky scales as `(2J)³`. * **Memory.** Per parameter ≈ 13 B (student weight, decoded cache, codes, temporaries), plus per input group a `d × d` Gram matrix and a `d × m` cross-Gram per projection. The largest object is the Gram of `ffn_down` (about 1.2 GB at `d = 17408`). * **What the fit does to fit on a 12 GB card:** decoded baseline copies are moved to host RAM during the fit; during the last group the finished weights are parked in host RAM; the solve picks its row batch from free GPU memory and retries with half the rows on out-of-memory. ### The CUDA "allocation failed" warning ``` [W...] CUDACachingAllocator.cpp: memory allocation failed with OOM ... ``` appearing during `mlp.down_proj` is **harmless**. It is PyTorch's allocator warning before the solve's retry path halves the row batch. If the run actually fails, you get a Python exception. In that case try, in this order: `--micro-batch 1`, a smaller `--seq-len`, then `train_types=`. --- ## 8. Known limitations 1. **Codes are not output-aware.** The solve only moves scales and offsets around codes that were chosen by weight-space rounding. Most of the remaining error comes from here. A GPTQ-style or coordinate-descent update of the codes against `C` is the natural next step; the Gram matrices it needs are already computed. 2. **Norm weights are not co-adapted**, unlike the old trainer. 3. **Sequential, not joint.** Each group is fitted against the teacher output of its own layer, not against the block output. Error from earlier groups is compensated only through the student inputs. 4. **The snap is a heuristic projection**, not a constrained optimum, so the unconstrained least-squares improvement is only partly retained. The guard prevents regressions but does not recover that loss. 5. **Weight-space code re-derivation** (round ≥ 1) can propose codes that are worse for the output; the guard filters them, but it can mean round 2 gains little. `verbose=True` shows the gain per round; use `ls_iters=1` to cut fitting time if it is negligible. 6. **Layout assumption.** Quantization groups are formed in the HF tensor layout, as in the previous trainer. Any architecture-specific column reordering between the HF and GGUF layouts is not modeled. 7. **Dense architectures only** (no mixture-of-experts handling). 8. **Overfitting protection is limited to the guard.** With few sequences relative to `2J` unknowns per row (large `d`), increase `--num-seqs` or `ls_ridge`. --- ## 9. Tuning guide | Symptom | Try | |---|---| | FINAL much worse than ALGEBRAIC | The converter's quantizer is not reproducing the lattice points; inspect the weight-RMSE line and the per-tensor row counts | | Many rows rejected by the guard | Raise `ls_ridge` (more shrinkage toward the baseline) or use more sequences | | Almost no rows rejected and gains plateau | Lower `ls_ridge` (e.g. `1e-4`) or raise `ls_iters` to 3 | | Round 2 gains almost nothing | `ls_iters=1` to save time | | Out of memory | `--micro-batch 1`, smaller `--seq-len`, `train_types=` | | FINAL error creeps up with depth | Compare BASELINE→FINAL per block; consider the code-refinement step (§8.1) |