| # ASHQ1 Technical Charter & Precision Guarantees |
|
|
| This document establishes the architectural principles, mathematical formulation, and invariant guarantees governing the AutoRound + ASHQ1 quantization suite. |
|
|
| --- |
|
|
| ## π 1. Architectural Principles |
|
|
| ### 1.1 Dual-Stage Optimization Model |
| The suite operates on a two-stage sequential precision boundary: |
| 1. **Weight-Space Reconditioning (AutoRound)**: |
| Minimizes local layer reconstruction error over calibration activations: |
| $$\min_{\mathbf{W}_q} \|\mathbf{W}\mathbf{X} - \mathbf{W}_q\mathbf{X}\|_F^2 + \lambda \mathcal{R}(\mathbf{W}_q)$$ |
| where $\mathbf{W}_q$ represents the discrete integer matrix with continuous scale and zero adjustments. |
| 2. **Activation-Space Precision Allocation (ASHQ1)**: |
| Given an empirical Fisher Information Proxy $\mathbf{F} = \text{diag}(\mathbf{H})$ derived from activation variances in `imatrix.gguf` (legacy `imatrix.dat` files are read as-is), ASHQ1 solves a constrained knapsack problem: |
| $$\max_{\{t_i\}} \sum_{i} \mathbf{F}_i \cdot \Delta \mathcal{Q}(t_i) \quad \text{subject to} \quad \sum_{i} \text{Size}(t_i, \text{params}_i) \le \text{Budget}_{\text{VRAM}}$$ |
| where $t_i \in \text{TIER\_ORDER}$ and $\Delta \mathcal{Q}(t_i) = 2^{-2 \cdot \text{eff\_bpw}(t_i)}$. |
|
|
| --- |
|
|
| ## π‘οΈ 2. Invariant Precision Guarantees |
|
|
| To prevent numerical drift and safeguard autoregressive context integrity, ASHQ1 enforces absolute tensor classification floors: |
|
|
| ``` |
| ββββββββββββββββββββββββββββββββ¬ββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββ |
| β Tensor Class β Minimum Floor β Rationale β |
| ββββββββββββββββββββββββββββββββΌββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββ€ |
| β Norms & Scales (ln, rms) β F16 / F32 β Preserves dynamic scale variance β |
| β Attention Gates (gate_proj) β Q6_K β Prevents non-linear token routing collapse β |
| β Recurrent State (ssm_alpha) β Q8_0 β Guarantees stable hidden state accumulationβ |
| β Speculative Heads (MTP/NextN)β Q6_K / Q8_0 β Maximizes draft token acceptance rate β |
| β Uncalibrated (no imatrix) β IQ4_XS β Keeps llama-quantize within safe tiers β |
| β Vision Projectors (mmproj) β F32 (critical)β Protects high-resolution visual grounding β |
| ββββββββββββββββββββββββββββββββ΄ββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββ |
| ``` |
|
|
| ### 2.1 Pinned Structural Layers |
| * **1D Vectors & Bias Tensors**: All vectors with fewer than 4096 elements or non-matrix shapes remain strictly in `F32` or `F16`. |
| * **Block Alignment Rules**: Tensors must adhere to integer multiples of the GGML block size (32 or 256 elements). Unaligned shapes are automatically promoted to the nearest valid container (`F16`/`Q8_0`). |
| * **Speculative Block Norms**: `enorm`, `hnorm` and any 1-D vector inside the MTP/NextN block stay pinned in `F32` and receive no tier rule, keeping the draft head numerically aligned with the trunk. |
| * **Imatrix Coverage Floor**: Every tensor whose statistics are absent from the merged importance matrix holds at `IQ4_XS` or above, both at assignment time and inside the downgrade queue. |
|
|
| ### 2.2 Tied-Weight Group Preservation |
| When tied layers are detected (such as shared input/output embeddings or symmetric MoE gates), ASHQ1 binds the corresponding parameters into unified equivalence sets. The optimizer updates their quantization tiers simultaneously to prevent directional divergence. |
|
|
| --- |
|
|
| ## π¬ 3. AutoRound Lineage & Saturation Bounds |
|
|
| When weights are derived from an AutoRound W4A16 source (recorded via `.provenance.json` sidecars): |
| * **Information Saturation**: Because the underlying weights have been pre-conditioned for 4-bit representations, assigning container formats above `Q5_K` yields negligible signal recovery ($\Delta \mathcal{Q} \to 0$). |
| * **Automated Capping**: The ASHQ1 optimizer caps attention and FFN blocks at `Q5_K`, attention gates at `Q6_K`, and MTP heads at `Q6_K`, while recurrent memory states hold `Q8_0`. The bit-budget freed by these ceilings flows to the imatrix-ranked tensors that convert it into measurable error reduction. |
| * **Tier Pruning**: On AutoRound int4 lineage, generation of both the 48% `Fidelity` and 36% `Quality` tiers is skipped by default, focusing compute resources on optimal `Compact` (33%), `Mini` (27%), and `Nano` (24% in the 2026-08-20 validation run) variants. |
| * **Cap Saturation**: On AutoRound int4 lineage, `Compact` (33%) drives every attention and FFN projection to the `Q5_K` cap. Measured on a 9B `qwen35` source (17 091 MiB BF16): `Compact` ships 33.06%, `Quality` ships 35.49% against a 36% target. Beyond `Compact`, the perplexity difference remains negligible across all model sizes (1B to 9B, with Ξ PPL β€ 0.0358 on 9B), as blocks 1 through 31 remain identical between tiers. `Compact` therefore represents the maximum standard tier for this lineage. |
| * **Discrete Ceiling**: Once every capped class saturates, the upgrade queue holds only large indivisible moves β `output` Q5_KβQ6_K at +129 MiB, `token_embd` Q6_KβQ8_0 at +235 MiB. Reachable footprints become discrete: on the reference 9B source, 35.49%, then 36.25%, then 37.62%. A target landing between two steps delivers the lower one, which the greedy scheduler reports as remaining slack. |
|
|
| --- |
|
|
| ## π 4. Validation Evidence |
|
|
| The 2026-08-20 validation run measured the AutoRound int4-lineage `Nano-24pc` artifact at **4,106 MiB** on disk, with a quantizer-reported effective size of **3.84 BPW**. On `wiki.test.raw` at `ctx=2048`, 64 chunks, batch 512, 15 threads, and Flash-Attention, it measured **PPL 10.3148** at **1,190.7 tok/s** over **90.3 s**. Against the documented `Quality-36pc` baseline of **8.0932**, this is **+2.2216 PPL**. |
|
|
| ## π 5. Calibration Dataset Standard |
|
|
| Calibration quality directly determines activation matrix fidelity. The suite implements an exact multi-domain quota: |
|
|
| * **55% Agentic Execution**: Tool-use traces, JSON schema parameters, and multi-turn function call loops. |
| * **20% Formal Logic & Code**: Formal mathematical proofs (Lean 4), algorithmic Python implementations, and syntactic structures. |
| * **15% Semantic Diversity**: Multilingual sequences, encyclopedic knowledge, and conversational discourse. |
| * **10% Frontier Reasoning**: Extended thinking traces and complex reasoning chains. |
|
|
| All samples undergo token-count round-trip verification to ensure consistent sequence boundaries during `llama-imatrix` evaluation. |
