File size: 7,098 Bytes
cb4574f 58ce54f cb4574f 58ce54f cb4574f 58ce54f cb4574f 58ce54f ae55163 c2d9522 cb4574f 81df991 cb4574f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # ASHQ1 Technical Charter & Precision Guarantees
This document establishes the architectural principles, mathematical formulation, and invariant guarantees governing the AutoRound + ASHQ1 quantization suite.
---
## π 1. Architectural Principles
### 1.1 Dual-Stage Optimization Model
The suite operates on a two-stage sequential precision boundary:
1. **Weight-Space Reconditioning (AutoRound)**:
Minimizes local layer reconstruction error over calibration activations:
$$\min_{\mathbf{W}_q} \|\mathbf{W}\mathbf{X} - \mathbf{W}_q\mathbf{X}\|_F^2 + \lambda \mathcal{R}(\mathbf{W}_q)$$
where $\mathbf{W}_q$ represents the discrete integer matrix with continuous scale and zero adjustments.
2. **Activation-Space Precision Allocation (ASHQ1)**:
Given an empirical Fisher Information Proxy $\mathbf{F} = \text{diag}(\mathbf{H})$ derived from activation variances in `imatrix.gguf` (legacy `imatrix.dat` files are read as-is), ASHQ1 solves a constrained knapsack problem:
$$\max_{\{t_i\}} \sum_{i} \mathbf{F}_i \cdot \Delta \mathcal{Q}(t_i) \quad \text{subject to} \quad \sum_{i} \text{Size}(t_i, \text{params}_i) \le \text{Budget}_{\text{VRAM}}$$
where $t_i \in \text{TIER\_ORDER}$ and $\Delta \mathcal{Q}(t_i) = 2^{-2 \cdot \text{eff\_bpw}(t_i)}$.
---
## π‘οΈ 2. Invariant Precision Guarantees
To prevent numerical drift and safeguard autoregressive context integrity, ASHQ1 enforces absolute tensor classification floors:
```
ββββββββββββββββββββββββββββββββ¬ββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββ
β Tensor Class β Minimum Floor β Rationale β
ββββββββββββββββββββββββββββββββΌββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββ€
β Norms & Scales (ln, rms) β F16 / F32 β Preserves dynamic scale variance β
β Attention Gates (gate_proj) β Q6_K β Prevents non-linear token routing collapse β
β Recurrent State (ssm_alpha) β Q8_0 β Guarantees stable hidden state accumulationβ
β Speculative Heads (MTP/NextN)β Q6_K / Q8_0 β Maximizes draft token acceptance rate β
β Uncalibrated (no imatrix) β IQ4_XS β Keeps llama-quantize within safe tiers β
β Vision Projectors (mmproj) β F32 (critical)β Protects high-resolution visual grounding β
ββββββββββββββββββββββββββββββββ΄ββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββ
```
### 2.1 Pinned Structural Layers
* **1D Vectors & Bias Tensors**: All vectors with fewer than 4096 elements or non-matrix shapes remain strictly in `F32` or `F16`.
* **Block Alignment Rules**: Tensors must adhere to integer multiples of the GGML block size (32 or 256 elements). Unaligned shapes are automatically promoted to the nearest valid container (`F16`/`Q8_0`).
* **Speculative Block Norms**: `enorm`, `hnorm` and any 1-D vector inside the MTP/NextN block stay pinned in `F32` and receive no tier rule, keeping the draft head numerically aligned with the trunk.
* **Imatrix Coverage Floor**: Every tensor whose statistics are absent from the merged importance matrix holds at `IQ4_XS` or above, both at assignment time and inside the downgrade queue.
### 2.2 Tied-Weight Group Preservation
When tied layers are detected (such as shared input/output embeddings or symmetric MoE gates), ASHQ1 binds the corresponding parameters into unified equivalence sets. The optimizer updates their quantization tiers simultaneously to prevent directional divergence.
---
## π¬ 3. AutoRound Lineage & Saturation Bounds
When weights are derived from an AutoRound W4A16 source (recorded via `.provenance.json` sidecars):
* **Information Saturation**: Because the underlying weights have been pre-conditioned for 4-bit representations, assigning container formats above `Q5_K` yields negligible signal recovery ($\Delta \mathcal{Q} \to 0$).
* **Automated Capping**: The ASHQ1 optimizer caps attention and FFN blocks at `Q5_K`, attention gates at `Q6_K`, and MTP heads at `Q6_K`, while recurrent memory states hold `Q8_0`. The bit-budget freed by these ceilings flows to the imatrix-ranked tensors that convert it into measurable error reduction.
* **Tier Pruning**: On AutoRound int4 lineage, generation of both the 48% `Fidelity` and 36% `Quality` tiers is skipped by default, focusing compute resources on optimal `Compact` (33%), `Mini` (27%), and `Nano` (24% in the 2026-08-20 validation run) variants.
* **Cap Saturation**: On AutoRound int4 lineage, `Compact` (33%) drives every attention and FFN projection to the `Q5_K` cap. Measured on a 9B `qwen35` source (17 091 MiB BF16): `Compact` ships 33.06%, `Quality` ships 35.49% against a 36% target. Beyond `Compact`, the perplexity difference remains negligible across all model sizes (1B to 9B, with Ξ PPL β€ 0.0358 on 9B), as blocks 1 through 31 remain identical between tiers. `Compact` therefore represents the maximum standard tier for this lineage.
* **Discrete Ceiling**: Once every capped class saturates, the upgrade queue holds only large indivisible moves β `output` Q5_KβQ6_K at +129 MiB, `token_embd` Q6_KβQ8_0 at +235 MiB. Reachable footprints become discrete: on the reference 9B source, 35.49%, then 36.25%, then 37.62%. A target landing between two steps delivers the lower one, which the greedy scheduler reports as remaining slack.
---
## π 4. Validation Evidence
The 2026-08-20 validation run measured the AutoRound int4-lineage `Nano-24pc` artifact at **4,106 MiB** on disk, with a quantizer-reported effective size of **3.84 BPW**. On `wiki.test.raw` at `ctx=2048`, 64 chunks, batch 512, 15 threads, and Flash-Attention, it measured **PPL 10.3148** at **1,190.7 tok/s** over **90.3 s**. Against the documented `Quality-36pc` baseline of **8.0932**, this is **+2.2216 PPL**.
## π 5. Calibration Dataset Standard
Calibration quality directly determines activation matrix fidelity. The suite implements an exact multi-domain quota:
* **55% Agentic Execution**: Tool-use traces, JSON schema parameters, and multi-turn function call loops.
* **20% Formal Logic & Code**: Formal mathematical proofs (Lean 4), algorithmic Python implementations, and syntactic structures.
* **15% Semantic Diversity**: Multilingual sequences, encyclopedic knowledge, and conversational discourse.
* **10% Frontier Reasoning**: Extended thinking traces and complex reasoning chains.
All samples undergo token-count round-trip verification to ensure consistent sequence boundaries during `llama-imatrix` evaluation. |