USAGE.md

# ASHQ1 Suite User Manual & Workflow Guide

A step-by-step practical guide for processing unquantized checkpoints into production-ready ASHQ1 GGUF models.

---

## 📁 1. Directory Structure Setup

Place the suite scripts in a unified working directory:

```
workspace/
├── 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py
├── 00b_BF16-GGUF-MTP-extract.py
├── 01_create-calibration-dataset-and-imatrix.py
├── 01b_BF16-GGUF-modules-fusion.py
├── 02_BF16-GGUF-to-ASHQ1.py
├── 03_perplexity_test.py
├── ASHQ1.py
├── ASHQ1-mmproj.py
├── llama-cpp/                # Cloned or linked llama.cpp repository
│   ├── convert_hf_to_gguf.py
│   ├── llama-quantize.exe
│   └── llama-imatrix.exe
└── safetensors/              # HuggingFace model source files
    ├── config.json
    ├── model.safetensors.index.json
    └── *.safetensors
```

---

## 🚀 2. Step-by-Step Execution Workflow

### Step 0: Optimize Weights & Build Pristine BF16 GGUF
Run script `00` to optimize the raw model via AutoRound W4A16 and produce initial GGUF files:

```bash
# Standard execution (reads ./safetensors by default)
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py

# Explicit path targeting
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py /path/to/my-model

# Advanced flags
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py --iters 50 --batch-size 4 --mtp-gguf
```

**Generated Outputs**:
* `model-BF16.gguf`: Complete full model (including MTP head if present).
* `model-no-mtp-BF16.gguf`: Base model trunk (used for calibration calculation).
* `model-BF16.provenance.json`: Lineage record certifying AutoRound conditioning.
* `mmproj-BF16.gguf`: Vision projector (for multimodal checkpoints).

---

### Step 1: Compute Activation Importance Matrix (Imatrix)
Run script `01` to construct the multi-source calibration dataset and calculate activation statistics:

```bash
# Standard workflow using default balanced calibration corpus
python 01_create-calibration-dataset-and-imatrix.py

# Use advanced reasoning corpus (experimental.txt)
python 01_create-calibration-dataset-and-imatrix.py --experimental

# Force execution on CPU for systems without dedicated GPU VRAM
python 01_create-calibration-dataset-and-imatrix.py --cpu-only
```

**Generated Outputs**:
* `imatrix.gguf`: Activation variance table for each layer. An `imatrix.dat` already present in the directory is reused and auto-discovered, and every additional `*.imatrix` / `*imatrix*.gguf` file found is merged by max reduction.

---

### Step 2: Batch Quantization to ASHQ1 Tiers
Run script `02` to orchestrate multi-tier quantization:

```bash
# Quantize all standard tiers (Nano, Mini, Compact on AutoRound int4; + Quality on plain BF16) + mmproj
python 02_BF16-GGUF-to-ASHQ1.py
```

**Targeted Tier Generation**:
To generate a single tier directly via `ASHQ1.py`:
```bash
python ASHQ1.py --model model-BF16.gguf --imatrix imatrix.gguf --tier quality --run
```

---

### Step 3: Perplexity Evaluation (`03_perplexity_test.py`)

Interactive CLI tool to benchmark GGUF files against `wiki.test.raw` (auto-downloaded from HuggingFace) or a local corpus:

- Auto-detects NVIDIA (CUDA) and AMD (ROCm) hardware and available VRAM.
- Automatically sets `-ngl`, `-b 512`, `-ub 512`, and `-fa` (Flash-Attention).
- Real-time ETA and chunk progression streaming.
- Generates a comparative summary table with Δ PPL.

The 2026-08-20 Nano validation used `model-AutoRound-ASHQ1-Nano-24pc.gguf` (4,106 MiB; quantizer-reported 3.84 BPW) and measured PPL **10.3148** at **1,190.7 tok/s** in **90.3 s** with `ctx=2048`, 64 chunks, batch 512, 15 threads, and Flash-Attention.

---

## 🎯 3. Recommended Minimum Tiers by Model Size

Smaller parameter architectures require higher relative bit precision to prevent degradation of core reasoning representations:

* **≥ 9B Models**: Select **Mini** (27% ratio) or higher. Large parameter capacity preserves semantic integrity at lower bit rates.
* **~ 4B Models**: Select **Compact** (33% ratio) or higher. Optimal balance between memory footprint and dense layer preservation.
* **~ 3B Models**: Select **Quality** (39% ratio, 36% on AutoRound int4 lineage) or higher. Higher baseline precision protects critical routing and attention projections.
* **≤ 1B Models**: Select **Fidelity** (48% ratio) or higher. Compact architectures require maximum parameter density.

---

## 🎛️ 4. Environment Variables & Overrides

Configure runtime behavior via optional environment variables:

| Variable | Values | Purpose |
| :--- | :---: | :--- |
| `ASHQ1_LINEAGE` | `auto`, `autoround`, `plain` | Overrides the file naming lineage tag. |
| `ASHQ1_INCLUDE_QUALITY` | `1`, `0` | Forces generation of the 36% Quality tier on int4 lineage models. |
| `ASHQ1_INCLUDE_FIDELITY` | `1`, `0` | Forces generation of the 48% Fidelity tier on int4 lineage models. |
| `GGML_CUDA_ENABLE_UNIFIED_MEMORY` | `1`, `0` | Enables CUDA unified memory for large activation processing in `llama-imatrix`. |
| `LLAMA_CPP_DIR` | `/path/to/llama.cpp` | Specifies custom path to `llama.cpp` binaries. |

---

## 🧩 5. Module Fusion & Specialized Utilities

### Fusing Multimodal & Speculative Heads (`01b`)
Combine standalone models with vision projectors and speculative heads into a unified deployable GGUF:
```bash
python 01b_BF16-GGUF-modules-fusion.py fused-model-Quality.gguf model-AutoRound-ASHQ1-Quality-39pc.gguf mmproj-ASHQ1-Balanced-72pc.gguf
```

### Standalone MTP Head Extraction (`00b`)
Extract the NextN speculative draft layer from any full GGUF:
```bash
python 00b_BF16-GGUF-MTP-extract.py model-BF16.gguf model-no-mtp-BF16.gguf mtp-BF16.gguf --keep-index
```

### Standalone Vision Quantization (`ASHQ1-mmproj`)
Quantize vision projectors with customized preservation profiles:
```bash
python ASHQ1-mmproj.py --model mmproj-BF16.gguf --profile balanced --deep-boost 3
```

---

## 💡 6. Best Practices for Deployment

1. **Host-Side Embedding Offload**:
   ASHQ1 places the input token embedding in `Q8_0` (or `Q5_K` on int4 lineage) outside the VRAM budget, leveraging system RAM bandwidth during prompt ingestion.
2. **Context Cache Quantization**:
   Pair ASHQ1 models with quantized key-value caches to preserve GPU memory headroom across long context windows:
   ```bash
   llama-server -m model-AutoRound-ASHQ1-Quality-39pc.gguf -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99
   ```
3. **Speculative Decoding**:
   Serve the extracted `mtp-*.gguf` alongside the main model to achieve high-speed speculative draft verification:
   ```bash
   llama-server -m model-AutoRound-ASHQ1-Quality-39pc.gguf --spec-type draft-mtp --spec-draft-n-max 6 -c 16384 -ngl 99
   ```