USAGE.md

ASHQ1 Suite User Manual & Workflow Guide

A step-by-step practical guide for processing unquantized checkpoints into production-ready ASHQ1 GGUF models.

---

📁 1. Directory Structure Setup

Place the suite scripts in a unified working directory:

workspace/
├── 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py
├── 00b_BF16-GGUF-MTP-extract.py
├── 01_create-calibration-dataset-and-imatrix.py
├── 01b_BF16-GGUF-modules-fusion.py
├── 02_BF16-GGUF-to-ASHQ1.py
├── 03_perplexity_test.py
├── ASHQ1.py
├── ASHQ1-mmproj.py
├── llama-cpp/                # Cloned or linked llama.cpp repository
│   ├── convert_hf_to_gguf.py
│   ├── llama-quantize.exe
│   └── llama-imatrix.exe
└── safetensors/              # HuggingFace model source files
    ├── config.json
    ├── model.safetensors.index.json
    └── *.safetensors

---

🚀 2. Step-by-Step Execution Workflow

Step 0: Optimize Weights & Build Pristine BF16 GGUF

Run script 00 to optimize the raw model via AutoRound W4A16 and produce initial GGUF files:

# Standard execution (reads ./safetensors by default)
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py

# Explicit path targeting
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py /path/to/my-model

# Advanced flags
python 00_SAFETENSORS-to-AutoRound-BF16-GGUF.py --iters 50 --batch-size 4 --mtp-gguf

Generated Outputs:

• model-BF16.gguf: Complete full model (including MTP head if present).
• model-no-mtp-BF16.gguf: Base model trunk (used for calibration calculation).
• model-BF16.provenance.json: Lineage record certifying AutoRound conditioning.
• mmproj-BF16.gguf: Vision projector (for multimodal checkpoints).

---

Step 1: Compute Activation Importance Matrix (Imatrix)

Run script 01 to construct the multi-source calibration dataset and calculate activation statistics:

# Standard workflow using default balanced calibration corpus
python 01_create-calibration-dataset-and-imatrix.py

# Use advanced reasoning corpus (experimental.txt)
python 01_create-calibration-dataset-and-imatrix.py --experimental

# Force execution on CPU for systems without dedicated GPU VRAM
python 01_create-calibration-dataset-and-imatrix.py --cpu-only

Generated Outputs:

• imatrix.gguf: Activation variance table for each layer. An imatrix.dat already present in the directory is reused and auto-discovered, and every additional *.imatrix / *imatrix*.gguf file found is merged by max reduction.

---

Step 2: Batch Quantization to ASHQ1 Tiers

Run script 02 to orchestrate multi-tier quantization:

# Quantize all standard tiers (Nano, Mini, Compact on AutoRound int4; + Quality on plain BF16) + mmproj
python 02_BF16-GGUF-to-ASHQ1.py

Targeted Tier Generation: To generate a single tier directly via ASHQ1.py:

python ASHQ1.py --model model-BF16.gguf --imatrix imatrix.gguf --tier quality --run

---

Step 3: Perplexity Evaluation (03_perplexity_test.py)

Interactive CLI tool to benchmark GGUF files against wiki.test.raw (auto-downloaded from HuggingFace) or a local corpus:

• Auto-detects NVIDIA (CUDA) and AMD (ROCm) hardware and available VRAM.
• Automatically sets -ngl, -b 512, -ub 512, and -fa (Flash-Attention).
• Real-time ETA and chunk progression streaming.
• Generates a comparative summary table with Δ PPL.

The 2026-08-20 Nano validation used model-AutoRound-ASHQ1-Nano-24pc.gguf (4,106 MiB; quantizer-reported 3.84 BPW) and measured PPL 10.3148 at 1,190.7 tok/s in 90.3 s with ctx=2048, 64 chunks, batch 512, 15 threads, and Flash-Attention.

---

🎯 3. Recommended Minimum Tiers by Model Size

Smaller parameter architectures require higher relative bit precision to prevent degradation of core reasoning representations:

• ≥ 9B Models: Select Mini (27% ratio) or higher. Large parameter capacity preserves semantic integrity at lower bit rates.
• ~ 4B Models: Select Compact (33% ratio) or higher. Optimal balance between memory footprint and dense layer preservation.
• ~ 3B Models: Select Quality (39% ratio, 36% on AutoRound int4 lineage) or higher. Higher baseline precision protects critical routing and attention projections.
• ≤ 1B Models: Select Fidelity (48% ratio) or higher. Compact architectures require maximum parameter density.

---

🎛️ 4. Environment Variables & Overrides

Configure runtime behavior via optional environment variables:

Variable	Values	Purpose
ASHQ1_LINEAGE	auto, autoround, plain	Overrides the file naming lineage tag.
ASHQ1_INCLUDE_QUALITY	1, 0	Forces generation of the 36% Quality tier on int4 lineage models.
ASHQ1_INCLUDE_FIDELITY	1, 0	Forces generation of the 48% Fidelity tier on int4 lineage models.
GGML_CUDA_ENABLE_UNIFIED_MEMORY	1, 0	Enables CUDA unified memory for large activation processing in llama-imatrix.
LLAMA_CPP_DIR	/path/to/llama.cpp	Specifies custom path to llama.cpp binaries.

---

🧩 5. Module Fusion & Specialized Utilities

Fusing Multimodal & Speculative Heads (01b)

Combine standalone models with vision projectors and speculative heads into a unified deployable GGUF:

python 01b_BF16-GGUF-modules-fusion.py fused-model-Quality.gguf model-AutoRound-ASHQ1-Quality-39pc.gguf mmproj-ASHQ1-Balanced-72pc.gguf

Standalone MTP Head Extraction (00b)

Extract the NextN speculative draft layer from any full GGUF:

python 00b_BF16-GGUF-MTP-extract.py model-BF16.gguf model-no-mtp-BF16.gguf mtp-BF16.gguf --keep-index

Standalone Vision Quantization (ASHQ1-mmproj)

Quantize vision projectors with customized preservation profiles:

python ASHQ1-mmproj.py --model mmproj-BF16.gguf --profile balanced --deep-boost 3

---

💡 6. Best Practices for Deployment

1. Host-Side Embedding Offload: ASHQ1 places the input token embedding in Q8_0 (or Q5_K on int4 lineage) outside the VRAM budget, leveraging system RAM bandwidth during prompt ingestion.
2. Context Cache Quantization: Pair ASHQ1 models with quantized key-value caches to preserve GPU memory headroom across long context windows:

   llama-server -m model-AutoRound-ASHQ1-Quality-39pc.gguf -c 32768 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99
   

3. Speculative Decoding: Serve the extracted mtp-*.gguf alongside the main model to achieve high-speed speculative draft verification:

   llama-server -m model-AutoRound-ASHQ1-Quality-39pc.gguf --spec-type draft-mtp --spec-draft-n-max 6 -c 16384 -ngl 99