QUANTIZATION_NOTES.md

# Quantization Notes

## Overview

This model was quantized using **The Geeked Out Quantizer**, a specialized Windows-native quantization environment designed for extreme compression with quality preservation.

## Quantization Specifications

| Parameter | Details |
|-----------|---------|
| **Source Format** | BF16 (bfloat16) or F16 (float16) |
| **Target Format** | IQ2_M (2.0 bits per weight) |
| **Compression Ratio** | 16x smaller than FP32 baseline |
| **Quantization Method** | Importance-aware quantization with IMatrix |
| **Quality Metric** | ~3-8% perplexity increase vs. baseline |

## The Importance Matrix (IMatrix) Method

### What is an Importance Matrix?

An importance matrix is a statistical analysis of a neural network that identifies which weights contribute most significantly to model output quality. Rather than applying uniform quantization across all tensors, this method:

- **Preserves precision** on high-impact weights
- **Aggressively compresses** low-impact weights
- **Maintains information flow** through the network architecture

### Why It Matters

Traditional uniform quantization to 2-bit precision typically causes 10-20% quality degradation. The importance matrix approach reduces this to 3-8%, making 2-bit models viable for production use.

## Calibration Process

### Data Selection

The importance matrix is generated using carefully selected calibration data that:
- Represents the model's intended use domain
- Contains diverse vocabulary and sentence structures
- Includes 100-500 text chunks of typical prompt length
- Matches the distribution of expected inference inputs

### Generation Parameters

| Setting | Typical Value | Purpose |
|---------|---------------|---------|
| Chunks | 200-500 | Balance quality vs. generation time |
| GPU Layers | 99 (max) | Accelerate processing via CUDA |
| Thread Count | Auto-detected | Optimize for hardware configuration |

## Memory & Hardware Optimization

The quantization process includes:
- **Dynamic memory management** — Reserves system RAM to maintain Windows responsiveness
- **Hardware detection** — Automatically detects CPU cores, memory type (DDR4/DDR5), and GPU capabilities
- **Thread optimization** — Adjusts parallelism based on available resources
- **Retry logic** — Handles transient memory pressure gracefully

## Model Selection Criteria

Source models are selected based on quality hierarchy:
1. **BF16** (preferred) — Best precision for quantization
2. **F16** — Good precision, widely available
3. **F32** — Acceptable but creates larger intermediate files

Models already in quantized formats are skipped unless explicitly re-quantizing.

## Output Format Details

### IQ2_M Characteristics

- **Bit depth:** 2.0 bits per weight
- **Speed:** 2-3x faster inference than F32
- **VRAM usage:** ~1/16th of FP32
- **Imatrix required:** Yes
- **Quality tier:** Best-in-class for 2-bit quantization

### Naming Convention

Quantized models follow this pattern:
```
OriginalModel-BF16.gguf → OriginalModel-IQ2_M.gguf
```

Sharded models preserve shard numbering:
```
Model-00001-of-00004.gguf → Model-IQ2_M-00001-of-00004.gguf
```

## Quality Verification

Models are validated through:
- Perplexity measurement against baseline
- Sample inference testing
- File integrity verification

## Use Cases

IQ2_M quantized models are ideal for:
- **Edge deployment** — Minimal storage footprint
- **Consumer hardware** — Reduced VRAM requirements
- **High-throughput inference** — Faster token generation
- **Bandwidth-constrained environments** — Efficient distribution

## Technical Notes

- Quantization performed on Windows with CUDA 12.4+ support
- GPU acceleration utilized for imatrix generation
- Multi-threaded processing with memory safety guards
- Compatible with llama.cpp inference engines

## Citation

If you use this quantized model in research or applications, please acknowledge:

> Quantized using The Geeked Out Quantizer with importance-aware IQ2_M optimization.

---

*For questions about the quantization method or collaboration inquiries, please open a discussion on this model's page.*