# The Geeked Out Quantizer

## What Is It?

**The Geeked Out Quantizer** is a production-ready quantization environment built for Windows systems. It specializes in extreme model compression using importance-aware quantization techniques, particularly the IQ2_M format which achieves 16x compression with minimal quality loss.

## The Mission

Traditional model quantization forces a choice: small file size or good quality. The Geeked Out Quantizer breaks this trade-off by using **importance matrices** — statistical analysis that identifies which weights matter most, allowing intelligent bit allocation.

## Core Capabilities

### 🎯 Importance-Aware Quantization
- Generates importance matrices automatically using calibration data
- Allocates precision where it matters most
- Achieves 2-bit quantization with only 3-8% quality loss

### ⚡ Hardware Optimization
- Auto-detects CPU, memory type (DDR4/DDR5), and GPU capabilities
- Optimizes thread counts and processing parameters
- GPU acceleration for 5-10x speedup on imatrix generation
- CUDA 12.4+ support with dynamic GPU layer offloading

### 🧠 Intelligent Memory Management
- Reserves system RAM to keep Windows responsive during conversion
- Monitors memory pressure and auto-pauses when needed
- Configurable retry logic for transient resource constraints

### 📦 Complete Workflow Support
- Scans directories for valid source models
- Selects optimal source format (BF16 > F16 > F32)
- Handles sharded models while preserving structure
- Batch processing for multiple models
- Desktop GUI for interactive use

## Quantization Pipeline

```
Source Model (BF16/F16)
        ↓
Calibration Data Analysis
        ↓
Importance Matrix Generation
        ↓
Smart Bit Allocation
        ↓
IQ2_M Quantization
        ↓
Quality Verification
        ↓
Production-Ready Model (16x smaller)
```

## Supported Formats

### Importance-Aware (IMatrix Required)
| Format | Bits/Weight | Best For |
|--------|-------------|----------|
| IQ1_M | 1.0 | Ultra-compact mobile/edge |
| IQ2_XXS | 2.0 | Maximum compression |
| IQ2_XS | 2.0 | Balanced compression |
| **IQ2_M** | **2.0** | **Best quality 2-bit** ⭐ |
| IQ2_S | 2.0 | Higher quality, slower |
| IQ3_M | 3.0 | Near-Q4 quality |
| IQ4_XS | 4.0 | Importance-aware 4-bit |

### Standard K-Quant Formats
Q2_K, Q3_K variants, Q4 variants, Q5 variants, Q6_K, Q8_0

### Ternary Formats
TQ2_0, TQ1_0 — experimental 3-value quantization

## Why IQ2_M?

IQ2_M represents the sweet spot for extreme quantization:

- **16x smaller** than FP32 models
- **2-3x faster** inference
- **VRAM usage** reduced to ~1/16th
- **Quality** approaches Q4_K with proper imatrix
- **Compatible** with llama.cpp inference stack

## Use Cases

- 🤖 **Edge AI** — Run large models on limited hardware
- 🌐 **Browser-Based Inference** — Smaller models for WebGPU/WebGL
- 📱 **Mobile Deployment** — Fit large models on phones/tablets
- 🚀 **High-Throughput APIs** — Serve more requests with less VRAM
- 💾 **Archive Storage** — Preserve models at minimal storage cost

## Technical Philosophy

The Geeked Out Quantizer focuses on:

1. **Quality Preservation** — Never sacrifice more quality than necessary
2. **Automation** — Minimize manual tuning through intelligent defaults
3. **Hardware Awareness** — Adapt to the system's capabilities
4. **Production Ready** — Robust error handling and retry logic
5. **Calibration Quality** — Emphasize representative data selection

## Model Curation

Not all models are equal candidates. The quantizer evaluates:
- Source format quality (BF16 preferred)
- Model architecture compatibility
- Existing quantization state
- Expected use case alignment

## Calibration Best Practices

The quality of your quantized model depends heavily on calibration data:

✅ **DO:**
- Use domain-relevant text (code for code models, medical for medical models)
- Include diverse topics and writing styles
- Provide 100-500 chunks of typical document length
- Ensure natural token distribution

❌ **DON'T:**
- Use repetitive or overly simple text
- Include corrupted or random data
- Rely on single-domain text for general-purpose models

## Collaboration & Research

The Geeked Out Quantizer methodology is available for:
- Research collaborations on quantization techniques
- Edge deployment optimization projects
- Custom calibration strategies for specialized domains
- Hardware-specific optimization studies

## Community

All models in this Hugging Face profile are quantized using this toolchain. Each model card includes:
- Quantization specifications
- Calibration methodology
- Quality metrics
- Use case recommendations

## Future Directions

- Expanded format support (new GGML quantization types)
- Domain-specific calibration datasets
- Hardware-specific optimization profiles
- Batch processing automation

---

*The Geeked Out Quantizer: Making extreme compression intelligent.*

For questions about quantization methodology, collaboration opportunities, or technical discussions, please open an issue or discussion on any model in this profile.