| --- |
| title: fraQtl |
| emoji: ⚡ |
| colorFrom: purple |
| colorTo: indigo |
| sdk: static |
| pinned: true |
| --- |
| |
| # fraQtl |
|
|
| ### Inference efficiency for transformer LLMs — end to end |
|
|
| **KV-cache compression. Weight quantization. Runtime memory optimization. And the diagnostics to measure if it actually helps your stack — before you commit.** |
|
|
| Run larger models, longer contexts, more concurrency, at the same quality. |
|
|
| --- |
|
|
| ## The stack |
|
|
| | Layer | What | Where | |
| |---|---|---| |
| | **KV cache** | live-VRAM reduction at long context via `llama.cpp` integration | runtime | |
| | **Weights** | higher-fidelity GGUF quants (fraQtl calibration) | artifact | |
| | **Runtime memory** | arena allocators, lifetime control, paged layouts | runtime | |
| | **Diagnostic** | drop-in measurement: projected savings, readiness, spectral fingerprint | free, open (Apache 2.0) | |
|
|
| Most "compression" projects ship one of the above. fraQtl ships all four, with reproducible receipts at every layer. |
|
|
| --- |
|
|
| ## What we ship |
|
|
| ### 🧩 Weight quantization — fraQtl calibration |
|
|
| Higher-fidelity GGUF quants. Same file size as a standard Q4_K_M; measurably closer to the full-precision teacher across code, math, chat, tool calling, and long-form text. |
|
|
| Measured on Qwen 3.6 35B-A3B (symmetric top-20 KLD vs the Q8 teacher, 400-record held-out slices): |
|
|
| | Lane | KLD vs Q8 ↓ | Top-1 vs Q8 ↑ | |
| |---|---:|---:| |
| | Code + math | **0.0203** | **97.2%** | |
| | General (chat + tools + long-form text) | **0.0485** | **93.2%** | |
|
|
| At the same file size as a standard Q4_K_M, **~30% lower KLD** across both slices. Reproducibility drift across three independent runs: 0.00000. |
|
|
| → [**Qwen 3.6 35B-A3B (Q4_K_M)**](https://huggingface.co/fraQtl/Qwen3.6-35B-A3B-GGUF) — Q4_K_M · 21.4 GB · drop-in for `llama.cpp` / `Ollama` / `LM Studio` / `koboldcpp` / `Jan` |
|
|
| ### 🛠 KV-cache compression — `llama.cpp` runtime |
|
|
| Measured live-VRAM reduction on Mistral-7B-Instruct-v0.3 Q4_K_M vs a true fp16-KV baseline: |
|
|
| | Context | Live VRAM Δ vs fp16 | % of fp16 KV saved | Quality | |
| |---|---:|---:|---| |
| | 8K | −302 MiB | small | PPL parity | |
| | 64K | **−2,610 MiB** | ~32% | PPL parity | |
| | 128K | **−9,422 MiB** | **~42%** | PPL drift ≤ 0.004 | |
|
|
| The benefit grows with context — exactly where fp16 KV becomes the bottleneck. Reproducibility drift across independent runs: ≤ 0.004 PPL. |
|
|
| ### 🔍 fraQtl Diagnostic — free + open (Apache 2.0) |
|
|
| `pip install fraqtl-diagnostic` — three tools in one package: |
|
|
| - **KV Savings Estimate** — drop in any HF model id → projected memory freed, GPU-tier impact, max-context extension, relative cost-per-token. Instant, no GPU. |
| - **Inference Readiness Scan** — config-level: KV memory, YaRN status, backend support, benchmark checklist. Instant, no GPU. |
| - **Compression Fingerprint** — per-layer spectral analysis (γ, k95, regime tags, Shannon ceiling). |
|
|
| → [**Run in your browser**](https://huggingface.co/spaces/fraQtl/fraqtl-diagnostic) · [**PyPI**](https://pypi.org/project/fraqtl-diagnostic/) · [**GitHub**](https://github.com/fraqtl-ai/fraqtl-diagnostic) |
|
|
| --- |
|
|
| ## Live demos |
|
|
| - 🪶 [**fraqtl-diagnostic**](https://huggingface.co/spaces/fraQtl/fraqtl-diagnostic) — measure your model's compression headroom + projected KV savings in seconds |
| - 🔥 [**fraQtl-demo**](https://huggingface.co/spaces/fraQtl/fraQtl-demo) — fraQtl-compressed Mistral-7B running live with KV-cache compression |
|
|
| --- |
|
|
| ## Approach |
|
|
| - Per-tensor protection policy — same total bit budget, smarter allocation |
| - Calibration tuned to measured optima (not "more is better") |
| - Standard `llama.cpp` kernel path — no patched runtime, no custom flags |
| - Deterministic builds — reproducibility drift `0.00000` across independent runs |
|
|
| The thesis: **compression should be calibration-aware and workload-aware**. |
|
|
| --- |
|
|
| ## Links |
|
|
| - 🌐 Website: [fraqtl.ai](https://fraqtl.ai) |
| - 📦 PyPI: [pypi.org/project/fraqtl-diagnostic](https://pypi.org/project/fraqtl-diagnostic/) |
| - 📄 Paper: [arxiv.org/abs/2604.11501](https://arxiv.org/abs/2604.11501) |
| - 📬 Contact: contact@fraqtl.ai |
|
|
| Patent pending. |
|
|
