KV cache compression, inference optimization, model compression
fraQtl is an inference company. We work on both ends of the same problem — serving (how many users one GPU can hold at long context) and on-device (how much of a model's behavior survives on a phone or laptop). Same math underneath, same standard on top: every number on every card is measured, gated on retrieval where retrieval is the risk, and backed by a receipt with a run ID you can regenerate yourself.
Latest: nine concurrent ≈128K-context users on one A100 — 4.5× the users fp16 holds without preemption and 1.8× fp8's, at 2× fp16's total decode throughput — 9/9 per-user retrieval checks passed. The receipt →
| You run models... | Start here |
|---|---|
| In production / vLLM — more concurrent users or longer contexts per GPU | Qwen3-4B KV sidecars — the 9-user receipt · Mistral-7B KV sidecars · SM80 runtime wheel |
| On a device / llama.cpp, LM Studio, Ollama — more fidelity at the same file size | Gemma-4-E2B Hi-Fi — runs an offline phone agent, MTP drafting measured · Qwen3.6-35B-A3B Hi-Fi — ≈23% lower KLD than the leading public Q4_K_M at identical size |
| Tier | Job |
|---|---|
| Hi-Fi | maximum fidelity at standard 4-bit size |
| Hi-Fi Phone | fits phone/laptop RAM, fidelity-first |
| SmartEdge | smallest footprint that survives |
A compression membrane between the model and vLLM's paged KV cache: writes store K/V in compressed pages (a small calibrated protected subspace at high precision + an INT4 tail); attention reads the compressed pages directly in-kernel at tensor-core speed — no decompress-then-attend, so capacity doesn't cost decode latency. Calibration is per-layer, per-KV-head; the calibrated sidecars are published, the calibration method is not.
Verification requests and model requests: open a discussion on any repo.
