| --- |
| license: mit |
| library_name: gguf |
| tags: |
| - fraqtl |
| - llama-cpp |
| - kv-cache-compression |
| - cuda |
| - inference-runtime |
| --- |
| |
| # fraQtl Membrane Runtime for llama.cpp (prebuilt, CUDA) |
|
|
| Prebuilt binaries of the fraQtl **compression membrane** integrated into a |
| llama.cpp fork: the KV cache is stored in calibrated compressed pages and the |
| attention kernel reads them **directly at tensor-core speed** — no |
| decompress-then-attend stage. At long context, where decode is |
| bandwidth-bound, reading fewer bytes makes generation faster. |
|
|
| Measured (receipts on the model cards): **1.79× faster than q8_0 KV at 128K |
| decode** on Qwen3-4B (92% of fp16 speed at ~2.45× less KV memory), **36 vs 28 |
| users** before OOM at 32K/user — and on Mistral-Nemo-12B, 32 vs 24 users, |
| second model with zero code changes. |
| |
| ## Platform |
| |
| - **Linux x86_64 · CUDA 12.4 · SM80 + SM90** (A100 / H100 class). This is |
| the server/workstation lane — no macOS/ARM build. |
| - Built 2026-08-24 (`MANIFEST.json` in this repo carries per-file sha256 — |
| verify your download against it). |
|
|
| ## Files |
|
|
| `llama-server`, `llama-cli`, `llama-fraqtl-niah-parallel` (the retrieval-gate |
| harness) and 7 shared libraries (`libllama`, `libllama-common`, `libggml`, |
| `-base`, `-cpu`, `-cuda`, `libmtmd`). Kernel source is not distributed — same |
| posture as our [vLLM runtime wheel](https://huggingface.co/fraQtl/fraqtl-sm80-runtime); |
| the binaries plus published sidecars are sufficient to run and verify every |
| number on the model cards. |
|
|
| ## How to run |
|
|
| ```bash |
| export FRAQTL_MEMBRANE=1 FRAQTL_MEMBRANE_EXCLUSIVE=1 |
| LD_LIBRARY_PATH=. ./llama-server -m <model>.gguf \ |
| --fraqtl-kv --fraqtl-eigenbasis <v-sidecar>.bin \ |
| --fraqtl-kv-protect 32 --fraqtl-k-eigenbasis <k-sidecar>.bin \ |
| --fraqtl-sink-tokens 0 --fraqtl-residual-window 0 |
| ``` |
|
|
| Calibrated sidecars per model: |
| [Qwen3-4B-Instruct-2507](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars) · |
| [Mistral-Nemo-Instruct-2407](https://huggingface.co/fraQtl/mistral-nemo-instruct-2407-kv-sidecars) |
|
|
| ## License and credit |
|
|
| This runtime is a fork of [llama.cpp](https://github.com/ggml-org/llama.cpp) |
| (MIT — license included; upstream commit pinned in the receipts). The |
| membrane kernels are fraQtl's; llama.cpp and its contributors are credited |
| at their best — the engine this builds on is excellent. |
|
|
| ## More from fraQtl |
|
|
| Same membrane, independently implemented in vLLM: nine concurrent ≈128K |
| users on one A100, 134.1 tok/s, 9/9 retrieval — receipt on the |
| [Qwen3-4B sidecar card](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars). |
| Calibration-aware Hi-Fi GGUFs (pair well with this runtime): |
| [org page](https://huggingface.co/fraQtl). |
|
|
