Runtime card (private, pre-flip)

README.md

@@ -0,0 +1,68 @@
+---
+license: mit
+library_name: gguf
+tags:
+  - fraqtl
+  - llama-cpp
+  - kv-cache-compression
+  - cuda
+  - inference-runtime
+---
+
+# fraQtl Membrane Runtime for llama.cpp (prebuilt, CUDA)
+
+Prebuilt binaries of the fraQtl **compression membrane** integrated into a
+llama.cpp fork: the KV cache is stored in calibrated compressed pages and the
+attention kernel reads them **directly at tensor-core speed** — no
+decompress-then-attend stage. At long context, where decode is
+bandwidth-bound, reading fewer bytes makes generation faster.
+
+Measured (receipts on the model cards): **1.79× faster than q8_0 KV at 128K
+decode** on Qwen3-4B (92% of fp16 speed at ~2.45× less KV memory), **36 vs 28
+users** before OOM at 32K/user — and on Mistral-Nemo-12B, 32 vs 24 users,
+second model with zero code changes.
+
+## Platform
+
+- **Linux x86_64 · CUDA 12.4 · SM80 + SM90** (A100 / H100 class). This is
+  the server/workstation lane — no macOS/ARM build.
+- Built 2026-08-24 (`MANIFEST.json` in this repo carries per-file sha256 —
+  verify your download against it).
+
+## Files
+
+`llama-server`, `llama-cli`, `llama-fraqtl-niah-parallel` (the retrieval-gate
+harness) and 7 shared libraries (`libllama`, `libllama-common`, `libggml`,
+`-base`, `-cpu`, `-cuda`, `libmtmd`). Kernel source is not distributed — same
+posture as our [vLLM runtime wheel](https://huggingface.co/fraQtl/fraqtl-sm80-runtime);
+the binaries plus published sidecars are sufficient to run and verify every
+number on the model cards.
+
+## How to run
+
+```bash
+export FRAQTL_MEMBRANE=1 FRAQTL_MEMBRANE_EXCLUSIVE=1
+LD_LIBRARY_PATH=. ./llama-server -m <model>.gguf \
+  --fraqtl-kv --fraqtl-eigenbasis <v-sidecar>.bin \
+  --fraqtl-kv-protect 32 --fraqtl-k-eigenbasis <k-sidecar>.bin \
+  --fraqtl-sink-tokens 0 --fraqtl-residual-window 0
+```
+
+Calibrated sidecars per model:
+[Qwen3-4B-Instruct-2507](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars) ·
+[Mistral-Nemo-Instruct-2407](https://huggingface.co/fraQtl/mistral-nemo-instruct-2407-kv-sidecars)
+
+## License and credit
+
+This runtime is a fork of [llama.cpp](https://github.com/ggml-org/llama.cpp)
+(MIT — license included; upstream commit pinned in the receipts). The
+membrane kernels are fraQtl's; llama.cpp and its contributors are credited
+at their best — the engine this builds on is excellent.
+
+## More from fraQtl
+
+Same membrane, independently implemented in vLLM: nine concurrent ≈128K
+users on one A100, 134.1 tok/s, 9/9 retrieval — receipt on the
+[Qwen3-4B sidecar card](https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars).
+Calibration-aware Hi-Fi GGUFs (pair well with this runtime):
+[org page](https://huggingface.co/fraQtl).