Update KV-cache card with accurate template and fork requirements

README.md

@@ -1,101 +1,135 @@
 ---
+license: mit
 base_model: deepseek-ai/DeepSeek-V3.2
-library_name: transformers
 tags:
   - rotorquant
   - kv-cache-quantization
   - deepseek
   - moe
   - quantized
-  - text-generation
-  - mixture-of-experts
-license: mit
+library_name: transformers
 pipeline_tag: text-generation
 ---
 
 # DeepSeek-V3.2-RotorQuant
 
-**KV cache quantization for DeepSeek-V3.2 using RotorQuant compression.**
+**RotorQuant KV cache compression** for [deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2).
 
-This repository provides RotorQuant-compressed KV cache configurations for [deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2), one of the most capable open-weight large language models available. RotorQuant achieves 5.3x faster prefill and 28% faster decode while maintaining near-lossless quality.
+This is a **documentation repository** that explains how to combine DeepSeek-V3.2's weights with RotorQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply RotorQuant via the Python package or llama.cpp fork.
 
-## Overview
+## What is this?
 
-| Attribute | Value |
-|-----------|-------|
-| Base model | [deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) |
-| Architecture | Mixture of Experts (MoE) |
-| Total parameters | ~671B |
-| Compression | RotorQuant KV cache |
-| Perplexity | 6.91 (vs 7.07 baseline) |
-| Prefill speedup | 5.3x |
-| Decode speedup | 1.28x (28% faster) |
-| License | MIT |
-| Task | Text generation |
+KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.
+
+| Technique | Where it's applied | Savings |
+|-----------|-------------------|---------|
+| Weight quantization (GGUF/MLX/AWQ) | Baked into model file | Reduces disk + weight memory |
+| **RotorQuant KV cache** | At inference time | Reduces attention memory (critical for long context) |
+
+Both can be combined for maximum efficiency.
 
 ## Quickstart
 
+### Option A — Python / transformers
+
+Install the `rotorquant` package:
+
+```bash
+pip install rotorquant
+```
+
+Then use it with the base model:
+
 ```python
+import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
-from turboquant import IsoQuantCache
-
-model_id = "deepseek-ai/DeepSeek-V3.2"
+from rotorquant import IsoQuantCache
 
-tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.2", trust_remote_code=True)
 model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    trust_remote_code=True,
+    "deepseek-ai/DeepSeek-V3.2",
+    torch_dtype=torch.bfloat16,
     device_map="auto",
-    torch_dtype="auto",
+    trust_remote_code=True,
 )
 
-# Apply RotorQuant KV cache compression
-cache = IsoQuantCache(
-    model,
-    residual_length=128,
-)
+# Apply RotorQuant to the KV cache
+cache = IsoQuantCache(bits=4)  # or bits=2 for more aggressive compression
 
-inputs = tokenizer("Explain mixture of experts architectures.", return_tensors="pt").to(model.device)
+inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
 outputs = model.generate(
     **inputs,
+    max_new_tokens=128,
     past_key_values=cache,
-    max_new_tokens=512,
+    use_cache=True,
 )
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
+
+
+### Option B — llama.cpp / LM Studio / Ollama (with fork)
+
+RotorQuant KV cache types (`iso3`) are **not** in upstream llama.cpp. They require:
+- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
+
+Once built:
+
+```bash
+llama-cli -m DeepSeek-V3.2.gguf \
+  --cache-type-k iso3 --cache-type-v iso3 \
+  -ngl 99 -fa \
+  -p "Hello"
 ```
 
+For standard runtimes (LM Studio, Ollama, upstream llama.cpp), use conventional KV cache types (`q8_0`, `q4_0`). You lose the RotorQuant-specific benefits but keep GGUF weight quantization.
+
+## Model Specifications
+
+| Property | Value |
+|----------|-------|
+| Base Model | [deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) |
+| Architecture | Sparse MoE |
+| Parameters | 671B total (MoE) |
+| Context Length | 128K |
+| BF16 Size | ~1340 GB |
+| Modalities | Text |
+| License | mit |
+
 ## What is RotorQuant?
 
-[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache quantization method that applies rotation-based transformations to compress the key-value cache during autoregressive generation. It achieves substantial speedups in both prefill and decode stages while actually improving perplexity in some configurations.
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors — a faster, more parameter-efficient alternative to Google's TurboQuant. Uses lightweight block-diagonal rotations (independent 2D/4D rotations per pair/quartet) achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.
+
+**Benchmarks** (from the RotorQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):
 
-### Performance Comparison
+- Prefill: 3,822 tok/s (vs TurboQuant 722 tok/s)
+- Decode: 119 tok/s (vs TurboQuant 93 tok/s)
+- Perplexity: 6.91 (vs TurboQuant 7.07)
+- Parameters: 4 per rotor (vs TurboQuant 16,384)
 
-| Metric | Baseline (FP16 KV) | RotorQuant |
-|--------|-------------------|------------|
-| Perplexity | 7.07 | 6.91 |
-| Prefill speed | 1.0x | 5.3x |
-| Decode speed | 1.0x | 1.28x |
-| KV cache memory | 100% | Substantially reduced |
+> Benchmarks are from the RotorQuant repository using Llama 3.1 8B. Performance on DeepSeek-V3.2 will differ. Please open a discussion if you have independent results.
 
-### Why KV Cache Compression Matters for DeepSeek-V3.2
+## Current Ecosystem Support
 
-DeepSeek-V3.2 is a massive 671B-parameter MoE model. At long context lengths, the KV cache can consume hundreds of gigabytes of memory. RotorQuant makes it feasible to serve this model with substantially reduced hardware requirements and faster throughput, especially for long-context workloads.
+| Runtime | RotorQuant Support | Notes |
+|---------|----------------------|-------|
+| Python transformers + `rotorquant` | ✅ Full | Drop-in cache class |
+| llama.cpp upstream | ❌ Not merged | Use fork below |
+| llama-cpp-turboquant fork | ✅ `planar3`, `iso3` | [GitHub](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache) |
+| LM Studio | ❌ [Requested](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) | Use `q8_0` as alternative |
+| Ollama | ❌ Not supported | Use `OLLAMA_KV_CACHE_TYPE=q8_0` |
+| vLLM | ❌ Not supported | — |
+| koboldcpp | ❌ Not supported | — |
 
-## Memory Estimates
+## Pre-quantized weight variants
 
-| Configuration | Approximate Size |
-|---------------|-----------------|
-| FP16 weights | ~1.3 TB |
-| FP8 weights (base) | ~671 GB |
-| KV cache (FP16, 128K context) | Very large -- scales with sequence length |
-| KV cache (RotorQuant) | Substantial reduction vs FP16 cache |
+If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:
 
-Note: RotorQuant compresses the KV cache only. Model weights remain in their original precision. For weight quantization, see the MLX variants below.
+- [MLX (Apple Silicon)](https://huggingface.co/majentik?search=DeepSeek-V3.2+MLX)
+- [GGUF (llama.cpp / Ollama / LM Studio)](https://huggingface.co/majentik?search=DeepSeek-V3.2+GGUF)
 
 ## See Also
 
-- [deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) -- Base model
-- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant) -- Source code and documentation
-- [majentik/DeepSeek-V3.2-TurboQuant](https://huggingface.co/majentik/DeepSeek-V3.2-TurboQuant) -- Alternative KV cache compression
-- [majentik/DeepSeek-V3.2-RotorQuant-MLX-2bit](https://huggingface.co/majentik/DeepSeek-V3.2-RotorQuant-MLX-2bit) -- MLX 2-bit weight quant + RotorQuant
-- [majentik/DeepSeek-V3.2-RotorQuant-MLX-1bit](https://huggingface.co/majentik/DeepSeek-V3.2-RotorQuant-MLX-1bit) -- MLX 1-bit weight quant + RotorQuant
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
+- [TurboQuant paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)
+- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
+- [Base model: deepseek-ai/DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)