Update KV-cache card with accurate template and fork requirements

README.md

@@ -1,87 +1,137 @@
 ---
+license: apache-2.0
 base_model: google/gemma-4-E2B
-library_name: transformers
 tags:
   - rotorquant
   - kv-cache-quantization
   - gemma
   - gemma4
-  - multimodal
+  - edge
   - quantized
-license: apache-2.0
+library_name: transformers
 pipeline_tag: image-text-to-text
 ---
 
-# Gemma 4 E2B - RotorQuant KV Cache
+# gemma-4-E2B-RotorQuant
 
-**RotorQuant KV-cache quantization** applied to [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B), enabling dramatically reduced memory usage during inference without modifying model weights. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant.
+**RotorQuant KV cache compression** for [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B).
 
-This repository provides the RotorQuant KV-cache configuration for Gemma 4 E2B. The model weights remain at their original precision; only the key-value cache is quantized at runtime.
+This is a **documentation repository** that explains how to combine gemma-4-E2B's weights with RotorQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply RotorQuant via the Python package or llama.cpp fork.
 
-## Model Specifications
+## What is this?
 
-| Property | Value |
-|---|---|
-| **Base Model** | [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) |
-| **Parameters** | ~2 billion |
-| **Architecture** | Dense transformer |
-| **Modality** | Multimodal: image + text input, text output |
-| **License** | Apache 2.0 |
-| **Quantization** | RotorQuant KV-cache only (weights unchanged) |
+KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.
+
+| Technique | Where it's applied | Savings |
+|-----------|-------------------|---------|
+| Weight quantization (GGUF/MLX/AWQ) | Baked into model file | Reduces disk + weight memory |
+| **RotorQuant KV cache** | At inference time | Reduces attention memory (critical for long context) |
+
+Both can be combined for maximum efficiency.
 
 ## Quickstart
 
+### Option A — Python / transformers
+
+Install the `rotorquant` package:
+
+```bash
+pip install rotorquant
+```
+
+Then use it with the base model:
+
 ```python
-from rotorquant import RotorQuantCache
-from transformers import AutoModelForImageTextToText, AutoProcessor
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from rotorquant import IsoQuantCache
+
+tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E2B", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    "google/gemma-4-E2B",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+# Apply RotorQuant to the KV cache
+cache = IsoQuantCache(bits=4)  # or bits=2 for more aggressive compression
+
+inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=128,
+    past_key_values=cache,
+    use_cache=True,
+)
+print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
 
-model_id = "google/gemma-4-E2B"
 
-processor = AutoProcessor.from_pretrained(model_id)
-model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
+### Option B — llama.cpp / LM Studio / Ollama (with fork)
 
-# Apply RotorQuant KV-cache quantization
-cache = RotorQuantCache(model)
+RotorQuant KV cache types (`iso3`) are **not** in upstream llama.cpp. They require:
+- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
 
-inputs = processor("Once upon a time", return_tensors="pt").to(model.device)
-outputs = model.generate(**inputs, past_key_values=cache)
-print(processor.decode(outputs[0], skip_special_tokens=True))
+Once built:
+
+```bash
+llama-cli -m gemma-4-E2B.gguf \
+  --cache-type-k iso3 --cache-type-v iso3 \
+  -ngl 99 -fa \
+  -p "Hello"
 ```
 
+For standard runtimes (LM Studio, Ollama, upstream llama.cpp), use conventional KV cache types (`q8_0`, `q4_0`). You lose the RotorQuant-specific benefits but keep GGUF weight quantization.
+
+## Model Specifications
+
+| Property | Value |
+|----------|-------|
+| Base Model | [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) |
+| Architecture | Dense transformer (Edge optimised) |
+| Parameters | ~2B |
+| Context Length | 128K |
+| BF16 Size | ~4 GB |
+| Modalities | Text + Image |
+| License | apache-2.0 |
+
 ## What is RotorQuant?
 
-[RotorQuant](https://github.com/scrya-com/rotorquant) is a high-performance KV-cache quantization method that achieves significantly better throughput than TurboQuant. Instead of quantizing the model weights, RotorQuant targets the memory bottleneck of the KV cache, which grows linearly with sequence length and batch size.
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a KV cache compression method based on Clifford algebra (Cl(3,0)) rotors — a faster, more parameter-efficient alternative to Google's TurboQuant. Uses lightweight block-diagonal rotations (independent 2D/4D rotations per pair/quartet) achieving O(d) complexity instead of O(d log d), fully parallelisable with no inter-element dependencies.
+
+**Benchmarks** (from the RotorQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):
+
+- Prefill: 3,822 tok/s (vs TurboQuant 722 tok/s)
+- Decode: 119 tok/s (vs TurboQuant 93 tok/s)
+- Perplexity: 6.91 (vs TurboQuant 7.07)
+- Parameters: 4 per rotor (vs TurboQuant 16,384)
 
-Key advantages over TurboQuant:
-- **5.3x faster prefill**
-- **28% faster decode**
-- **No weight modification** -- model weights stay at original precision
-- **Reduced inference memory** -- KV cache is compressed significantly
-- **Longer context windows** -- fit more tokens in the same GPU memory
+> Benchmarks are from the RotorQuant repository using Llama 3.1 8B. Performance on gemma-4-E2B will differ. Please open a discussion if you have independent results.
 
-## KV-Cache Quantization Comparison
+## Current Ecosystem Support
 
-| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
-|---|---|---|---|---|
-| **TurboQuant** | Baseline | Baseline | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
-| **RotorQuant** | 5.3x faster | 28% faster | High | [GitHub](https://github.com/scrya-com/rotorquant) |
+| Runtime | RotorQuant Support | Notes |
+|---------|----------------------|-------|
+| Python transformers + `rotorquant` | ✅ Full | Drop-in cache class |
+| llama.cpp upstream | ❌ Not merged | Use fork below |
+| llama-cpp-turboquant fork | ✅ `planar3`, `iso3` | [GitHub](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache) |
+| LM Studio | ❌ [Requested](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1719) | Use `q8_0` as alternative |
+| Ollama | ❌ Not supported | Use `OLLAMA_KV_CACHE_TYPE=q8_0` |
+| vLLM | ❌ Not supported | — |
+| koboldcpp | ❌ Not supported | — |
 
-## Memory Estimates (Gemma 4 E2B)
+## Pre-quantized weight variants
 
-| Precision | Approximate Size |
-|---|---|
-| FP16 (original) | ~4 GB |
-| 8-bit quantized | ~2 GB |
-| 4-bit quantized | ~1.2 GB |
-| 2-bit quantized | ~0.6 GB |
+If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:
 
-Note: These estimates are for weight quantization. This repository applies KV-cache quantization only, so model weight memory remains at the precision you load the model in. The KV-cache memory savings are realized during generation.
+- [MLX (Apple Silicon)](https://huggingface.co/majentik?search=gemma-4-E2B+MLX)
+- [GGUF (llama.cpp / Ollama / LM Studio)](https://huggingface.co/majentik?search=gemma-4-E2B+GGUF)
 
 ## See Also
 
-- [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B) -- Base model
-- [majentik/gemma-4-E2B-TurboQuant](https://huggingface.co/majentik/gemma-4-E2B-TurboQuant) -- TurboQuant KV-cache variant
-- [majentik/gemma-4-E2B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-E2B-RotorQuant-MLX-8bit) -- MLX 8-bit weight-quantized variant
-- [majentik/gemma-4-E2B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-E2B-RotorQuant-MLX-4bit) -- MLX 4-bit weight-quantized variant
-- [majentik/gemma-4-E2B-RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-E2B-RotorQuant-MLX-2bit) -- MLX 2-bit weight-quantized variant
 - [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
+- [TurboQuant paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)
+- [llama-cpp-turboquant fork](https://github.com/johndpope/llama-cpp-turboquant/tree/feature/planarquant-kv-cache)
+- [Base model: google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B)
+- [gemma-4-E2B announcement](https://blog.google/technology/developers/gemma-4/)