Add model card

README.md

@@ -0,0 +1,87 @@
+---
+base_model: google/gemma-4-31B-it
+library_name: transformers
+tags:
+  - rotorquant
+  - kv-cache-quantization
+  - gemma
+  - gemma4
+  - multimodal
+  - quantized
+license: apache-2.0
+pipeline_tag: image-text-to-text
+---
+
+# Gemma 4 31B-it - RotorQuant KV Cache
+
+**RotorQuant KV-cache quantization** applied to [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it), enabling dramatically reduced memory usage during inference without modifying model weights. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant.
+
+This repository provides the RotorQuant KV-cache configuration for Gemma 4 31B-it. The model weights remain at their original precision; only the key-value cache is quantized at runtime.
+
+## Model Specifications
+
+| Property | Value |
+|---|---|
+| **Base Model** | [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) |
+| **Parameters** | 31 billion |
+| **Architecture** | Dense transformer |
+| **Modality** | Multimodal: image + text input, text output |
+| **License** | Apache 2.0 |
+| **Quantization** | RotorQuant KV-cache only (weights unchanged) |
+
+## Quickstart
+
+```python
+from rotorquant import RotorQuantCache
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+model_id = "google/gemma-4-31B-it"
+
+processor = AutoProcessor.from_pretrained(model_id)
+model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto")
+
+# Apply RotorQuant KV-cache quantization
+cache = RotorQuantCache(model)
+
+inputs = processor("Describe this image.", images=image, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, past_key_values=cache)
+print(processor.decode(outputs[0], skip_special_tokens=True))
+```
+
+## What is RotorQuant?
+
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a high-performance KV-cache quantization method that achieves significantly better throughput than TurboQuant. Instead of quantizing the model weights, RotorQuant targets the memory bottleneck of the KV cache, which grows linearly with sequence length and batch size.
+
+Key advantages over TurboQuant:
+- **5.3x faster prefill**
+- **28% faster decode**
+- **No weight modification** -- model weights stay at original precision
+- **Reduced inference memory** -- KV cache is compressed significantly
+- **Longer context windows** -- fit more tokens in the same GPU memory
+
+## KV-Cache Quantization Comparison
+
+| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
+|---|---|---|---|---|
+| **TurboQuant** | Baseline | Baseline | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
+| **RotorQuant** | 5.3x faster | 28% faster | High | [GitHub](https://github.com/scrya-com/rotorquant) |
+
+## Memory Estimates (Gemma 4 31B-it)
+
+| Precision | Approximate Size |
+|---|---|
+| FP16 (original) | ~62 GB |
+| 8-bit quantized | ~31 GB |
+| 4-bit quantized | ~17 GB |
+| 2-bit quantized | ~9 GB |
+
+Note: These estimates are for weight quantization. This repository applies KV-cache quantization only, so model weight memory remains at the precision you load the model in. The KV-cache memory savings are realized during generation.
+
+## See Also
+
+- [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) -- Base model
+- [majentik/gemma-4-31B-it-TurboQuant](https://huggingface.co/majentik/gemma-4-31B-it-TurboQuant) -- TurboQuant KV-cache variant
+- [majentik/gemma-4-31B-it-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-31B-it-RotorQuant-MLX-8bit) -- MLX 8-bit weight-quantized variant
+- [majentik/gemma-4-31B-it-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-31B-it-RotorQuant-MLX-4bit) -- MLX 4-bit weight-quantized variant
+- [majentik/gemma-4-31B-it-RotorQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-31B-it-RotorQuant-MLX-2bit) -- MLX 2-bit weight-quantized variant
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)