Add model card

README.md

@@ -0,0 +1,116 @@
+---
+library_name: transformers
+base_model: aisingapore/MERaLiON-3-10B
+tags:
+  - rotorquant
+  - kv-cache-quantization
+  - efficient-inference
+  - meralion
+  - speech-to-text
+  - multimodal
+  - audio
+  - gemma-2
+license: other
+---
+
+# MERaLiON-3-10B-RotorQuant — RotorQuant KV Cache Compression
+
+KV cache quantized variant of [aisingapore/MERaLiON-3-10B](https://huggingface.co/aisingapore/MERaLiON-3-10B) using [RotorQuant](https://github.com/scrya-com/rotorquant) block-diagonal rotations. MERaLiON-3-10B is a multimodal audio-language model that uses google/gemma-2-9b as its decoder backbone.
+
+This is not weight quantization — the model weights remain unchanged. RotorQuant compresses the KV cache at inference time using learned Clifford algebra rotations, enabling longer audio contexts and lower VRAM usage with no training or calibration required.
+
+## What is RotorQuant?
+
+RotorQuant applies block-diagonal rotations (Clifford algebra) for online KV cache quantization during inference — no training or calibration required. It achieves **5.3x faster prefill** and **28% faster decode** compared to TurboQuant while using **44x fewer parameters**.
+
+| Metric | RotorQuant | TurboQuant |
+|--------|-----------|-----------|
+| Perplexity | 6.91 | 7.07 |
+| Decode Speed | 119 tok/s | 93 tok/s |
+| Prefill Speed | 3,822 tok/s | 722 tok/s |
+| Parameters | 128 | 16,384 |
+| Complexity | O(d) | O(d log d) |
+
+## Quickstart
+
+```python
+import torch
+from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
+from rotorquant import IsoQuantCache
+from datasets import load_dataset
+
+model_id = "majentik/MERaLiON-3-10B-RotorQuant"
+
+processor = AutoProcessor.from_pretrained(model_id)
+model = AutoModelForSpeechSeq2Seq.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+# Load audio sample
+dataset = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True)
+sample = next(iter(dataset))
+audio = sample["audio"]
+
+# Process audio input
+inputs = processor(
+    audio=audio["array"],
+    sampling_rate=audio["sampling_rate"],
+    return_tensors="pt",
+).to(model.device)
+
+# Use RotorQuant KV cache (3-bit recommended)
+cache = IsoQuantCache(bits=3)
+output = model.generate(
+    **inputs,
+    past_key_values=cache,
+    use_cache=True,
+    max_new_tokens=256,
+)
+transcription = processor.batch_decode(output, skip_special_tokens=True)[0]
+print(transcription)
+```
+
+## Backends
+
+- **PlanarQuant** (2D Givens rotations) — fastest, recommended for production
+- **IsoQuant** (4D quaternion rotations) — balanced quality/speed
+- **RotorQuant** (3D Clifford algebra) — research
+
+```python
+from rotorquant import PlanarQuantCache, IsoQuantCache, RotorQuantCache
+
+# Production (fastest)
+cache = PlanarQuantCache(bits=3)
+
+# Balanced (recommended default)
+cache = IsoQuantCache(bits=3)
+
+# Research
+cache = RotorQuantCache(bits=3)
+```
+
+## Configuration
+
+| Bits | KV Cache Compression | Quality | Recommended For |
+|------|---------------------|---------|-----------------|
+| 3-bit | ~10x | Excellent | Production — best speed/quality tradeoff |
+| 4-bit | ~5x | Near-lossless | Quality-critical applications |
+
+## Memory Savings
+
+VRAM usage for the Gemma-2-9B decoder at different audio context lengths:
+
+| Context Length | FP16 KV Cache | 3-bit RotorQuant | 4-bit RotorQuant |
+|---------------|---------------|-------------------|-------------------|
+| 8K | 0.9 GB | 0.09 GB | 0.18 GB |
+| 32K | 3.6 GB | 0.36 GB | 0.72 GB |
+| 64K | 7.2 GB | 0.72 GB | 1.44 GB |
+| 128K | 14.4 GB | 1.44 GB | 2.88 GB |
+
+## See Also
+
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
+- [TurboQuant variant](https://huggingface.co/majentik/MERaLiON-3-10B-TurboQuant)
+- [Base model](https://huggingface.co/aisingapore/MERaLiON-3-10B)