Add comprehensive README with benchmarks and usage

README.md

@@ -0,0 +1,247 @@
+---
+license: mit
+base_model: UsefulSensors/moonshine-streaming-tiny
+language:
+- en
+tags:
+  - onnx
+  - int8
+  - fp16
+  - quantized
+  - optimized
+  - speech-recognition
+  - asr
+  - streaming
+  - moonshine
+library_name: onnxruntime
+pipeline_tag: automatic-speech-recognition
+---
+
+# Moonshine Streaming Tiny — Optimized
+
+Optimized variants of [UsefulSensors/moonshine-streaming-tiny](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny), a 34M parameter streaming ASR model designed for real-time, on-device English speech recognition.
+
+Based on: [Moonshine v2: Ergodic Streaming Encoder ASR](https://arxiv.org/abs/2602.12241)
+
+## Optimized Variants
+
+| Variant | Total Size | Size Reduction | Best For |
+|---------|-----------|---------------|----------|
+| **Original FP32** | 168.1 MB | — | Reference |
+| **ONNX INT8** | 79.8 MB | **52%** | CPU deployment, edge devices |
+| **FP16 SafeTensors** | 88.1 MB | **48%** | GPU inference |
+| **ONNX FP32** | 297 MB | — | ONNX Runtime without quantization |
+
+## Benchmark Results
+
+Tested with 5 seconds of audio, generating up to 64 tokens:
+
+| Variant | Avg Latency | RTF | Speedup vs FP32 CPU |
+|---------|------------|-----|---------------------|
+| **PyTorch FP16 (GPU)** | 47.7 ms | 0.0095 | **1.71x** |
+| PyTorch INT8 (CPU) | 78.6 ms | 0.0157 | 1.03x |
+| PyTorch FP32 (CPU) | 81.3 ms | 0.0163 | 1.00x (baseline) |
+| ONNX FP32 (CPU) | 115.5 ms | 0.0231 | 0.70x |
+| ONNX INT8 (CPU) | 153.2 ms | 0.0306 | 0.53x |
+
+> **Note**: ONNX benchmarks include session overhead and were run on a single test. For production deployment with session reuse on real audio, ONNX Runtime typically provides better throughput, especially for long-running services. The Moonshine team reports 50ms response latency on Apple M3 with their C++ ONNX Runtime backend.
+
+## File Structure
+
+```
+├── onnx_int8/                          # ONNX INT8 quantized (recommended for CPU)
+│   ├── encoder_model_int8.onnx         # 9.8 MB
+│   ├── decoder_model_int8.onnx         # 36 MB
+│   ├── decoder_with_past_model_int8.onnx # 32 MB
+│   ├── tokenizer.json
+│   ├── config.json
+│   └── quantize_config.json
+├── onnx/                               # ONNX FP32
+│   ├── encoder_model.onnx + .data
+│   ├── decoder_model.onnx + .data
+│   ├── decoder_with_past_model.onnx + .data
+│   └── ...
+└── fp16/                               # FP16 SafeTensors (for GPU)
+    ├── model.safetensors               # 88.1 MB
+    ├── config.json
+    └── tokenizer.json
+```
+
+## Usage
+
+### ONNX INT8 Inference (CPU — Recommended for Edge)
+
+```bash
+pip install onnxruntime numpy tokenizers
+```
+
+```python
+import numpy as np
+import onnxruntime as ort
+from tokenizers import Tokenizer
+
+MODEL_DIR = "onnx_int8"  # or download from this repo
+BOS, EOS = 1, 2
+
+# Load models
+opts = ort.SessionOptions()
+opts.intra_op_num_threads = 4
+providers = ["CPUExecutionProvider"]
+
+encoder = ort.InferenceSession(f"{MODEL_DIR}/encoder_model_int8.onnx", opts, providers=providers)
+decoder = ort.InferenceSession(f"{MODEL_DIR}/decoder_model_int8.onnx", opts, providers=providers)
+decoder_past = ort.InferenceSession(f"{MODEL_DIR}/decoder_with_past_model_int8.onnx", opts, providers=providers)
+tokenizer = Tokenizer.from_file(f"{MODEL_DIR}/tokenizer.json")
+
+# Prepare audio (16kHz float32, padded to multiple of 80 samples)
+audio = np.random.randn(16000 * 5).astype(np.float32)  # replace with real audio
+remainder = len(audio) % 80
+if remainder:
+    audio = np.pad(audio, (0, 80 - remainder))
+
+audio_input = audio[np.newaxis, :]
+attention_mask = np.ones_like(audio_input, dtype=np.int64)
+
+# Encode audio
+(enc_out,) = encoder.run(None, {
+    "input_values": audio_input,
+    "attention_mask": attention_mask,
+})
+
+# First decode step
+outs = decoder.run(None, {
+    "decoder_input_ids": np.array([[BOS]], dtype=np.int64),
+    "encoder_hidden_states": enc_out,
+})
+logits, past_kvs = outs[0], outs[1:]
+token = int(np.argmax(logits[0, -1, :]))
+
+# Build KV cache mapping
+dec_out_names = [o.name for o in decoder.get_outputs()][1:]
+past_in_names = {i.name for i in decoder_past.get_inputs()} - {"decoder_input_ids", "encoder_hidden_states"}
+
+kv_dict = {}
+for name, tensor in zip(dec_out_names, past_kvs):
+    mapped = name.replace("present_", "past_", 1)
+    if mapped in past_in_names:
+        kv_dict[mapped] = tensor
+
+# Autoregressive decode loop
+past_out_names = [o.name for o in decoder_past.get_outputs()][1:]
+tokens = [token]
+
+for _ in range(255):
+    if token == EOS:
+        break
+    inputs = {
+        "decoder_input_ids": np.array([[token]], dtype=np.int64),
+        "encoder_hidden_states": enc_out,
+    }
+    inputs.update(kv_dict)
+    outs = decoder_past.run(None, inputs)
+    token = int(np.argmax(outs[0][0, -1, :]))
+    tokens.append(token)
+    
+    kv_dict = {}
+    for name, tensor in zip(past_out_names, outs[1:]):
+        mapped = name.replace("present_", "past_", 1)
+        if mapped in past_in_names:
+            kv_dict[mapped] = tensor
+
+text = tokenizer.decode(tokens)
+print(text)
+```
+
+### FP16 PyTorch Inference (GPU)
+
+```python
+from transformers import MoonshineStreamingForConditionalGeneration, AutoProcessor
+import torch
+
+model = MoonshineStreamingForConditionalGeneration.from_pretrained(
+    "felixem/moonshine-streaming-tiny-optimized",
+    subfolder="fp16",
+    torch_dtype=torch.float16,
+).to("cuda")
+
+processor = AutoProcessor.from_pretrained(
+    "felixem/moonshine-streaming-tiny-optimized",
+    subfolder="fp16",
+)
+
+# Process audio
+inputs = processor(audio_array, return_tensors="pt", sampling_rate=16000)
+inputs = {k: v.to("cuda", torch.float16) for k, v in inputs.items()}
+
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+text = processor.decode(generated_ids[0], skip_special_tokens=True)
+```
+
+### PyTorch Dynamic INT8 (CPU — Quick Setup)
+
+```python
+import torch
+from transformers import MoonshineStreamingForConditionalGeneration, AutoProcessor
+
+model = MoonshineStreamingForConditionalGeneration.from_pretrained(
+    "UsefulSensors/moonshine-streaming-tiny"
+).eval()
+
+# Quantize Linear layers to INT8
+model = torch.quantization.quantize_dynamic(
+    model, {torch.nn.Linear}, dtype=torch.qint8
+)
+
+processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-streaming-tiny")
+inputs = processor(audio_array, return_tensors="pt", sampling_rate=16000)
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+text = processor.decode(generated_ids[0], skip_special_tokens=True)
+```
+
+## ONNX Export Details
+
+- **Encoder**: Exported with `torch.onnx.export(dynamo=True)` to handle vmap-based sliding-window attention masking
+- **Decoder**: Separate models for first step (no KV cache) and autoregressive steps (with KV cache)
+- **Quantization**: `onnxruntime.quantization.quantize_dynamic` with symmetric INT8, per-channel, reduce_range=True
+
+### KV Cache Structure
+
+Each decoder layer produces 4 KV tensors:
+- `present_{layer}_self_key` / `present_{layer}_self_value`: Self-attention cache [B, 8, S, 40]
+- `present_{layer}_cross_key` / `present_{layer}_cross_value`: Cross-attention cache [B, 8, T, 40]
+
+For `decoder_with_past_model`, feed these back as `past_{layer}_*` inputs.
+
+## Quantization Impact
+
+Based on the [Edge-ASR paper](https://arxiv.org/abs/2507.07877) (Table 14), INT8 quantization on Moonshine Tiny has negligible WER impact:
+
+| Config | Avg WER | vs FP32 |
+|--------|---------|---------|
+| FP32 baseline | 12.72% | — |
+| **W8-A8 (INT8)** | **12.81%** | **+0.09%** |
+| W4-A16 (SpQR) | 13.61% | +0.89% |
+
+INT8 is the sweet spot for Moonshine Tiny — virtually no accuracy loss with ~50% model size reduction.
+
+## Limitations
+
+- English only
+- Optimized for short utterances (streaming chunks of 1-5 seconds)
+- ONNX models use external data files (`.onnx.data`) for FP32 variant
+- The decoder uses autoregressive generation, so output latency scales with transcript length
+
+## Citation
+
+```bibtex
+@article{kudlur2025moonshine,
+  title={Moonshine v2: Ergodic Streaming Encoder ASR},
+  author={Kudlur, Manjunath and King, Evan and Wang, James and Warden, Pete},
+  journal={arXiv preprint arXiv:2602.12241},
+  year={2025}
+}
+```
+
+## License
+
+MIT (same as base model)