Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,123 @@
+---
+tags:
+  - audio
+  - speech-to-text
+  - streaming
+  - voxtral
+  - mistral
+language:
+  - en
+library_name: custom
+pipeline_tag: automatic-speech-recognition
+license: apache-2.0
+---
+
+# Voxtral Realtime 4B
+
+Streaming speech-to-text model with ~4 billion parameters. Weights in BF16 safetensors format, extracted from [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602).
+
+## Architecture
+
+**Pipeline:**
+```
+WAV → 16kHz → Mel Spectrogram → Conv Stem → Encoder → Downsample 4x → Adapter → Decoder → Tokens
+```
+
+- **Audio Encoder**: ~0.6B params — causal transformer, 32 layers
+- **Audio-Language Adapter**: 2-layer MLP with 4x downsample
+- **LLM Decoder**: ~3.4B params — Ministral-3 based, 26 layers with GQA
+
+### Audio Preprocessing
+
+| Parameter | Value |
+|-----------|-------|
+| Sample rate | 16,000 Hz |
+| Frame rate | 12.5 Hz |
+| Mel bins | 128 |
+| Hop length | 160 samples (10ms) |
+| Window size | 400 samples (25ms) |
+| 1 text token | 80ms of audio |
+
+### Encoder (Causal Transformer)
+
+| Parameter | Value |
+|-----------|-------|
+| dim | 1280 |
+| layers | 32 |
+| heads | 32 (MHA) |
+| head_dim | 64 |
+| hidden_dim | 5120 |
+| FFN | SwiGLU |
+| Norm | RMSNorm (eps=1e-5) |
+| Position | RoPE (theta=1e6, interleaved) |
+| Attention | causal, sliding window=750 |
+
+Conv stem: `conv1d(128→1280, k=3, s=1)` → GELU → `conv1d(1280→1280, k=3, s=2)` → GELU
+
+### Adapter
+
+```
+[seq/4, 5120] → Linear(5120→3072) → GELU → Linear(3072→3072) → [seq/4, 3072]
+```
+
+### Decoder (LLM)
+
+| Parameter | Value |
+|-----------|-------|
+| dim | 3072 |
+| layers | 26 |
+| heads | 32 |
+| KV heads | 8 (GQA 4:1) |
+| head_dim | 128 |
+| hidden_dim | 9216 |
+| Norm | RMSNorm (eps=1e-5) |
+| Position | RoPE (theta=1e6) |
+| Attention | causal, sliding window=8192 |
+| Vocab size | 131,072 |
+| Tied embeddings | yes |
+
+The decoder uses adaptive RMS normalization conditioned on transcription delay (6 delay tokens = 480ms).
+
+## Weight Format
+
+- **`consolidated.safetensors`** (8.3 GB) — 711 tensors, all BF16
+- **`params.json`** — model config
+- **`tekken.json`** (14.9 MB) — Tekken tokenizer
+
+## Tokenizer (Tekken)
+
+| Token | ID |
+|-------|----|
+| BOS | 1 |
+| EOS | 2 |
+| STREAMING_PAD | 32 |
+
+Token IDs 0–999 are special tokens. IDs 1000+ index into the vocabulary (base64-encoded byte sequences in `tekken.json`).
+
+### Audio Streaming Config
+
+| Parameter | Value |
+|-----------|-------|
+| sampling_rate | 16,000 |
+| frame_rate | 12.5 (80ms per token) |
+| transcription_delay_ms | 480 (6 delay tokens) |
+| left_pad_tokens | 32 |
+| right_pad_tokens (offline) | 17 |
+
+## Decode Schedule (Offline)
+
+1. **Prompt**: `[BOS] + [STREAMING_PAD] × 38` (1 + 32 left-pad + 6 delay)
+2. **Prefill**: Feed `audio_embed[i] + tok_embed(prompt[i])` for positions 0..L-2
+3. **First token**: Greedy argmax from position L-1
+4. **Autoregressive decode**: For each remaining audio position, feed `audio_embed[pos] + tok_embed(prev_token)`, greedy argmax
+5. **Stop**: On EOS or end of audio span
+
+## C Implementation
+
+A pure C implementation of this model is available at [voxtral.c](https://github.com/tantk/mistralhack/tree/master/voxtral.c) — runs on Apple Silicon (Metal) and CPU (BLAS), with streaming microphone input.
+
+## Credits
+
+Original model by [Mistral AI](https://mistral.ai/): [`mistralai/Voxtral-Mini-4B-Realtime-2602`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602)
+
+Built for the [Mistral Hackathon 2026](https://huggingface.co/mistral-hackaton-2026).