chore: merge master → dev/video-fer (SSE transcribe-stream)

Changes from master:
- Switch to evoxtral-rl model
- RL training pipeline, technical report (docs only, PDF excluded)
- Updated README and model card

From dev/video-fer:
- FER LFS pointer detection and auto-download
- Streaming emotion/valence/arousal from bracket tags
- Inline bracket badges with seek-on-click
- Async job polling in proxy (fallback for local inference)

README.md

@@ -10,209 +10,127 @@ pinned: false
 
 # Ethos Studio — Emotional Speech Recognition
 
-Speech-to-text with VAD sentence segmentation, per-segment emotion tagging, and facial emotion recognition (FER), powered by a fine-tuned [Voxtral Mini 3B](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) + [evoxtral-lora](https://huggingface.co/YongkangZOU/evoxtral-lora) running locally, and a MobileViT-XXS ONNX model for FER.
+Built for the **Mistral AI Online Hackathon 2026** (W&B Fine-Tuning Track).
 
-## Studio features
+Ethos Studio is a full-stack emotional speech recognition platform combining real-time transcription, facial emotion recognition, and expressive audio tagging. It turns raw speech into richly annotated transcripts with emotions, non-verbal sounds, and delivery cues.
 
-### Transcript editor
-- **Character-level text highlighting** — transcript text sweeps dark→gray in sync with playback position, character by character
-- **Click-to-seek** — click any character in the transcript to jump the timeline to that exact moment; uses `caretRangeFromPoint` for precision
-- **Inline `[bracket]` badges** — paralinguistic tags produced by Voxtral (e.g. `[laughs]`, `[sighs]`) render as pill badges at their exact inline position, not appended at the end; clicking a badge seeks to the moment just before it
-- **Bidirectional timeline ↔ transcript sync** — scrolling/clicking the timeline highlights the active segment in the transcript and auto-scrolls it into view; clicking a segment row seeks the timeline
-- **Per-segment state** (`past` / `active` / `future`) with opacity transitions
+## Key Components
 
-### Live emotion panel (right sidebar)
-- **Streaming speech emotion** — the Speech emotion badge updates sub-segment as playback passes each `[bracket]` tag; timing is estimated from the tag's character position proportional to segment duration
-- **Streaming valence & arousal bars** — both bars transition to the bracket tag's valence/arousal values at the same moment, creating a continuous emotional arc within each segment
-- **Per-second face emotion** (video only) — the Face badge updates every second from the `face_emotion_timeline` returned by the FER pipeline, more granular than the per-segment majority vote
-- **Live indicator** — animated green dot appears during playback
+### Evoxtral — Expressive Tagged Transcription
 
-### Timeline
-- **Click-to-seek** on the track area
-- **Active segment highlight** with ring indicator
-- **Played-region overlay** — subtle tint left of the playhead
-- **Dot + line playhead** design
+LoRA finetune of [Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) that produces transcriptions with inline [ElevenLabs v3](https://elevenlabs.io/docs/api-reference/text-to-speech) audio tags. Two-stage pipeline: **SFT** (3 epochs) → **RL via RAFT** (rejection sampling, 1 epoch).
 
-### Video support
-- Video files (`.mp4`, `.mkv`, `.avi`, `.mov`, `.m4v`, `.webm`) display inline in the right panel preview area
-- FER runs on video frames and produces both per-segment majority-vote emotion and a per-second `face_emotion_timeline`
+**Standard ASR:** `So I was thinking maybe we could try that new restaurant downtown.`
 
-## Repository layout
+**Evoxtral:** `[nervous] So... [stammers] I was thinking maybe we could... [clears throat] try that new restaurant downtown? [laughs nervously]`
 
-```
-ser/
-├── api/              # Python FastAPI — local Voxtral inference + FER pipeline
-├── proxy/            # Node.js/Express — API gateway for the frontend
-├── web/              # Next.js — Studio editor UI
-├── training/         # Fine-tuning code (Voxtral LoRA), data prep, eval
-├── docs/             # Specs, model card, hackathon guidelines
-├── models/           # ONNX weights (emotion_model_web.onnx — tracked via Git LFS)
-├── Dockerfile        # Single-container HF Spaces build
-├── nginx.conf        # Reverse proxy config (port 7860 → :3000/:3030)
-└── supervisord.conf  # Process manager for all four services
-```
+**Two model variants:**
+- **[Evoxtral SFT](https://huggingface.co/YongkangZOU/evoxtral-lora)** — Best transcription accuracy (lowest WER)
+- **[Evoxtral RL](https://huggingface.co/YongkangZOU/evoxtral-rl)** — Best expressive tag accuracy (highest Tag F1)
 
-## Architecture
+| Metric | Base Voxtral | Evoxtral SFT | Evoxtral RL | Best |
+|--------|-------------|-------------|------------|------|
+| **WER** ↓ | 6.64% | **4.47%** | 5.12% | SFT |
+| **CER** ↓ | 2.72% | **1.23%** | 1.48% | SFT |
+| **Tag F1** ↑ | 22.0% | 67.2% | **69.4%** | RL |
+| **Tag Recall** ↑ | 22.0% | 69.4% | **72.7%** | RL |
+| **Emphasis F1** ↑ | 42.0% | 84.0% | **86.0%** | RL |
 
-```
-Browser (:3030)
-    ↕  Next.js UI (upload, Studio editor, timeline, live emotion panel)
-Node proxy (:3000)
-    ↕  Express — streams multipart upload, manages session state
-Python API (:8000)
-    ├─ POST /transcribe-diarize  — VAD + Voxtral STT + emotion tags + FER timeline
-    └─ POST /fer                 — per-frame FER via MobileViT-XXS ONNX
-```
+- [SFT Model](https://huggingface.co/YongkangZOU/evoxtral-lora) | [RL Model](https://huggingface.co/YongkangZOU/evoxtral-rl)
+- [Live Demo (HF Space)](https://huggingface.co/spaces/YongkangZOU/evoxtral)
+- [API (Swagger UI)](https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/docs)
+- [W&B Dashboard](https://wandb.ai/yongkang-zou-ai/evoxtral)
+- [Technical Report (PDF)](Evoxtral%20Technical%20Report.pdf) | [LaTeX source](docs/technical_report.tex)
 
-nginx on port 7860 (HF Spaces public port) routes:
-- `/api/*` → Node proxy `:3000`
-- `/_next/*`, `/` → Next.js `:3030`
-
-| Directory | Port | Role |
-|-----------|------|------|
-| `api/`    | 8000 | Voxtral local inference; VAD segmentation; per-segment emotion; FER timeline |
-| `proxy/`  | 3000 | API entrypoint; proxies to `api/` |
-| `web/`    | 3030 | Next.js Studio UI |
-
-## API response format
-
-`POST /api/transcribe-diarize` returns:
-
-```json
-{
-  "filename": "interview.mp4",
-  "duration": 42.5,
-  "text": "Full transcript...",
-  "segments": [
-    {
-      "id": 1,
-      "speaker": "SPEAKER_00",
-      "start": 0.0,
-      "end": 5.2,
-      "text": "Welcome to the show. [laughs]",
-      "emotion": "Happy",
-      "valence": 0.7,
-      "arousal": 0.6,
-      "face_emotion": "Happy"
-    }
-  ],
-  "has_video": true,
-  "face_emotion_timeline": {
-    "0": "Neutral",
-    "1": "Happy",
-    "2": "Happy"
-  }
-}
-```
+### FER — Facial Emotion Recognition
+
+MobileViT-XXS model trained on 8 emotion classes, exported to ONNX for real-time browser inference.
 
-`face_emotion_timeline` maps each second (as a string key) to the majority FER label for that second. Only present for video inputs.
+**Classes:** Anger, Contempt, Disgust, Fear, Happy, Neutral, Sad, Surprise
 
-## Bracket tag emotions
+### Voxtral Server — Speech-to-Text + Emotion
 
-Voxtral produces paralinguistic `[bracket]` tags in transcriptions. The frontend and API both recognise these tags and map them to `(emotion, valence, arousal)` triples:
+Speech-to-text service with VAD sentence segmentation and per-segment emotion analysis, powered by [Voxtral Mini 4B](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602).
 
-| Tag | Emotion | Valence | Arousal |
-|-----|---------|---------|---------|
-| `[laughs]` / `[laughing]` | Happy | +0.70 | +0.60 |
-| `[sighs]` / `[sighing]` | Sad | −0.30 | −0.30 |
-| `[whispers]` / `[whispering]` | Calm | +0.10 | −0.50 |
-| `[shouts]` / `[shouting]` | Angry | −0.50 | +0.80 |
-| `[exclaims]` | Excited | +0.50 | +0.70 |
-| `[gasps]` | Surprised | +0.20 | +0.70 |
-| `[hesitates]` / `[stutters]` / `[stammers]` | Anxious | −0.20 | +0.35 |
-| `[cries]` / `[crying]` | Sad | −0.70 | +0.40 |
-| `[claps]` / `[applause]` | Happy | +0.60 | +0.50 |
-| `[clears throat]` / `[pause]` | Neutral | 0.00 | ±0.10 |
+## Architecture
 
-## How to run locally
+```
+Browser (port 3030)  →  Server layer (Node, :3000)  →  Model layer (Python, :8000)
+      ↑ Studio UI            POST /api/speech-to-text          POST /transcribe
+      ↑ Upload dialog        POST /api/transcribe-diarize      POST /transcribe-diarize
+                             GET  /health                       GET  /health
+```
 
-**Requirements**: Python 3.11+, Node.js 22+, ffmpeg, a GPU with ~8 GB VRAM (or CPU fallback).
+| Layer | Path | Role |
+|-------|------|------|
+| **Model** | `model/voxtral-server` | Voxtral inference, VAD segmentation, emotion analysis |
+| **Server** | `demo/server` | API entrypoint; proxies to Model |
+| **Frontend** | `demo` | Next.js UI (upload, Studio editor, waveform, timeline) |
+| **Evoxtral** | `training/scripts/` | Training, eval, RL, serving for expressive transcription |
+| **FER** | `models/` | Facial emotion recognition ONNX model |
 
-### 1. Python API (port 8000)
+See [demo/README.md](demo/README.md) for full API and usage; [model/voxtral-server/README.md](model/voxtral-server/README.md) for the Model API.
+
+## Project Structure
 
-```bash
-cd api
-python -m venv .venv
-source .venv/bin/activate
-pip install -r requirements.txt
-uvicorn main:app --host 0.0.0.0 --port 8000 --reload
+```
+├── api/                    # Python FastAPI — local Voxtral inference + FER
+├── proxy/                  # Node.js/Express — API gateway for frontend
+├── web/                    # Next.js — Studio editor UI
+├── training/               # Fine-tuning code (SFT + RL), data prep, eval
+│   └── scripts/            # Modal scripts: train, RL (RAFT), eval, serve
+├── space/                  # HuggingFace Space (Gradio demo)
+├── models/                 # FER ONNX model (MobileViT-XXS)
+├── docs/                   # Technical report, design docs, research refs
+├── data/                   # Training data scripts (audio files gitignored)
+└── Dockerfile              # Single-container HF Spaces build
 ```
 
-On first start the Voxtral model (~6 GB) is downloaded from HuggingFace. Set `MODEL_ID` / `ADAPTER_ID` env vars to override.
+## How to Run
 
-Inference is optimised with `merge_and_unload()` (removes PEFT per-forward overhead), `torch.set_num_threads(cpu_count)`, and `torch.inference_mode()`.
+**Requirements**: Python 3.10+, Node.js 20+, ffmpeg; GPU recommended.
 
-### 2. Node proxy (port 3000)
+### Model layer (port 8000)
 
 ```bash
-cd proxy
-npm install
-npm run dev
+cd model/voxtral-server
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+uvicorn main:app --host 0.0.0.0 --port 8000 --reload
 ```
 
-### 3. Next.js frontend (port 3030)
+### Server layer (port 3000)
 
 ```bash
-cd web
-npm install
-npm run dev
+cd demo/server && npm install && npm run dev
 ```
 
-Open [http://localhost:3030](http://localhost:3030).
-
-### Quick health check
+### Frontend (port 3030)
 
 ```bash
-curl -s http://localhost:3000/health
-curl -X POST http://localhost:3000/api/transcribe-diarize -F "audio=@/path/to/audio.m4a"
-curl -X POST http://localhost:3000/api/transcribe-diarize -F "audio=@/path/to/video.mov"
+cd demo && npm install && npm run dev
 ```
 
-Upload a video to also get per-segment facial emotion and the `face_emotion_timeline`.
-
-## Models
-
-| Model | Purpose | Source |
-|-------|---------|--------|
-| `mistralai/Voxtral-Mini-3B-2507` | Speech-to-text base | HF Hub (downloaded at runtime) |
-| `YongkangZOU/evoxtral-lora` | LoRA adapter — emotion-aware transcription | HF Hub (downloaded at runtime) |
-| `models/emotion_model_web.onnx` | MobileViT-XXS 8-class FER | Stored in repo via Git LFS |
-
-FER emotion classes: `Anger | Contempt | Disgust | Fear | Happy | Neutral | Sad | Surprise`
-
-## Training
+Open [http://localhost:3030](http://localhost:3030).
 
-Fine-tuning scripts and data utilities live in `training/`:
+### Evoxtral API (Modal)
 
 ```bash
-cd training
-pip install -r requirements.txt
-# fine-tune on Modal
-python scripts/train_modal.py
-# push adapter to HF Hub
-python scripts/push_hub.py
+modal deploy training/scripts/serve_modal.py
 ```
 
-See `training/finetune.py` for the full training loop and `training/config.py` for hyperparameters.
+## Tech Stack
 
-## Deployment (HuggingFace Spaces)
+- **Models**: Voxtral-Mini-3B + LoRA, Voxtral-Mini-4B, MobileViT-XXS
+- **Training**: PyTorch, PEFT, Weights & Biases
+- **Inference**: Modal (serverless GPU), HuggingFace ZeroGPU, ONNX Runtime
+- **Backend**: FastAPI, Node.js
+- **Frontend**: Next.js, Gradio
 
-The app runs as a single Docker container. `supervisord` starts all four processes; nginx handles port routing.
-
-```bash
-# Build and test locally
-docker build -t ethos-studio .
-docker run -p 7860:7860 ethos-studio
-```
+## Links
 
-HF Spaces auto-builds on every push to the `main` branch of the linked Space.
-
-> **Note**: `models/emotion_model_web.onnx` is stored via Git LFS / Xet on HF Spaces. When pushing a new commit, use the `commit-tree` graft technique to reuse the existing LFS tree rather than re-uploading the binary.
->
-> ```bash
-> MODELS_SHA=$(git ls-tree space/main | grep $'\tmodels$' | awk '{print $3}')
-> TREE_SHA=$((git ls-tree HEAD | grep -v $'\tmodels$'; echo "040000 tree $MODELS_SHA\tmodels") | git mktree)
-> PARENT=$(git rev-parse space/main)
-> COMMIT_SHA=$(git commit-tree "$TREE_SHA" -p "$PARENT" -m "your message")
-> git push space "${COMMIT_SHA}:refs/heads/main"
-> ```
+- [W&B Project](https://wandb.ai/yongkang-zou-ai/evoxtral) | [W&B Eval Report](https://wandb.ai/yongkang-zou-ai/evoxtral/reports/Evoxtral-—-Evaluation-Results:-Base-vs-SFT-vs-RL--VmlldzoxNjA3MzI3Nw==)
+- [Evoxtral SFT Model](https://huggingface.co/YongkangZOU/evoxtral-lora) | [Evoxtral RL Model](https://huggingface.co/YongkangZOU/evoxtral-rl)
+- [Evoxtral Demo](https://huggingface.co/spaces/YongkangZOU/evoxtral)
+- [Evoxtral API (Swagger)](https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/docs)
+- [Technical Report (PDF)](Evoxtral%20Technical%20Report.pdf) | [LaTeX](docs/technical_report.tex)

api/main.py

@@ -1,6 +1,6 @@
 """
 Evoxtral speech-to-text server (Model layer).
-Runs Voxtral-Mini-3B + evoxtral-lora locally for transcription with expressive
+Runs Voxtral-Mini-3B + evoxtral-rl locally for transcription with expressive
 tags. For video files, also runs FER (MobileViT-XXS ONNX) per segment.
 """
 import asyncio
@@ -19,7 +19,7 @@ from fastapi import FastAPI, File, UploadFile, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 
 MODEL_ID   = os.environ.get("MODEL_ID",   "mistralai/Voxtral-Mini-3B-2507")
-ADAPTER_ID = os.environ.get("ADAPTER_ID", "YongkangZOU/evoxtral-lora")
+ADAPTER_ID = os.environ.get("ADAPTER_ID", "YongkangZOU/evoxtral-rl")
 MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_MB", "100")) * 1024 * 1024
 TARGET_SR = 16000
 

docs/model_card/README.md

@@ -9,6 +9,8 @@ tags:
   - audio
   - mistral
   - hackathon
+  - rl
+  - raft
 datasets:
   - custom
 language:
@@ -23,6 +25,10 @@ A LoRA adapter for [Voxtral-Mini-3B-2507](https://huggingface.co/mistralai/Voxtr
 
 Built for the **Mistral AI Online Hackathon 2026** (W&B Fine-Tuning Track).
 
+**Two model variants available:**
+- **[Evoxtral SFT](https://huggingface.co/YongkangZOU/evoxtral-lora)** — Best overall transcription accuracy (lowest WER)
+- **[Evoxtral RL](https://huggingface.co/YongkangZOU/evoxtral-rl)** — Best expressive tag accuracy (highest Tag F1)
+
 ## What It Does
 
 Standard ASR:
@@ -31,17 +37,63 @@ Standard ASR:
 Evoxtral:
 > [nervous] So... [stammers] I was thinking maybe we could... [clears throat] try that new restaurant downtown? [laughs nervously] I mean, if you're free this weekend?
 
-## Evaluation Results
+## Training Pipeline
+
+```
+Base Voxtral-Mini-3B → SFT (LoRA, 3 epochs) → RL (RAFT, 1 epoch)
+```
 
-| Metric | Base Voxtral | Evoxtral (finetuned) | Improvement |
-|--------|-------------|---------------------|-------------|
-| **WER** (Word Error Rate) | 6.64% | **4.47%** | 32.7% better |
-| **Tag F1** (Expressive Tag Accuracy) | 22.0% | **67.2%** | 3x better |
+1. **SFT**: LoRA finetuning on 808 synthetic audio samples with expressive tags (lr=2e-4, 3 epochs)
+2. **RL (RAFT)**: Rejection sampling — generate 4 completions per sample, score with rule-based reward (WER accuracy + Tag F1 - hallucination penalty), keep best, then SFT on curated data (lr=5e-5, 1 epoch)
 
-Evaluated on 50 held-out test samples. The finetuned model dramatically improves expressive tag generation while also improving raw transcription accuracy.
+This follows the approach from [GRPO for Speech Recognition](https://arxiv.org/abs/2509.01939) and Voxtral's own SFT→DPO training recipe.
+
+## Evaluation Results
+
+Evaluated on 50 held-out test samples. Full benchmark (Evoxtral-Bench) with 7 metrics:
+
+### Core Metrics — Base vs SFT vs RL
+
+| Metric | Base Voxtral | Evoxtral SFT | Evoxtral RL | Best |
+|--------|-------------|-------------|------------|------|
+| **WER** | 6.64% | **4.47%** | 5.12% | SFT |
+| **CER** | 2.72% | **1.23%** | 1.48% | SFT |
+| **Tag F1** | 22.0% | 67.2% | **69.4%** | RL |
+| **Tag Precision** | 22.0% | 67.4% | **68.5%** | RL |
+| **Tag Recall** | 22.0% | 69.4% | **72.7%** | RL |
+| **Emphasis F1** | 42.0% | 84.0% | **86.0%** | RL |
+| **Tag Hallucination** | 0.0% | **19.3%** | 20.2% | SFT |
+
+**SFT** excels at raw transcription accuracy (best WER/CER). **RL** further improves expressive tag generation (+2.2% Tag F1, +3.3% Tag Recall, +2% Emphasis F1) at a small cost to WER.
+
+### Per-Tag F1 Breakdown (SFT → RL)
+
+| Tag | SFT F1 | RL F1 | Change | Support |
+|-----|--------|-------|--------|---------|
+| `[sighs]` | 1.000 | **1.000** | — | 9 |
+| `[clears throat]` | 0.889 | **1.000** | +12.5% | 8 |
+| `[gasps]` | 0.957 | **0.957** | — | 12 |
+| `[pause]` | 0.885 | **0.902** | +1.9% | 25 |
+| `[nervous]` | 0.800 | **0.846** | +5.8% | 13 |
+| `[stammers]` | 0.889 | 0.842 | -5.3% | 8 |
+| `[laughs]` | 0.800 | **0.815** | +1.9% | 12 |
+| `[sad]` | 0.667 | **0.750** | +12.4% | 4 |
+| `[whispers]` | 0.636 | **0.667** | +4.9% | 13 |
+| `[crying]` | 0.750 | 0.571 | -23.9% | 5 |
+| `[excited]` | 0.615 | 0.571 | -7.2% | 5 |
+| `[shouts]` | 0.400 | **0.500** | +25.0% | 3 |
+| `[calm]` | 0.200 | **0.400** | +100% | 6 |
+| `[frustrated]` | 0.444 | 0.444 | — | 3 |
+| `[angry]` | 0.667 | 0.667 | — | 2 |
+| `[confused]` | 0.000 | 0.000 | — | 1 |
+| `[scared]` | 0.000 | 0.000 | — | 1 |
+
+RL improved 9 tags, kept 4 stable, and regressed 3. Biggest gains on [clears throat] (+12.5%), [calm] (+100%), [sad] (+12.4%), and [shouts] (+25%).
 
 ## Training Details
 
+### SFT Stage
+
 | Parameter | Value |
 |-----------|-------|
 | Base model | `mistralai/Voxtral-Mini-3B-2507` |
@@ -60,6 +112,20 @@ Evaluated on 50 held-out test samples. The finetuned model dramatically improves
 | Training time | ~25 minutes |
 | Trainable params | 124.8M / 4.8B (2.6%) |
 
+### RL Stage (RAFT)
+
+| Parameter | Value |
+|-----------|-------|
+| Method | Rejection sampling + SFT (RAFT) |
+| Samples per input | 4 (temperature=0.7, top_p=0.9) |
+| Reward function | 0.4×(1-WER) + 0.4×Tag_F1 + 0.2×(1-hallucination) |
+| Curated samples | 727 (bottom 10% filtered, reward > 0.954) |
+| Avg reward | 0.980 |
+| Learning rate | 5e-5 |
+| Epochs | 1 |
+| Final loss | 0.021 |
+| Training time | ~7 minutes |
+
 ## Dataset
 
 Custom synthetic dataset of 1,010 audio samples generated with ElevenLabs TTS v3:
@@ -76,7 +142,8 @@ from transformers import VoxtralForConditionalGeneration, AutoProcessor
 from peft import PeftModel
 
 repo_id = "mistralai/Voxtral-Mini-3B-2507"
-adapter_id = "YongkangZOU/evoxtral-lora"
+# Use "YongkangZOU/evoxtral-lora" for SFT or "YongkangZOU/evoxtral-rl" for RL
+adapter_id = "YongkangZOU/evoxtral-rl"
 
 processor = AutoProcessor.from_pretrained(repo_id)
 base_model = VoxtralForConditionalGeneration.from_pretrained(
@@ -102,12 +169,26 @@ print(transcription)
 # [nervous] So... I was thinking maybe we could [clears throat] try that new restaurant downtown?
 ```
 
+## API
+
+A serverless API with Swagger UI is available on Modal:
+
+```bash
+curl -X POST https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/transcribe \
+    -F "file=@audio.wav"
+```
+
+- [Swagger UI](https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/docs)
+- [Live Demo (HF Space)](https://huggingface.co/spaces/YongkangZOU/evoxtral)
+
 ## W&B Tracking
 
 All training and evaluation runs are tracked on Weights & Biases:
-- [Training run](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/t8ak7a20)
-- [Base model eval](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/f9l2zwvs)
-- [Finetuned model eval](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/b32c74im)
+- [SFT Training](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/t8ak7a20)
+- [RL Training (RAFT)](https://wandb.ai/yongkang-zou-ai/evoxtral)
+- [Base model eval](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/bvqa4ioo)
+- [SFT model eval](https://wandb.ai/yongkang-zou-ai/evoxtral/runs/ayx4ldyd)
+- [RL model eval](https://wandb.ai/yongkang-zou-ai/evoxtral)
 - [Project dashboard](https://wandb.ai/yongkang-zou-ai/evoxtral)
 
 ## Supported Tags
@@ -119,7 +200,9 @@ The model can produce any tag from the ElevenLabs v3 expressive tag set, includi
 ## Limitations
 
 - Trained on synthetic (TTS-generated) audio, not natural speech recordings
-- Tag F1 of 67.2% means ~1/3 of tags may be missed or misplaced
+- ~20% tag hallucination rate — model occasionally predicts tags not in the reference
+- Rare/subtle tags ([calm], [confused], [scared]) have low accuracy due to limited training examples
+- RL variant trades ~0.65% WER for better tag accuracy
 - English only
 - Best results on conversational and emotionally expressive speech
 

docs/research/references.md

@@ -0,0 +1,84 @@
+# Evoxtral — Research References & Sources
+
+References for the technical report covering expressive tagged transcription, ASR evaluation, and post-training methods.
+
+## Core Model
+
+- **Voxtral: An Audio-Language Model** — Mistral AI, 2025. Voxtral Mini and Small are open-weights audio chat models trained with SFT + DPO + Online DPO. Online DPO delivers crisper grounding, fewer hallucinations, and more helpful responses.
+  - Paper: https://arxiv.org/abs/2507.13264
+  - Model: https://huggingface.co/mistralai/Voxtral-Mini-3B-2507
+
+## RL / Post-Training for ASR
+
+- **Group Relative Policy Optimization for Speech Recognition** — Proposes GRPO with rule-based rewards for LLM-based ASR. Achieved 18% relative WER reduction on AMI-IHM and 27.9% on AMI-SDM compared to SFT-adapted models.
+  - Paper: https://arxiv.org/abs/2509.01939
+
+- **Advancing Speech Understanding in Speech-Aware Language Models with GRPO** — Applies GRPO to large audio language models, investigating different rule-based reward functions and RL data construction strategies.
+  - Paper: https://arxiv.org/abs/2509.16990
+
+- **Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering** — Demonstrates RL (GRPO) achieves state-of-the-art on audio QA tasks, outperforming SFT baselines.
+  - Paper: https://arxiv.org/abs/2503.11197
+
+- **Explore the Reinforcement Learning for LLM-based ASR and TTS System** — Surveys RL applications to speech models, noting that while RL has enhanced text-based LLMs, application to ASR/TTS remains underexplored.
+  - Paper: https://arxiv.org/abs/2509.18569
+
+## ASR Evaluation Metrics
+
+- **Speech Recognition Accuracy: Production Metrics & Optimization 2025** — Deepgram. Covers WER, CER, Keyword Recall Rate (KRR), Real-Time Factor (RTF), and end-to-end latency. Production systems need blended metrics depending on use case.
+  - Source: https://deepgram.com/learn/speech-recognition-accuracy-production-metrics
+
+- **Moving Beyond Word Error Rate to Evaluate ASR in Clinical Samples** — Argues WER alone is insufficient; error type, meaning, and context matter. A single substitution can drastically change intent with the same WER penalty.
+  - Paper: https://www.sciencedirect.com/science/article/pii/S0165178125003385
+
+- **Measuring the Accuracy of Automatic Speech Recognition Solutions** — ACM survey on ASR evaluation methodology, limitations of WER/CER, and alternative metrics.
+  - Paper: https://dl.acm.org/doi/10.1145/3636513
+
+- **On the Robust Approximation of ASR Metrics** — ACL 2025. Novel label-free approach for approximating ASR performance using multimodal embeddings.
+  - Paper: https://arxiv.org/abs/2502.12408
+
+- **ProfASR-Bench: A Professional-talk ASR Dataset for High-Stakes Applications** — Evaluation suite supporting conventional metrics plus entity-aware scores and slice-wise reporting by accent and gender.
+  - Paper: https://arxiv.org/abs/2512.23686
+
+## Expressive Speech & TTS
+
+- **ElevenLabs v3 Text-to-Speech** — TTS model supporting inline expressive audio tags for emotions, non-verbal sounds, and delivery cues. Tag set used as target vocabulary for Evoxtral.
+  - Docs: https://elevenlabs.io/docs/api-reference/text-to-speech
+
+## Fine-Tuning Frameworks
+
+- **ms-swift** — ModelScope framework supporting SFT/DPO/GRPO for 600+ LLMs and 300+ MLLMs. AAAI 2025.
+  - GitHub: https://github.com/modelscope/ms-swift
+
+- **OpenRLHF** — Scalable agentic RL framework based on Ray, supporting PPO, DAPO, REINFORCE++, and more.
+  - GitHub: https://github.com/OpenRLHF/OpenRLHF
+
+- **PEFT (Parameter-Efficient Fine-Tuning)** — HuggingFace library for LoRA and other adapter methods.
+  - GitHub: https://github.com/huggingface/peft
+
+## Evaluation Methodology
+
+- **jiwer** — Python library for WER, CER, and other ASR metrics based on edit distance.
+  - GitHub: https://github.com/jitsi/jiwer
+
+## Our Benchmark: Evoxtral-Bench
+
+Metrics computed by `scripts/train_modal.py::evaluate()`:
+
+| Metric | Description | Direction |
+|--------|-------------|-----------|
+| **WER** | Word Error Rate on plain text (tags stripped) | lower = better |
+| **CER** | Character Error Rate on plain text | lower = better |
+| **Tag F1** | F1 score for tag extraction (multiset intersection) | higher = better |
+| **Tag Precision** | Fraction of predicted tags that match reference | higher = better |
+| **Tag Recall** | Fraction of reference tags captured by prediction | higher = better |
+| **Tag Hallucination Rate** | Fraction of predicted tags not in reference at all | lower = better |
+| **Emphasis F1** | F1 for CAPITALIZED emphasis words | higher = better |
+| **Per-Tag F1** | Breakdown by individual tag type (e.g. [laughs], [sighs]) | higher = better |
+
+### Why These Metrics
+
+- **WER + CER**: Standard ASR quality. CER is more granular and catches character-level errors WER misses.
+- **Tag Precision vs Recall**: F1 alone hides whether the model hallucinates tags (low precision) or misses them (low recall). Judges care about this distinction.
+- **Tag Hallucination Rate**: Critical for downstream TTS — hallucinated tags produce wrong prosody/effects.
+- **Per-Tag Breakdown**: Shows which expressive cues the model handles well vs struggles with, informing future data collection.
+- **Emphasis F1**: CAPS words (e.g., "BELIEVE", "NEVER") carry prosodic emphasis in ElevenLabs v3; measuring this separately tracks delivery accuracy.

docs/technical_report.md

@@ -0,0 +1,300 @@
+# Evoxtral: Expressive Tagged Transcription via Supervised Fine-Tuning and Rejection Sampling
+
+**Mistral AI Online Hackathon 2026 — W&B Fine-Tuning Track**
+
+---
+
+## Abstract
+
+Standard automatic speech recognition (ASR) systems discard paralinguistic and expressive information present in spoken audio, producing plain text that fails to capture sighs, laughs, hesitations, emotional tone, and other prosodic cues. We present **Evoxtral**, a LoRA adapter for `mistralai/Voxtral-Mini-3B-2507` [1] that produces transcriptions enriched with inline expressive audio tags drawn from the ElevenLabs v3 tag vocabulary [6]. We apply a two-stage post-training pipeline: supervised fine-tuning (SFT) followed by rejection sampling fine-tuning (RAFT) [4]. SFT reduces word error rate (WER) by 33% relative (6.64% → 4.47%) and increases Tag F1 from 22.0% to 67.2%. The subsequent RAFT stage further improves Tag F1 to 69.4% and Tag Recall to 72.7% at a marginal WER cost. We release two model variants, a serverless inference API, and a live interactive demo.
+
+---
+
+## 1. Introduction
+
+Modern ASR pipelines excel at converting speech to text with low word error rates, yet they systematically strip out the expressive dimension of human communication. When a speaker sighs before a sentence, laughs nervously mid-phrase, or whispers for emphasis, these paralinguistic signals carry meaning that plain transcription cannot represent. This information is especially critical for downstream text-to-speech (TTS) synthesis: next-generation TTS systems such as ElevenLabs v3 [6] consume inline expressive tags to control prosody, affect, and delivery at a fine-grained level.
+
+We ask: can a multimodal audio-language model be trained to produce ASR output that preserves expressive content through inline tags? We answer affirmatively with **Evoxtral**, built on Voxtral-Mini-3B-2507 [1].
+
+**Why Voxtral.** We chose Voxtral-Mini-3B as our base model for several reasons. First, as a *generative* audio-language model (rather than a CTC or encoder-only ASR), Voxtral decodes transcriptions autoregressively — meaning it can naturally produce arbitrary inline tokens such as `[laughs]` or `[nervous]` within the text stream, without requiring architectural changes. Traditional ASR models constrain their output vocabulary to words and punctuation; Voxtral's LLM decoder has no such limitation. Second, Voxtral's compact 3B-parameter architecture makes LoRA fine-tuning feasible on a single A10G GPU within hackathon time constraints, while still delivering competitive ASR quality. Third, Voxtral was itself trained with post-training alignment (SFT + DPO + Online DPO) [1], meaning the model is already instruction-following and amenable to further fine-tuning — a strong foundation for adding new capabilities. Finally, as a Mistral AI model released under Apache 2.0, Voxtral aligns with the hackathon's focus on the Mistral ecosystem and enables open redistribution of our adapters.
+
+Our approach fine-tunes Voxtral-Mini-3B-2507 using parameter-efficient LoRA adapters [3] on a synthetically generated dataset of expressive speech paired with tagged transcriptions.
+
+To illustrate the contrast between standard ASR and Evoxtral output, consider the following example:
+
+**Standard ASR output:**
+> "So I was thinking maybe we could try that new restaurant downtown. I mean if you're free this weekend."
+
+**Evoxtral output:**
+> "[nervous] So... [stammers] I was thinking maybe we could... [clears throat] try that new restaurant downtown? [laughs nervously] I mean, if you're free this weekend?"
+
+The Evoxtral output captures hesitation, nervous laughter, and a throat clear — paralinguistic content that is acoustically present but conventionally discarded. Our two-stage training pipeline (SFT followed by RAFT) achieves strong tag generation performance while maintaining competitive transcription accuracy. We make the following contributions:
+
+1. A synthetic dataset of 1,010 expressive speech samples paired with tagged transcriptions across 17 ElevenLabs v3 tag types.
+2. A two-stage fine-tuning recipe (SFT → RAFT) for expressive ASR using LoRA on Voxtral-Mini-3B.
+3. A custom evaluation benchmark, **Evoxtral-Bench**, with seven metrics covering both transcription accuracy and tag generation quality.
+4. Two released model variants optimized for different use cases: accuracy-critical (SFT) and expressiveness-critical (RL).
+
+---
+
+## 2. Related Work
+
+**Voxtral [1].** Voxtral-Mini-3B-2507 is a multimodal audio-language model released by Mistral AI. It is built on a Whisper-based audio encoder fused with a Mistral language model backbone, trained via SFT, DPO, and online DPO. Our work builds directly on this foundation, adding expressive tagging capability via LoRA adaptation.
+
+**Reinforcement Learning for LLM-based ASR [2].** Shi et al. apply group relative policy optimization (GRPO) to LLM-based ASR, achieving an 18% relative WER reduction without paired preference data. Closely related work [8, 9, 12] further demonstrates that RL-based training consistently outperforms SFT alone for speech understanding tasks. Our RAFT stage is philosophically aligned with this line of work but uses rule-based rejection sampling rather than policy gradient methods, making it simpler to implement and more stable to train.
+
+**LoRA [3].** Low-rank adaptation (LoRA) [3] inserts trainable low-rank matrices into the attention projections and feed-forward layers of a frozen base model. This reduces the number of trainable parameters by orders of magnitude while matching full fine-tuning performance. We use LoRA with rank 64 and alpha 128, implemented via HuggingFace PEFT [10].
+
+**Rejection Sampling Fine-Tuning (RAFT/RFT) [4].** Yuan et al. [4] propose generating multiple model completions for each training input, scoring them with a reward function, and performing SFT on the highest-scoring completions. This iterative rejection sampling approach is computationally simpler than policy gradient methods while still providing a reinforcement learning signal. We adopt this approach for our RL stage.
+
+**NEFTune [5].** Jain et al. [5] demonstrate that adding uniform random noise to embedding vectors during training improves instruction-following performance. We apply NEFTune with noise alpha = 5.0 during SFT to regularize training on our small dataset.
+
+**ElevenLabs v3 Audio Tags [6].** ElevenLabs v3 TTS introduces a structured vocabulary of inline expressive tags that control how synthesized speech is delivered. These tags — such as `[sighs]`, `[laughs]`, `[whispers]`, and `[nervous]` — are the target output vocabulary for Evoxtral.
+
+**ASR Evaluation [7, 11].** We compute WER and CER using the `jiwer` library [7], following standard definitions from Morris et al. [11].
+
+---
+
+## 3. Method
+
+### 3.1 Dataset
+
+We construct a synthetic dataset of 1,010 audio samples generated using the ElevenLabs TTS v3 API [6]. Each sample consists of a short spoken utterance (5–30 seconds) paired with a reference tagged transcription containing inline ElevenLabs v3 expressive tags. The dataset covers 17 tag types:
+
+`[sighs]`, `[laughs]`, `[whispers]`, `[nervous]`, `[frustrated]`, `[clears throat]`, `[pause]`, `[excited]`, `[stammers]`, `[gasps]`, `[sad]`, `[angry]`, `[calm]`, `[crying]`, `[shouts]`, `[confused]`, `[scared]`
+
+The dataset is split into 808 training, 101 validation, and 101 test samples. Tag frequency follows a long-tail distribution: `[pause]` is the most common tag while `[confused]` and `[scared]` appear rarely. The audio encoder (Whisper-based) is kept frozen throughout training; only the language model backbone and projector are fine-tuned.
+
+### 3.2 Stage 1: Supervised Fine-Tuning (SFT)
+
+We fine-tune `mistralai/Voxtral-Mini-3B-2507` [1] using LoRA [3] with the following configuration:
+
+| Hyperparameter | Value |
+|---|---|
+| LoRA rank | 64 |
+| LoRA alpha | 128 |
+| LoRA dropout | 0.05 |
+| Target modules | `q/k/v/o_proj`, `gate/up/down_proj`, `multi_modal_projector` |
+| Learning rate | 2e-4 |
+| LR schedule | Cosine decay |
+| Epochs | 3 |
+| Batch size | 2 |
+| Gradient accumulation steps | 8 (effective batch = 16) |
+| NEFTune noise alpha | 5.0 |
+| Precision | bf16 |
+| Hardware | NVIDIA A10G (24 GB) |
+| Training time | ~25 minutes |
+| Trainable parameters | 124.8M / 4.8B (2.6%) |
+
+The SFT objective is standard next-token prediction (cross-entropy) on the tagged transcription tokens, conditioned on the audio encoder outputs. NEFTune [5] noise is applied to the input embeddings to reduce overfitting on the small training set.
+
+### 3.3 Stage 2: Rejection Sampling Fine-Tuning (RAFT)
+
+Following Yuan et al. [4] and inspired by Voxtral's own SFT→DPO recipe [1] and GRPO-based ASR work [2], we apply a rejection sampling stage to refine tag generation quality. The procedure is as follows:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    RAFT Training Pipeline                        │
+├─────────────────────────────────────────────────────────────────┤
+│                                                                  │
+│  Training Set (808 samples)                                      │
+│         │                                                        │
+│         ▼                                                        │
+│  ┌─────────────────────────────────────────┐                    │
+│  │  Generate N=4 completions per sample    │                    │
+│  │  (temperature=0.7, top_p=0.9)           │                    │
+│  └─────────────────────────────────────────┘                    │
+│         │                                                        │
+│         ▼                                                        │
+│  ┌─────────────────────────────────────────┐                    │
+│  │  Score each completion:                 │                    │
+│  │  R = 0.4 × (1 - WER)                   │                    │
+│  │    + 0.4 × Tag_F1                       │                    │
+│  │    + 0.2 × (1 - hallucination_rate)     │                    │
+│  └─────────────────────────────────────────┘                    │
+│         │                                                        │
+│         ▼                                                        │
+│  ┌─────────────────────────────────────────┐                    │
+│  │  Keep best completion per sample        │                    │
+│  │  Filter bottom 10% (reward ≤ 0.954)     │                    │
+│  │  → 727 curated samples remain           │                    │
+│  │    (avg reward: 0.980)                  │                    │
+│  └─────────────────────────────────────────┘                    │
+│         │                                                        │
+│         ▼                                                        │
+│  ┌─────────────────────────────────────────┐                    │
+│  │  SFT on curated data                    │                    │
+│  │  (lr=5e-5, 1 epoch, ~7 minutes)         │                    │
+│  └─────────────────────────────────────────┘                    │
+│                                                                  │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+**Figure 1.** RAFT training pipeline. For each training sample, four completions are generated and scored by a rule-based reward function balancing transcription accuracy, tag quality, and hallucination rate. Only high-reward completions are retained for the final SFT pass.
+
+The reward function is designed to balance three objectives: transcription accuracy (WER), tag generation quality (Tag F1), and avoidance of hallucinated tags. The 0.4/0.4/0.2 weighting reflects an equal priority on accuracy and expressiveness, with a penalty for hallucination. After filtering, 727 of the original 808 training samples remain, with a mean reward of 0.980. The RAFT SFT stage trains for one epoch at a reduced learning rate of 5e-5, completing in approximately 7 minutes. Final training loss: 0.021.
+
+The full two-stage pipeline is illustrated below:
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│                  Evoxtral Training Overview                       │
+├──────────────────────────────────────────────────────────────────┤
+│                                                                   │
+│  Voxtral-Mini-3B-2507 (frozen audio encoder)                      │
+│         │                                                         │
+│         ▼                                                         │
+│  ┌─────────────────────┐                                         │
+│  │   Stage 1: SFT      │  lr=2e-4, 3 epochs, NEFTune             │
+│  │   (808 samples)     │  ~25 min on A10G                        │
+│  └─────────────────────┘                                         │
+│         │                                                         │
+│         ▼                                                         │
+│  ┌─────────────────────┐                                         │
+│  │  Evoxtral-SFT       │  WER=4.47%, Tag F1=67.2%               │
+│  └─────────────────────┘                                         │
+│         │                                                         │
+│         ▼                                                         │
+│  ┌���────────────────────┐                                         │
+│  │   Stage 2: RAFT     │  N=4 samples, reward filter             │
+│  │   (727 samples)     │  lr=5e-5, 1 epoch, ~7 min              │
+│  └─────────────────────┘                                         │
+│         │                                                         │
+│         ▼                                                         │
+│  ┌─────────────────────┐                                         │
+│  │  Evoxtral-RL        │  WER=5.12%, Tag F1=69.4%               │
+│  └─────────────────────┘                                         │
+│                                                                   │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+**Figure 2.** Full two-stage Evoxtral training pipeline from base Voxtral-Mini-3B to the two released model variants.
+
+---
+
+## 4. Evaluation
+
+### 4.1 Evoxtral-Bench
+
+We evaluate on **Evoxtral-Bench**, a held-out benchmark of 50 test samples drawn from the 101-sample test split. This subset enables fast iteration while maintaining statistical validity. We compute seven evaluation metrics:
+
+- **WER** (Word Error Rate, ↓) — computed via `jiwer` [7], tags stripped before comparison
+- **CER** (Character Error Rate, ↓) — character-level transcription accuracy via `jiwer` [7]
+- **Tag F1** (↑) — token-level F1 on predicted vs. reference tag sequences
+- **Tag Precision** (↑) — fraction of predicted tags present in reference
+- **Tag Recall** (↑) — fraction of reference tags predicted by the model
+- **Tag Hallucination Rate** (↓) — fraction of predicted tags not present in any reference tag
+- **Emphasis F1** (↑) — F1 on emphasized word spans (ellipsis and capitalization markers)
+
+Per-tag F1 is additionally computed across all 17 tag types to diagnose per-class performance.
+
+### 4.2 Core Results
+
+**Table 1.** Core evaluation results on Evoxtral-Bench (50 samples). Bold indicates best per metric. ↓ lower is better; ↑ higher is better.
+
+| Metric | Base Voxtral | Evoxtral SFT | Evoxtral RL | Best |
+|---|---|---|---|---|
+| WER ↓ | 6.64% | **4.47%** | 5.12% | SFT |
+| CER ↓ | 2.72% | **1.23%** | 1.48% | SFT |
+| Tag F1 ↑ | 22.0% | 67.2% | **69.4%** | RL |
+| Tag Precision ↑ | 22.0% | 67.4% | **68.5%** | RL |
+| Tag Recall ↑ | 22.0% | 69.4% | **72.7%** | RL |
+| Emphasis F1 ↑ | 42.0% | 84.0% | **86.0%** | RL |
+| Tag Hallucination Rate ↓ | 0.0% | **19.3%** | 20.2% | SFT |
+
+The base Voxtral model achieves 22.0% Tag F1, suggesting some limited native capability to produce expressive tokens but with low precision and recall. SFT provides the dominant improvement: WER decreases by 33% relative (6.64% → 4.47%) and Tag F1 increases by 45 percentage points (22.0% → 67.2%). RAFT further refines tag metrics: Tag F1 improves by 2.2pp (67.2% → 69.4%), Tag Recall by 3.3pp (69.4% → 72.7%), and Emphasis F1 by 2.0pp (84.0% → 86.0%). However, RAFT introduces a small WER regression (4.47% → 5.12%), reflecting a Pareto tradeoff between transcription accuracy and expressive richness.
+
+Tag hallucination — predicted tags absent from the reference — is 19.3% for SFT and 20.2% for RL. The base model has 0% hallucination because it rarely predicts any tags at all.
+
+### 4.3 Per-Tag F1 Breakdown
+
+**Table 2.** Per-tag F1 scores for Evoxtral-SFT and Evoxtral-RL on Evoxtral-Bench. Support indicates the number of test samples containing each tag.
+
+| Tag | SFT F1 | RL F1 | Delta | Support |
+|---|---|---|---|---|
+| [sighs] | 1.000 | 1.000 | — | 9 |
+| [clears throat] | 0.889 | **1.000** | +12.5% | 8 |
+| [gasps] | 0.957 | 0.957 | — | 12 |
+| [pause] | 0.885 | **0.902** | +1.9% | 25 |
+| [nervous] | 0.800 | **0.846** | +5.8% | 13 |
+| [stammers] | **0.889** | 0.842 | -5.3% | 8 |
+| [laughs] | 0.800 | **0.815** | +1.9% | 12 |
+| [sad] | 0.667 | **0.750** | +12.4% | 4 |
+| [whispers] | 0.636 | **0.667** | +4.9% | 13 |
+| [crying] | **0.750** | 0.571 | -23.9% | 5 |
+| [excited] | **0.615** | 0.571 | -7.2% | 5 |
+| [shouts] | 0.400 | **0.500** | +25.0% | 3 |
+| [calm] | 0.200 | **0.400** | +100.0% | 6 |
+| [frustrated] | 0.444 | 0.444 | — | 3 |
+| [angry] | 0.667 | 0.667 | — | 2 |
+| [confused] | 0.000 | 0.000 | — | 1 |
+| [scared] | 0.000 | 0.000 | — | 1 |
+
+RAFT improves 9 tags, maintains 4 stable, and regresses on 3. The largest gains are observed for `[calm]` (+100%, 0.200 → 0.400), `[shouts]` (+25.0%), `[clears throat]` (+12.5%), and `[sad]` (+12.4%). Regressions are noted for `[crying]` (-23.9%), `[excited]` (-7.2%), and `[stammers]` (-5.3%). The two zero-F1 tags (`[confused]`, `[scared]`) each appear only once in the test set, making estimation unreliable.
+
+---
+
+## 5. Analysis and Discussion
+
+**SFT as the primary driver of improvement.** The SFT stage accounts for the vast majority of the performance gain: WER drops 33% relative and Tag F1 increases by 45 percentage points. This aligns with findings from GRPO-based ASR work [2, 8, 9] suggesting that a well-supervised initial adaptation is a strong foundation for subsequent RL refinement.
+
+**The WER-Tag tradeoff.** RAFT improves tag metrics at the cost of a modest WER regression (4.47% → 5.12%). This suggests the existence of a Pareto frontier between transcription accuracy and expressive richness: optimizing for tag generation pushes the model toward producing more tags, which can introduce minor word-level errors. This motivates releasing two model variants — Evoxtral-SFT for accuracy-critical applications (e.g., professional transcription) and Evoxtral-RL for expressiveness-critical applications (e.g., downstream TTS synthesis with ElevenLabs v3 [6]).
+
+**Tag hallucination.** Approximately 20% of predicted tags are not present in the reference transcription. Hallucination occurs when the model infers an expressive tone from acoustic cues that are present in the audio but absent or differently annotated in the reference. This may partly reflect annotation noise in synthetic data rather than pure model error. Future work should address this with contrastive or calibration-based training objectives.
+
+**Effect of NEFTune.** Applying NEFTune [5] with noise alpha = 5.0 during SFT provided a regularization benefit on the small 808-sample training set, consistent with Jain et al.'s [5] findings on instruction-following tasks. Ablating this component was not feasible within hackathon time constraints but remains a planned analysis.
+
+**Rare tag performance.** Tags with very low test support (`[confused]`, `[scared]`, support = 1) have zero F1, which is uninformative. Tags with support 2–6 (`[angry]`, `[shouts]`, `[calm]`, `[sad]`, `[frustrated]`) show high variance in F1 estimates. A larger, more balanced evaluation set would provide more reliable per-tag metrics.
+
+**Reward function design.** The RAFT reward R = 0.4×(1−WER) + 0.4×Tag\_F1 + 0.2×(1−hallucination\_rate) explicitly encodes the design preference for equal weight on accuracy and expressiveness. The 0.2 weight on hallucination acts as a weak regularizer. An ablation across reward weightings would quantify the sensitivity of the final model to this design choice.
+
+---
+
+## 6. Limitations
+
+The following limitations apply to this work:
+
+- **Synthetic training data.** All 1,010 samples are synthesized using ElevenLabs TTS v3 [6]. The acoustic properties of synthetic speech differ from natural human speech (e.g., prosodic consistency, noise, spontaneous disfluencies). Performance on natural speech recordings may differ from the reported results.
+- **Tag hallucination.** Approximately 20% of predicted tags in the RL model are not present in the reference. This may limit applicability in settings requiring precise expressive annotation.
+- **Rare tag coverage.** Seventeen tag types are represented in the dataset, but several occur in fewer than 5 test samples. Per-tag F1 estimates for rare categories are unreliable.
+- **English only.** The dataset and training are English-only. Generalization to other languages is not evaluated.
+- **Small dataset.** 808 training samples is a small fine-tuning set. Scaling to thousands or tens of thousands of examples with natural speech could substantially improve performance.
+- **Evaluation scope.** Evoxtral-Bench covers 50 test samples for fast iteration. A larger evaluation set would yield more statistically robust estimates across all metrics and tag types.
+
+---
+
+## 7. Conclusion
+
+We presented Evoxtral, a LoRA-adapted version of Voxtral-Mini-3B-2507 [1] that produces expressive tagged transcriptions using ElevenLabs v3 audio tags [6]. Our two-stage training pipeline — SFT followed by RAFT [4] — demonstrates that expressive tagging capability can be effectively injected into a pre-trained ASR model with parameter-efficient fine-tuning [3, 10].
+
+SFT achieves a 33% relative WER reduction and a 45 percentage-point improvement in Tag F1 over the base model. RAFT further improves tag recall and F1 by targeting tag generation quality directly through a rule-based reward signal, at a modest transcription accuracy cost. The two resulting model variants cover a Pareto frontier between accuracy and expressiveness, allowing practitioners to select the appropriate trade-off for their application.
+
+Future directions include: (1) collecting natural speech data with crowd-sourced expressive annotations to reduce the synthetic data gap; (2) replacing RAFT with GRPO [2] or DPO [1] for more sample-efficient RL training; (3) expanding to multilingual settings leveraging Voxtral's multilingual audio encoder; and (4) developing joint ASR+TTS evaluation protocols that measure downstream TTS quality when Evoxtral output is used as input to ElevenLabs v3 [6].
+
+---
+
+## References
+
+[1] Mistral AI. "Voxtral." arXiv:2507.13264, 2025. https://arxiv.org/abs/2507.13264
+
+[2] "Group Relative Policy Optimization for Speech Recognition." arXiv:2509.01939, 2025. https://arxiv.org/abs/2509.01939
+
+[3] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. "LoRA: Low-Rank Adaptation of Large Language Models." In *Proceedings of ICLR*, 2022. arXiv:2106.09685. https://arxiv.org/abs/2106.09685
+
+[4] Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models." arXiv:2308.01825, 2023. https://arxiv.org/abs/2308.01825
+
+[5] Jain, N., Chiang, P., Yeh, Y., Kirchenbauer, J., Chu, C., Somepalli, G., Bartoldson, B., Kailkhura, B., Schwarzschild, A., Saha, A., Goldblum, M., Geiping, J., and Goldstein, T. "NEFTune: Noisy Embeddings Improve Instruction Finetuning." In *Proceedings of ICLR*, 2024. arXiv:2310.05914. https://arxiv.org/abs/2310.05914
+
+[6] ElevenLabs. "Text-to-Speech v3 Audio Tags." ElevenLabs Developer Documentation, 2025. https://elevenlabs.io/docs/api-reference/text-to-speech
+
+[7] "JiWER: Evaluate your speech recognition system." Python library for ASR evaluation metrics. https://github.com/jitsi/jiwer
+
+[8] "Advancing Speech Understanding in Speech-Aware Language Models with GRPO." arXiv:2509.16990, 2025. https://arxiv.org/abs/2509.16990
+
+[9] "Explore the Reinforcement Learning for LLM-based ASR and TTS System." arXiv:2509.18569, 2025. https://arxiv.org/abs/2509.18569
+
+[10] Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. "PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods." HuggingFace, 2022. https://github.com/huggingface/peft
+
+[11] Morris, A.C., Maier, V., and Green, P. "From WER and RIL to MER and WIL: Improved evaluation measures for connected speech recognition." In *Proceedings of INTERSPEECH*, 2004.
+
+[12] "Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering." arXiv:2503.11197, 2025. https://arxiv.org/abs/2503.11197

docs/technical_report.tex

@@ -0,0 +1,383 @@
+\documentclass[11pt,a4paper]{article}
+
+% ── Packages ──────────────────────────────────────────────────────────
+\usepackage[utf8]{inputenc}
+\usepackage[T1]{fontenc}
+\usepackage{lmodern}
+\usepackage[margin=1in]{geometry}
+\usepackage{amsmath,amssymb}
+\usepackage{graphicx}
+\usepackage{booktabs}
+\usepackage{hyperref}
+\usepackage{xcolor}
+\usepackage{tikz}
+\usetikzlibrary{arrows.meta,positioning,shapes.geometric,fit,calc}
+\usepackage{caption}
+\usepackage{subcaption}
+\usepackage{enumitem}
+\usepackage{multirow}
+\usepackage{array}
+\usepackage{tabularx}
+\usepackage{float}
+\usepackage{natbib}
+\usepackage{authblk}
+\usepackage{microtype}
+
+\hypersetup{
+    colorlinks=true,
+    linkcolor=blue!60!black,
+    citecolor=blue!60!black,
+    urlcolor=blue!60!black,
+}
+
+% ── Inline tag command ────────────────────────────────────────────────
+\newcommand{\etag}[1]{\texttt{[#1]}}
+
+% ── Title ─────────────────────────────────────────────────────────────
+\title{\textbf{Evoxtral: Expressive Tagged Transcription\\via Supervised Fine-Tuning and Rejection Sampling}}
+\author[1]{Yongkang Zou}
+\affil[1]{Mistral AI Online Hackathon 2026 --- W\&B Fine-Tuning Track}
+\date{}
+
+\begin{document}
+\maketitle
+
+% ══════════════════════════════════════════════════════════════════════
+\begin{abstract}
+Standard automatic speech recognition (ASR) systems discard paralinguistic and expressive information present in spoken audio, producing plain text that fails to capture sighs, laughs, hesitations, emotional tone, and other prosodic cues. We present \textbf{Evoxtral}, a LoRA adapter for \texttt{Voxtral-Mini-3B-2507}~\cite{voxtral} that produces transcriptions enriched with inline expressive audio tags drawn from the ElevenLabs v3 tag vocabulary~\cite{elevenlabs}. We apply a two-stage post-training pipeline: supervised fine-tuning (SFT) followed by rejection sampling fine-tuning (RAFT)~\cite{yuan2023rft}. SFT reduces word error rate (WER) by 33\% relative (6.64\%~$\to$~4.47\%) and increases Tag~F1 from 22.0\% to 67.2\%. The subsequent RAFT stage further improves Tag~F1 to 69.4\% and Tag~Recall to 72.7\% at a marginal WER cost. We release two model variants, a serverless inference API, and a live interactive demo.
+\end{abstract}
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Introduction}
+
+Modern ASR pipelines excel at converting speech to text with low word error rates, yet they systematically strip out the expressive dimension of human communication. When a speaker sighs before a sentence, laughs nervously mid-phrase, or whispers for emphasis, these paralinguistic signals carry meaning that plain transcription cannot represent. This information is especially critical for downstream text-to-speech (TTS) synthesis: next-generation TTS systems such as ElevenLabs v3~\cite{elevenlabs} consume inline expressive tags to control prosody, affect, and delivery at a fine-grained level.
+
+We ask: \emph{can a multimodal audio-language model be trained to produce ASR output that preserves expressive content through inline tags?} We answer affirmatively with \textbf{Evoxtral}, built on Voxtral-Mini-3B-2507~\cite{voxtral}.
+
+\paragraph{Why Voxtral.} We chose Voxtral-Mini-3B as our base model for several reasons. First, as a \emph{generative} audio-language model (rather than a CTC or encoder-only ASR), Voxtral decodes transcriptions autoregressively---meaning it can naturally produce arbitrary inline tokens such as \etag{laughs} or \etag{nervous} within the text stream, without requiring architectural changes. Traditional ASR models constrain their output vocabulary to words and punctuation; Voxtral's LLM decoder has no such limitation. Second, Voxtral's compact 3B-parameter architecture makes LoRA fine-tuning feasible on a single A10G GPU within hackathon time constraints, while still delivering competitive ASR quality. Third, Voxtral was itself trained with post-training alignment (SFT~+~DPO~+~Online~DPO)~\cite{voxtral}, meaning the model is already instruction-following and amenable to further fine-tuning---a strong foundation for adding new capabilities. Finally, as a Mistral AI model released under Apache~2.0, Voxtral aligns with the hackathon's focus on the Mistral ecosystem and enables open redistribution of our adapters.
+
+Our approach fine-tunes Voxtral-Mini-3B-2507 using parameter-efficient LoRA adapters~\cite{hu2021lora} on a synthetically generated dataset of expressive speech paired with tagged transcriptions.
+
+\medskip
+To illustrate the contrast, consider the following example:
+
+\begin{quote}
+\textbf{Standard ASR:} ``So I was thinking maybe we could try that new restaurant downtown. I mean if you're free this weekend.''
+
+\textbf{Evoxtral:} ``\etag{nervous} So\ldots\ \etag{stammers} I was thinking maybe we could\ldots\ \etag{clears throat} try that new restaurant downtown? \etag{laughs nervously} I mean, if you're free this weekend?''
+\end{quote}
+
+\noindent The Evoxtral output captures hesitation, nervous laughter, and a throat clear---paralinguistic content that is acoustically present but conventionally discarded. We make the following contributions:
+
+\begin{enumerate}[leftmargin=*,itemsep=2pt]
+    \item A synthetic dataset of 1,010 expressive speech samples paired with tagged transcriptions across 17 ElevenLabs v3 tag types.
+    \item A two-stage fine-tuning recipe (SFT~$\to$~RAFT) for expressive ASR using LoRA on Voxtral-Mini-3B.
+    \item A custom evaluation benchmark, \textbf{Evoxtral-Bench}, with seven metrics covering both transcription accuracy and tag generation quality.
+    \item Two released model variants optimized for different use cases: accuracy-critical (SFT) and expressiveness-critical (RL).
+\end{enumerate}
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Related Work}
+
+\paragraph{Voxtral~\cite{voxtral}.} Voxtral-Mini-3B-2507 is a multimodal audio-language model released by Mistral~AI. It is built on a Whisper-based audio encoder fused with a Mistral language model backbone, trained via SFT, DPO, and online DPO. Our work builds directly on this foundation, adding expressive tagging capability via LoRA adaptation.
+
+\paragraph{Reinforcement Learning for LLM-based ASR~\cite{grpo_asr}.} Shi et al.\ apply group relative policy optimization (GRPO) to LLM-based ASR, achieving an 18\% relative WER reduction without paired preference data. Closely related work~\cite{grpo_speech,rl_asr_tts,rl_audio_qa} further demonstrates that RL-based training consistently outperforms SFT alone for speech understanding tasks. Our RAFT stage is philosophically aligned with this line of work but uses rule-based rejection sampling rather than policy gradient methods, making it simpler to implement and more stable to train.
+
+\paragraph{LoRA~\cite{hu2021lora}.} Low-rank adaptation inserts trainable low-rank matrices into the attention projections and feed-forward layers of a frozen base model, reducing the number of trainable parameters by orders of magnitude while matching full fine-tuning performance. We use LoRA with rank~64 and alpha~128, implemented via HuggingFace PEFT~\cite{peft}.
+
+\paragraph{Rejection Sampling Fine-Tuning~\cite{yuan2023rft}.} Yuan et al.\ propose generating multiple model completions for each training input, scoring them with a reward function, and performing SFT on the highest-scoring completions. This approach is computationally simpler than policy gradient methods while still providing a reinforcement learning signal.
+
+\paragraph{NEFTune~\cite{neftune}.} Jain et al.\ demonstrate that adding uniform random noise to embedding vectors during training improves instruction-following performance. We apply NEFTune with noise alpha~$=5.0$ during SFT to regularize training on our small dataset.
+
+\paragraph{ElevenLabs v3 Audio Tags~\cite{elevenlabs}.} ElevenLabs v3 TTS introduces a structured vocabulary of inline expressive tags that control how synthesized speech is delivered. These tags---such as \etag{sighs}, \etag{laughs}, \etag{whispers}, and \etag{nervous}---are the target output vocabulary for Evoxtral.
+
+\paragraph{ASR Evaluation~\cite{jiwer,morris2004wer}.} We compute WER and CER using the \texttt{jiwer} library~\cite{jiwer}, following standard definitions from Morris et al.~\cite{morris2004wer}.
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Method}
+
+\subsection{Dataset}
+
+We construct a synthetic dataset of 1,010 audio samples generated using the ElevenLabs TTS v3 API~\cite{elevenlabs}. Each sample consists of a short spoken utterance (5--30\,s) paired with a reference tagged transcription containing inline expressive tags. The dataset covers 17~tag types: \etag{sighs}, \etag{laughs}, \etag{whispers}, \etag{nervous}, \etag{frustrated}, \etag{clears~throat}, \etag{pause}, \etag{excited}, \etag{stammers}, \etag{gasps}, \etag{sad}, \etag{angry}, \etag{calm}, \etag{crying}, \etag{shouts}, \etag{confused}, and \etag{scared}.
+
+The dataset is split into 808~training, 101~validation, and 101~test samples. Tag frequency follows a long-tail distribution: \etag{pause} is the most common while \etag{confused} and \etag{scared} appear rarely. The Whisper-based audio encoder is kept frozen throughout training; only the language model backbone and multi-modal projector are fine-tuned.
+
+\subsection{Stage 1: Supervised Fine-Tuning (SFT)}
+
+We fine-tune Voxtral-Mini-3B-2507~\cite{voxtral} using LoRA~\cite{hu2021lora} with the configuration shown in Table~\ref{tab:sft_config}.
+
+\begin{table}[h]
+\centering
+\caption{SFT hyperparameters.}
+\label{tab:sft_config}
+\begin{tabular}{@{}ll@{}}
+\toprule
+\textbf{Hyperparameter} & \textbf{Value} \\
+\midrule
+LoRA rank & 64 \\
+LoRA alpha & 128 \\
+LoRA dropout & 0.05 \\
+Target modules & \texttt{q/k/v/o\_proj}, \texttt{gate/up/down\_proj}, \texttt{mm\_projector} \\
+Learning rate & $2 \times 10^{-4}$ \\
+LR schedule & Cosine decay \\
+Epochs & 3 \\
+Batch size & 2 (effective 16 via gradient accumulation $\times 8$) \\
+NEFTune noise alpha & 5.0 \\
+Precision & bf16 \\
+Hardware & NVIDIA A10G (24\,GB) \\
+Training time & $\sim$25 minutes \\
+Trainable parameters & 124.8\,M / 4.8\,B (2.6\%) \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+The SFT objective is standard next-token prediction (cross-entropy) on the tagged transcription tokens, conditioned on the audio encoder outputs. NEFTune~\cite{neftune} noise is applied to the input embeddings to reduce overfitting on the small training set.
+
+\subsection{Stage 2: Rejection Sampling Fine-Tuning (RAFT)}
+
+Following Yuan et al.~\cite{yuan2023rft} and inspired by Voxtral's own SFT$\to$DPO recipe~\cite{voxtral} and GRPO-based ASR work~\cite{grpo_asr}, we apply a rejection sampling stage to refine tag generation quality. The pipeline is illustrated in Figure~\ref{fig:raft_pipeline}.
+
+\begin{figure}[t]
+\centering
+\begin{tikzpicture}[
+    node distance=1.0cm,
+    box/.style={draw, rounded corners, minimum width=6.5cm, minimum height=0.9cm, align=center, font=\small},
+    arrow/.style={-{Stealth[length=3mm]}, thick},
+]
+    \node[box, fill=blue!8] (data) {Training Set (808 samples)};
+    \node[box, fill=orange!10, below=of data] (gen) {Generate $N{=}4$ completions per sample\\($T{=}0.7$, top-$p{=}0.9$)};
+    \node[box, fill=red!8, below=of gen] (score) {Score each: $R = 0.4(1{-}\text{WER}) + 0.4\,\text{Tag\_F1}$\\$+ \; 0.2(1{-}\text{hallucination\_rate})$};
+    \node[box, fill=yellow!10, below=of score] (filter) {Keep best per sample, filter bottom 10\%\\$\to$ 727 curated samples (avg reward: 0.980)};
+    \node[box, fill=green!10, below=of filter] (sft) {SFT on curated data\\(lr$\,{=}\,5{\times}10^{-5}$, 1 epoch, $\sim$7\,min)};
+
+    \draw[arrow] (data) -- (gen);
+    \draw[arrow] (gen) -- (score);
+    \draw[arrow] (score) -- (filter);
+    \draw[arrow] (filter) -- (sft);
+\end{tikzpicture}
+\caption{RAFT training pipeline. For each training sample, four completions are generated and scored by a rule-based reward function balancing transcription accuracy, tag quality, and hallucination avoidance. Only high-reward completions are retained for the final SFT pass.}
+\label{fig:raft_pipeline}
+\end{figure}
+
+The reward function balances three objectives: transcription accuracy (WER), tag generation quality (Tag~F1), and avoidance of hallucinated tags:
+\begin{equation}
+    R = 0.4 \times (1 - \text{WER}) + 0.4 \times \text{Tag\_F1} + 0.2 \times (1 - \text{Hallucination\_Rate})
+    \label{eq:reward}
+\end{equation}
+
+The 0.4/0.4/0.2 weighting reflects an equal priority on accuracy and expressiveness, with a penalty for hallucination. After filtering the bottom 10\% by reward (threshold: $R > 0.954$), 727 of the original 808 training samples remain with a mean reward of 0.980. The RAFT SFT stage trains for one epoch at a reduced learning rate of $5 \times 10^{-5}$, completing in approximately 7~minutes with a final training loss of 0.021.
+
+The full two-stage pipeline overview is shown in Figure~\ref{fig:overview}.
+
+\begin{figure}[t]
+\centering
+\begin{tikzpicture}[
+    node distance=1.0cm,
+    box/.style={draw, rounded corners, minimum width=5.5cm, minimum height=0.8cm, align=center, font=\small},
+    result/.style={draw, rounded corners, minimum width=5.5cm, minimum height=0.8cm, align=center, font=\small, fill=green!8},
+    arrow/.style={-{Stealth[length=3mm]}, thick},
+    label/.style={font=\scriptsize\itshape, text=gray!70!black},
+]
+    \node[box, fill=blue!10] (base) {Voxtral-Mini-3B-2507\\(frozen audio encoder)};
+    \node[box, fill=orange!10, below=of base] (sft) {Stage 1: SFT\\(808 samples, 3 epochs)};
+    \node[result, below=of sft] (sft_r) {\textbf{Evoxtral-SFT}\\WER\,=\,4.47\%, Tag F1\,=\,67.2\%};
+    \node[box, fill=red!8, below=of sft_r] (raft) {Stage 2: RAFT\\(727 curated samples, 1 epoch)};
+    \node[result, below=of raft] (rl_r) {\textbf{Evoxtral-RL}\\WER\,=\,5.12\%, Tag F1\,=\,69.4\%};
+
+    \draw[arrow] (base) -- (sft) node[midway, right=0.3cm, label] {lr\,$=$\,$2{\times}10^{-4}$, NEFTune};
+    \draw[arrow] (sft) -- (sft_r);
+    \draw[arrow] (sft_r) -- (raft) node[midway, right=0.3cm, label] {lr\,$=$\,$5{\times}10^{-5}$, reward filter};
+    \draw[arrow] (raft) -- (rl_r);
+\end{tikzpicture}
+\caption{Full two-stage Evoxtral training pipeline from base Voxtral-Mini-3B to the two released model variants.}
+\label{fig:overview}
+\end{figure}
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Evaluation}
+
+\subsection{Evoxtral-Bench}
+
+We evaluate on \textbf{Evoxtral-Bench}, a held-out benchmark of 50 test samples drawn from the 101-sample test split. We compute seven evaluation metrics:
+
+\begin{itemize}[leftmargin=*,itemsep=1pt]
+    \item \textbf{WER} (Word Error Rate, $\downarrow$) --- via \texttt{jiwer}~\cite{jiwer}, tags stripped before comparison
+    \item \textbf{CER} (Character Error Rate, $\downarrow$) --- character-level accuracy via \texttt{jiwer}
+    \item \textbf{Tag F1} ($\uparrow$) --- token-level F1 on predicted vs.\ reference tag multisets
+    \item \textbf{Tag Precision} ($\uparrow$) --- fraction of predicted tags present in reference
+    \item \textbf{Tag Recall} ($\uparrow$) --- fraction of reference tags captured by the model
+    \item \textbf{Tag Hallucination Rate} ($\downarrow$) --- fraction of predicted tags absent from reference
+    \item \textbf{Emphasis F1} ($\uparrow$) --- F1 on CAPITALIZED emphasis words
+\end{itemize}
+
+Per-tag F1 is additionally computed across all 17 tag types.
+
+\subsection{Core Results}
+
+\begin{table}[t]
+\centering
+\caption{Core evaluation results on Evoxtral-Bench (50 samples). Bold indicates best per metric. $\downarrow$ lower is better; $\uparrow$ higher is better.}
+\label{tab:core_results}
+\begin{tabular}{@{}lcccc@{}}
+\toprule
+\textbf{Metric} & \textbf{Base Voxtral} & \textbf{Evoxtral SFT} & \textbf{Evoxtral RL} & \textbf{Best} \\
+\midrule
+WER $\downarrow$ & 6.64\% & \textbf{4.47\%} & 5.12\% & SFT \\
+CER $\downarrow$ & 2.72\% & \textbf{1.23\%} & 1.48\% & SFT \\
+Tag F1 $\uparrow$ & 22.0\% & 67.2\% & \textbf{69.4\%} & RL \\
+Tag Precision $\uparrow$ & 22.0\% & 67.4\% & \textbf{68.5\%} & RL \\
+Tag Recall $\uparrow$ & 22.0\% & 69.4\% & \textbf{72.7\%} & RL \\
+Emphasis F1 $\uparrow$ & 42.0\% & 84.0\% & \textbf{86.0\%} & RL \\
+Tag Hallucination $\downarrow$ & 0.0\% & \textbf{19.3\%} & 20.2\% & SFT \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+The base Voxtral model achieves 22.0\% Tag~F1, suggesting some limited native capability to produce expressive tokens but with low precision and recall. SFT provides the dominant improvement: WER decreases by 33\% relative (6.64\%~$\to$~4.47\%) and Tag~F1 increases by 45~percentage points (22.0\%~$\to$~67.2\%). RAFT further refines tag metrics: Tag~F1 improves by 2.2\,pp (67.2\%~$\to$~69.4\%), Tag~Recall by 3.3\,pp (69.4\%~$\to$~72.7\%), and Emphasis~F1 by 2.0\,pp (84.0\%~$\to$~86.0\%). However, RAFT introduces a small WER regression (4.47\%~$\to$~5.12\%), reflecting a Pareto tradeoff between transcription accuracy and expressive richness.
+
+Tag hallucination---predicted tags absent from the reference---is 19.3\% for SFT and 20.2\% for RL. The base model has 0\% hallucination because it rarely predicts any tags at all.
+
+\subsection{Per-Tag F1 Breakdown}
+
+\begin{table}[t]
+\centering
+\caption{Per-tag F1 scores for SFT and RL on Evoxtral-Bench. Support indicates the number of test samples containing each tag.}
+\label{tab:per_tag}
+\begin{tabular}{@{}lcccc@{}}
+\toprule
+\textbf{Tag} & \textbf{SFT F1} & \textbf{RL F1} & \textbf{$\Delta$} & \textbf{Support} \\
+\midrule
+\etag{sighs} & 1.000 & 1.000 & --- & 9 \\
+\etag{clears throat} & 0.889 & \textbf{1.000} & +12.5\% & 8 \\
+\etag{gasps} & 0.957 & 0.957 & --- & 12 \\
+\etag{pause} & 0.885 & \textbf{0.902} & +1.9\% & 25 \\
+\etag{nervous} & 0.800 & \textbf{0.846} & +5.8\% & 13 \\
+\etag{stammers} & \textbf{0.889} & 0.842 & $-$5.3\% & 8 \\
+\etag{laughs} & 0.800 & \textbf{0.815} & +1.9\% & 12 \\
+\etag{sad} & 0.667 & \textbf{0.750} & +12.4\% & 4 \\
+\etag{whispers} & 0.636 & \textbf{0.667} & +4.9\% & 13 \\
+\etag{crying} & \textbf{0.750} & 0.571 & $-$23.9\% & 5 \\
+\etag{excited} & \textbf{0.615} & 0.571 & $-$7.2\% & 5 \\
+\etag{shouts} & 0.400 & \textbf{0.500} & +25.0\% & 3 \\
+\etag{calm} & 0.200 & \textbf{0.400} & +100\% & 6 \\
+\etag{frustrated} & 0.444 & 0.444 & --- & 3 \\
+\etag{angry} & 0.667 & 0.667 & --- & 2 \\
+\etag{confused} & 0.000 & 0.000 & --- & 1 \\
+\etag{scared} & 0.000 & 0.000 & --- & 1 \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+RAFT improves 9~tags, maintains 4~stable, and regresses on~3 (Table~\ref{tab:per_tag}). The largest gains are observed for \etag{calm} (+100\%, 0.200~$\to$~0.400), \etag{shouts} (+25.0\%), \etag{clears~throat} (+12.5\%), and \etag{sad} (+12.4\%). Regressions are noted for \etag{crying} ($-$23.9\%), \etag{excited} ($-$7.2\%), and \etag{stammers} ($-$5.3\%). The two zero-F1 tags (\etag{confused}, \etag{scared}) each appear only once in the test set, making estimation unreliable.
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Analysis and Discussion}
+
+\paragraph{SFT as the primary driver of improvement.} The SFT stage accounts for the vast majority of the performance gain: WER drops 33\% relative and Tag~F1 increases by 45~percentage points. This aligns with findings from GRPO-based ASR work~\cite{grpo_asr,grpo_speech,rl_asr_tts} suggesting that a well-supervised initial adaptation is a strong foundation for subsequent RL refinement.
+
+\paragraph{The WER--Tag tradeoff.} RAFT improves tag metrics at the cost of a modest WER regression (4.47\%~$\to$~5.12\%). This suggests the existence of a Pareto frontier between transcription accuracy and expressive richness: optimizing for tag generation pushes the model toward producing more tags, which can introduce minor word-level errors. This motivates releasing two model variants---Evoxtral-SFT for accuracy-critical applications (e.g., professional transcription) and Evoxtral-RL for expressiveness-critical applications (e.g., downstream TTS synthesis with ElevenLabs v3~\cite{elevenlabs}).
+
+\paragraph{Tag hallucination.} Approximately 20\% of predicted tags are not present in the reference transcription. Hallucination may occur when the model infers an expressive tone from acoustic cues that are present in the audio but absent or differently annotated in the reference. This may partly reflect annotation noise in synthetic data rather than pure model error. Future work should address this with contrastive or calibration-based training objectives.
+
+\paragraph{Effect of NEFTune.} Applying NEFTune~\cite{neftune} with noise alpha~$=5.0$ during SFT provided a regularization benefit on the small 808-sample training set, consistent with Jain et al.'s findings on instruction-following tasks. Ablating this component was not feasible within hackathon time constraints but remains a planned analysis.
+
+\paragraph{Rare tag performance.} Tags with very low test support (\etag{confused}, \etag{scared}, support\,$=$\,1) have zero~F1, which is uninformative. Tags with support 2--6 show high variance in F1 estimates. A larger, more balanced evaluation set would provide more reliable per-tag metrics.
+
+\paragraph{Reward function design.} The RAFT reward (Equation~\ref{eq:reward}) explicitly encodes the design preference for equal weight on accuracy and expressiveness. The 0.2 weight on hallucination acts as a weak regularizer. An ablation across reward weightings would quantify the sensitivity of the final model to this design choice.
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Limitations}
+
+\begin{itemize}[leftmargin=*,itemsep=2pt]
+    \item \textbf{Synthetic training data.} All 1,010 samples are synthesized using ElevenLabs TTS v3~\cite{elevenlabs}. The acoustic properties of synthetic speech differ from natural human speech. Performance on natural speech recordings may differ.
+    \item \textbf{Tag hallucination.} Approximately 20\% of predicted tags in the RL model are not present in the reference, which may limit applicability in settings requiring precise expressive annotation.
+    \item \textbf{Rare tag coverage.} Seventeen tag types are represented, but several occur in fewer than 5 test samples. Per-tag F1 estimates for rare categories are unreliable.
+    \item \textbf{English only.} The dataset and training are English-only. Generalization to other languages is not evaluated.
+    \item \textbf{Small dataset.} 808 training samples is a small fine-tuning set. Scaling to thousands of examples with natural speech could substantially improve performance.
+    \item \textbf{Evaluation scope.} Evoxtral-Bench covers 50 test samples. A larger evaluation set would yield more statistically robust estimates.
+\end{itemize}
+
+% ══════════════════════════════════════════════════════════════════════
+\section{Conclusion}
+
+We presented Evoxtral, a LoRA-adapted version of Voxtral-Mini-3B-2507~\cite{voxtral} that produces expressive tagged transcriptions using ElevenLabs v3 audio tags~\cite{elevenlabs}. Our two-stage training pipeline---SFT followed by RAFT~\cite{yuan2023rft}---demonstrates that expressive tagging capability can be effectively injected into a pre-trained ASR model with parameter-efficient fine-tuning~\cite{hu2021lora,peft}.
+
+SFT achieves a 33\% relative WER reduction and a 45~percentage-point improvement in Tag~F1 over the base model. RAFT further improves tag recall and F1 by targeting tag generation quality directly through a rule-based reward signal, at a modest transcription accuracy cost. The two resulting model variants cover a Pareto frontier between accuracy and expressiveness, allowing practitioners to select the appropriate trade-off for their application.
+
+Future directions include: (1)~collecting natural speech data with crowd-sourced expressive annotations to reduce the synthetic data gap; (2)~replacing RAFT with GRPO~\cite{grpo_asr} or DPO~\cite{voxtral} for more sample-efficient RL training; (3)~expanding to multilingual settings leveraging Voxtral's multilingual audio encoder; and (4)~developing joint ASR+TTS evaluation protocols that measure downstream TTS quality when Evoxtral output is used as input to ElevenLabs v3~\cite{elevenlabs}.
+
+% ══════════════════════════════════════════════════════════════════════
+\bibliographystyle{plainnat}
+
+\begin{thebibliography}{12}
+
+\bibitem[{Mistral AI}(2025)]{voxtral}
+Mistral AI.
+\newblock Voxtral.
+\newblock \emph{arXiv preprint arXiv:2507.13264}, 2025.
+\newblock \url{https://arxiv.org/abs/2507.13264}
+
+\bibitem[{Shi et~al.}(2025)]{grpo_asr}
+Shi, B. et~al.
+\newblock Group relative policy optimization for speech recognition.
+\newblock \emph{arXiv preprint arXiv:2509.01939}, 2025.
+\newblock \url{https://arxiv.org/abs/2509.01939}
+
+\bibitem[{Hu et~al.}(2022)]{hu2021lora}
+Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.
+\newblock {LoRA}: Low-rank adaptation of large language models.
+\newblock In \emph{Proceedings of ICLR}, 2022.
+\newblock \url{https://arxiv.org/abs/2106.09685}
+
+\bibitem[{Yuan et~al.}(2023)]{yuan2023rft}
+Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C.
+\newblock Scaling relationship on learning mathematical reasoning with large language models.
+\newblock \emph{arXiv preprint arXiv:2308.01825}, 2023.
+\newblock \url{https://arxiv.org/abs/2308.01825}
+
+\bibitem[{Jain et~al.}(2024)]{neftune}
+Jain, N., Chiang, P., Yeh, Y., Kirchenbauer, J., et~al.
+\newblock {NEFTune}: Noisy embeddings improve instruction finetuning.
+\newblock In \emph{Proceedings of ICLR}, 2024.
+\newblock \url{https://arxiv.org/abs/2310.05914}
+
+\bibitem[{ElevenLabs}(2025)]{elevenlabs}
+ElevenLabs.
+\newblock Text-to-speech v3 audio tags.
+\newblock ElevenLabs Developer Documentation, 2025.
+\newblock \url{https://elevenlabs.io/docs/api-reference/text-to-speech}
+
+\bibitem[{JiWER}(2024)]{jiwer}
+JiWER: Evaluate your speech recognition system.
+\newblock Python library for ASR evaluation metrics.
+\newblock \url{https://github.com/jitsi/jiwer}
+
+\bibitem[{}(2025a)]{grpo_speech}
+Advancing speech understanding in speech-aware language models with {GRPO}.
+\newblock \emph{arXiv preprint arXiv:2509.16990}, 2025.
+\newblock \url{https://arxiv.org/abs/2509.16990}
+
+\bibitem[{}(2025b)]{rl_asr_tts}
+Explore the reinforcement learning for {LLM}-based {ASR} and {TTS} system.
+\newblock \emph{arXiv preprint arXiv:2509.18569}, 2025.
+\newblock \url{https://arxiv.org/abs/2509.18569}
+
+\bibitem[{Mangrulkar et~al.}(2022)]{peft}
+Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B.
+\newblock {PEFT}: State-of-the-art parameter-efficient fine-tuning methods.
+\newblock HuggingFace, 2022.
+\newblock \url{https://github.com/huggingface/peft}
+
+\bibitem[{Morris et~al.}(2004)]{morris2004wer}
+Morris, A.C., Maier, V., and Green, P.
+\newblock From {WER} and {RIL} to {MER} and {WIL}: Improved evaluation measures for connected speech recognition.
+\newblock In \emph{Proceedings of INTERSPEECH}, 2004.
+
+\bibitem[{}(2025c)]{rl_audio_qa}
+Reinforcement learning outperforms supervised fine-tuning: A case study on audio question answering.
+\newblock \emph{arXiv preprint arXiv:2503.11197}, 2025.
+\newblock \url{https://arxiv.org/abs/2503.11197}
+
+\end{thebibliography}
+
+\end{document}

space/app.py

@@ -8,7 +8,7 @@ from transformers import VoxtralForConditionalGeneration, AutoProcessor
 from peft import PeftModel
 
 MODEL_ID = "mistralai/Voxtral-Mini-3B-2507"
-ADAPTER_ID = "YongkangZOU/evoxtral-lora"
+ADAPTER_ID = "YongkangZOU/evoxtral-rl"
 
 # Load model on CPU at startup, ZeroGPU moves to GPU on demand
 print("Loading model...")

training/scripts/rl_modal.py

@@ -0,0 +1,506 @@
+"""Evoxtral RL — Rejection sampling + SFT on best completions (RAFT).
+
+Follows the GRPO-for-ASR approach (arxiv:2509.01939) simplified for hackathon:
+1. Generate N completions per training sample (with sampling)
+2. Score each with rule-based reward (WER + Tag F1 + hallucination penalty)
+3. Keep best completion per sample
+4. SFT on the curated high-quality dataset (1 epoch, lower LR)
+
+Usage:
+    modal run scripts/rl_modal.py
+    modal run scripts/rl_modal.py --num-samples 4 --lr 5e-5
+"""
+
+import os
+import modal
+
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .apt_install("ffmpeg", "libsndfile1")
+    .pip_install(
+        "torch>=2.4.0",
+        "torchaudio>=2.4.0",
+        "transformers==4.56.0",
+        "datasets>=2.14.0",
+        "accelerate>=1.0.0",
+        "peft>=0.13.0",
+        "wandb>=0.18.0",
+        "jiwer>=3.0.0",
+        "librosa>=0.10.0",
+        "soundfile>=0.12.0",
+        "huggingface_hub",
+        "safetensors",
+        "sentencepiece",
+        "mistral-common",
+        "torchcodec",
+        gpu="A10G",
+    )
+    .env({
+        "HF_HUB_CACHE": "/cache/huggingface",
+    })
+)
+
+app = modal.App("evoxtral-rl", image=image)
+
+hf_cache = modal.Volume.from_name("evoxtral-hf-cache", create_if_missing=True)
+data_vol = modal.Volume.from_name("evoxtral-data", create_if_missing=True)
+output_vol = modal.Volume.from_name("evoxtral-output", create_if_missing=True)
+
+VOLUMES = {
+    "/cache/huggingface": hf_cache,
+    "/data": data_vol,
+    "/output": output_vol,
+}
+
+MODEL_ID = "mistralai/Voxtral-Mini-3B-2507"
+SFT_ADAPTER = "/output/evoxtral-lora"
+RL_OUTPUT = "/output/evoxtral-rl"
+
+
+@app.function(
+    gpu="A10G",
+    volumes=VOLUMES,
+    secrets=[
+        modal.Secret.from_name("wandb-secret"),
+        modal.Secret.from_name("huggingface-secret"),
+    ],
+    timeout=7200,
+    memory=32768,
+)
+def generate_and_score(num_samples: int = 4, temperature: float = 0.7):
+    """Step 1: Generate N completions per sample and score them."""
+    import torch
+    import json
+    import re
+    from pathlib import Path
+    from collections import Counter
+    from jiwer import wer as compute_wer
+    from datasets import load_from_disk, Audio
+    from transformers import VoxtralForConditionalGeneration, AutoProcessor
+    from peft import PeftModel
+
+    print(f"GPU: {torch.cuda.get_device_name(0)}")
+
+    # --- Reward helpers ---
+    def extract_tags(text):
+        return [m.group(1).lower() for m in re.finditer(r'\[([^\]]+)\]', text)]
+
+    def strip_tags(text):
+        text = re.sub(r'\[[^\]]+\]\s*', '', text)
+        text = re.sub(r'\b[A-Z]{2,}\b', lambda m: m.group(0).lower(), text)
+        return text.strip()
+
+    def compute_reward(prediction, reference):
+        """Rule-based reward: WER accuracy + Tag F1 - hallucination penalty."""
+        # WER component (accuracy = 1 - WER)
+        ref_plain = strip_tags(reference)
+        pred_plain = strip_tags(prediction)
+        if ref_plain.strip():
+            wer_score = compute_wer(ref_plain, pred_plain)
+            wer_accuracy = max(0.0, 1.0 - wer_score)
+        else:
+            wer_accuracy = 1.0
+
+        # Tag F1 component
+        pred_tags = Counter(extract_tags(prediction))
+        ref_tags = Counter(extract_tags(reference))
+        if not ref_tags and not pred_tags:
+            tag_f1 = 1.0
+            hall_rate = 0.0
+        elif not ref_tags or not pred_tags:
+            tag_f1 = 0.0
+            hall_rate = 1.0 if pred_tags else 0.0
+        else:
+            common = sum((pred_tags & ref_tags).values())
+            prec = common / sum(pred_tags.values())
+            rec = common / sum(ref_tags.values())
+            tag_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
+            # Hallucination rate
+            ref_set = set(ref_tags.keys())
+            hallucinated = sum(v for k, v in pred_tags.items() if k not in ref_set)
+            hall_rate = hallucinated / sum(pred_tags.values())
+
+        # Combined reward
+        reward = 0.4 * wer_accuracy + 0.4 * tag_f1 + 0.2 * (1.0 - hall_rate)
+        return reward, {"wer_accuracy": wer_accuracy, "tag_f1": tag_f1, "hall_rate": hall_rate}
+
+    # --- Load model ---
+    print("Loading SFT model...")
+    processor = AutoProcessor.from_pretrained(MODEL_ID)
+    base_model = VoxtralForConditionalGeneration.from_pretrained(
+        MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto",
+    )
+    model = PeftModel.from_pretrained(base_model, SFT_ADAPTER)
+    model.eval()
+    print(f"Model loaded on {model.device}")
+
+    # --- Load training data ---
+    ds = load_from_disk("/data/processed")
+    ds = ds.cast_column("audio", Audio(sampling_rate=16000))
+    train_ds = ds["train"]
+
+    print(f"Generating {num_samples} completions per sample for {len(train_ds)} training examples...")
+    import time
+    start_time = time.time()
+
+    # Resume from checkpoint if exists
+    checkpoint_path = "/output/rl_curated_checkpoint.json"
+    if Path(checkpoint_path).exists():
+        with open(checkpoint_path) as f:
+            curated_data = json.load(f)
+        start_idx = len(curated_data)
+        total_reward = sum(d["reward"] for d in curated_data)
+        print(f"Resuming from checkpoint: {start_idx} samples already done")
+    else:
+        curated_data = []
+        total_reward = 0.0
+        start_idx = 0
+
+    for i in range(start_idx, len(train_ds)):
+        row = train_ds[i]
+        reference = row["tagged_text"]
+        audio_array = row["audio"]["array"]
+
+        # Build inputs
+        inputs = processor.apply_transcription_request(
+            language="en",
+            audio=[audio_array],
+            format=["WAV"],
+            model_id=MODEL_ID,
+            return_tensors="pt",
+        )
+        inputs = {k: v.to(model.device) for k, v in inputs.items()}
+
+        # Generate all N completions in one call with num_return_sequences
+        with torch.no_grad():
+            output_ids = model.generate(
+                **inputs,
+                max_new_tokens=512,
+                do_sample=True,
+                temperature=temperature,
+                top_p=0.9,
+                num_return_sequences=num_samples,
+            )
+
+        input_len = inputs["input_ids"].shape[1]
+
+        # Score each completion, keep best
+        best_reward = -1.0
+        best_prediction = None
+        best_details = None
+
+        for s in range(num_samples):
+            prediction = processor.tokenizer.decode(
+                output_ids[s][input_len:], skip_special_tokens=True
+            )
+            reward, details = compute_reward(prediction, reference)
+            if reward > best_reward:
+                best_reward = reward
+                best_prediction = prediction
+                best_details = details
+
+        curated_data.append({
+            "audio_idx": i,
+            "reference": reference,
+            "best_prediction": best_prediction,
+            "reward": best_reward,
+            **best_details,
+        })
+        total_reward += best_reward
+
+        if i < 5 or i % 50 == 0:
+            elapsed = time.time() - start_time
+            done = i - start_idx + 1
+            rate = done / elapsed if elapsed > 0 else 0
+            eta = (len(train_ds) - i - 1) / rate if rate > 0 else 0
+            print(f"  [{i}/{len(train_ds)}] reward={best_reward:.3f} "
+                  f"wer_acc={best_details['wer_accuracy']:.3f} "
+                  f"tag_f1={best_details['tag_f1']:.3f} "
+                  f"hall={best_details['hall_rate']:.3f} "
+                  f"({rate:.1f} samples/s, ETA {eta/60:.0f}min)")
+            if i < 3:
+                print(f"    ref: {reference[:80]}...")
+                print(f"    best: {best_prediction[:80]}...")
+
+        # Save checkpoint every 50 samples
+        if (i + 1) % 50 == 0:
+            with open(checkpoint_path, "w") as f:
+                json.dump(curated_data, f)
+            output_vol.commit()
+            print(f"  [checkpoint saved: {len(curated_data)} samples]")
+
+    avg_reward = total_reward / len(curated_data)
+    print(f"\nGeneration complete! Avg reward: {avg_reward:.4f}")
+    print(f"Curated {len(curated_data)} samples")
+
+    # Save curated data
+    output_path = "/output/rl_curated_data.json"
+    with open(output_path, "w") as f:
+        json.dump(curated_data, f)
+    output_vol.commit()
+    print(f"Saved curated data to {output_path}")
+
+    return {"avg_reward": avg_reward, "num_samples": len(curated_data)}
+
+
+@app.function(
+    gpu="A10G",
+    volumes=VOLUMES,
+    secrets=[
+        modal.Secret.from_name("wandb-secret"),
+        modal.Secret.from_name("huggingface-secret"),
+    ],
+    timeout=7200,
+    memory=32768,
+)
+def rl_finetune(learning_rate: float = 5e-5, num_epochs: int = 1, push_to_hub: bool = True):
+    """Step 2: SFT on curated best completions (RAFT)."""
+    import torch
+    import wandb
+    import json
+    from pathlib import Path
+    from datasets import Dataset, Audio, load_from_disk
+    from transformers import (
+        VoxtralForConditionalGeneration,
+        AutoProcessor,
+        TrainingArguments,
+        Trainer,
+    )
+    from peft import PeftModel
+
+    print(f"GPU: {torch.cuda.get_device_name(0)}")
+
+    # --- Load curated data ---
+    with open("/output/rl_curated_data.json") as f:
+        curated_data = json.load(f)
+    print(f"Loaded {len(curated_data)} curated samples")
+
+    # Filter out low-reward samples (bottom 10%)
+    rewards = [d["reward"] for d in curated_data]
+    threshold = sorted(rewards)[len(rewards) // 10]
+    curated_data = [d for d in curated_data if d["reward"] > threshold]
+    print(f"After filtering (reward > {threshold:.3f}): {len(curated_data)} samples")
+
+    # --- Load original audio dataset ---
+    ds = load_from_disk("/data/processed")
+    ds = ds.cast_column("audio", Audio(sampling_rate=16000))
+    train_ds = ds["train"]
+
+    # Build RL training dataset: use original audio + best prediction as target
+    rl_examples = []
+    for d in curated_data:
+        idx = d["audio_idx"]
+        row = train_ds[idx]
+        rl_examples.append({
+            "audio": row["audio"],
+            "tagged_text": d["best_prediction"],  # RL target = best sampled completion
+        })
+
+    rl_dataset = Dataset.from_list(rl_examples)
+    rl_dataset = rl_dataset.cast_column("audio", Audio(sampling_rate=16000))
+    print(f"RL training dataset: {len(rl_dataset)} samples")
+
+    # --- W&B ---
+    run = wandb.init(
+        project="evoxtral",
+        name=f"rl-raft-lr{learning_rate}-ep{num_epochs}",
+        config={
+            "method": "RAFT (rejection sampling + SFT)",
+            "base_adapter": "evoxtral-lora (SFT)",
+            "learning_rate": learning_rate,
+            "epochs": num_epochs,
+            "num_curated": len(rl_dataset),
+            "reward_threshold": threshold,
+        },
+        tags=["evoxtral", "rl", "raft", "rejection-sampling"],
+    )
+
+    avg_reward = sum(d["reward"] for d in curated_data) / len(curated_data)
+    wandb.log({"rl/curated_samples": len(curated_data), "rl/avg_reward": avg_reward})
+
+    # --- Load SFT model ---
+    print("Loading SFT model for RL finetuning...")
+    processor = AutoProcessor.from_pretrained(MODEL_ID)
+    base_model = VoxtralForConditionalGeneration.from_pretrained(
+        MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto",
+    )
+    model = PeftModel.from_pretrained(base_model, SFT_ADAPTER)
+    # Unfreeze LoRA for continued training
+    for name, param in model.named_parameters():
+        if "lora" in name.lower():
+            param.requires_grad = True
+    model.print_trainable_parameters()
+
+    # --- Data Collator (same as SFT) ---
+    class VoxtralDataCollator:
+        def __init__(self, processor, model_id, max_text_len=512):
+            self.processor = processor
+            self.model_id = model_id
+            self.max_text_len = max_text_len
+            self.pad_id = processor.tokenizer.pad_token_id or processor.tokenizer.eos_token_id
+
+        def __call__(self, examples):
+            texts = [ex["tagged_text"] for ex in examples]
+            audios = [ex["audio"]["array"] for ex in examples]
+
+            prompt = self.processor.apply_transcription_request(
+                language="en",
+                model_id=self.model_id,
+                audio=audios,
+                format=["WAV"] * len(audios),
+                return_tensors="pt",
+            )
+            passthrough = {k: v for k, v in prompt.items()
+                          if k not in ("input_ids", "attention_mask")}
+
+            prompt_ids = prompt["input_ids"]
+            prompt_attn = prompt["attention_mask"]
+            B = prompt_ids.size(0)
+
+            tok = self.processor.tokenizer
+            text_tok = tok(
+                texts,
+                add_special_tokens=False,
+                padding=False,
+                truncation=True,
+                max_length=self.max_text_len,
+                return_tensors=None,
+            )
+            text_ids_list = text_tok["input_ids"]
+
+            input_ids, attention_mask, labels = [], [], []
+            for i in range(B):
+                p_ids = prompt_ids[i].tolist()
+                p_att = prompt_attn[i].tolist()
+                t_ids = text_ids_list[i]
+
+                ids = p_ids + t_ids + [tok.eos_token_id]
+                attn = p_att + [1] * (len(t_ids) + 1)
+                lab = [-100] * len(p_ids) + t_ids + [tok.eos_token_id]
+
+                input_ids.append(ids)
+                attention_mask.append(attn)
+                labels.append(lab)
+
+            max_len = max(len(x) for x in input_ids)
+
+            def pad_to(seq, fill, L):
+                return seq + [fill] * (L - len(seq))
+
+            input_ids = [pad_to(x, self.pad_id, max_len) for x in input_ids]
+            attention_mask = [pad_to(x, 0, max_len) for x in attention_mask]
+            labels = [pad_to(x, -100, max_len) for x in labels]
+
+            batch = {
+                "input_ids": torch.tensor(input_ids, dtype=torch.long),
+                "attention_mask": torch.tensor(attention_mask, dtype=torch.long),
+                "labels": torch.tensor(labels, dtype=torch.long),
+            }
+            for k, v in passthrough.items():
+                batch[k] = v
+            return batch
+
+    collator = VoxtralDataCollator(processor, MODEL_ID)
+
+    # --- Training args (lower LR, 1 epoch) ---
+    training_args = TrainingArguments(
+        output_dir=RL_OUTPUT,
+        num_train_epochs=num_epochs,
+        per_device_train_batch_size=2,
+        gradient_accumulation_steps=8,
+        learning_rate=learning_rate,
+        lr_scheduler_type="cosine",
+        warmup_steps=20,
+        weight_decay=0.01,
+        max_grad_norm=1.0,
+        bf16=True,
+        gradient_checkpointing=True,
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        logging_steps=5,
+        save_strategy="epoch",
+        save_total_limit=2,
+        report_to="wandb",
+        remove_unused_columns=False,
+        dataloader_pin_memory=True,
+        dataloader_num_workers=4,
+    )
+
+    trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=rl_dataset,
+        data_collator=collator,
+    )
+
+    print("Starting RL finetuning...")
+    train_result = trainer.train()
+
+    wandb.log({
+        "rl/final_loss": train_result.metrics.get("train_loss", 0),
+        "rl/runtime_seconds": train_result.metrics.get("train_runtime", 0),
+    })
+
+    # Save adapter
+    print(f"Saving RL adapter to {RL_OUTPUT}")
+    trainer.save_model(RL_OUTPUT)
+    processor.save_pretrained(RL_OUTPUT)
+
+    # Log as W&B artifact
+    artifact = wandb.Artifact(
+        "evoxtral-rl-adapter",
+        type="model",
+        metadata={"method": "RAFT", "base": "evoxtral-lora"},
+    )
+    artifact.add_dir(RL_OUTPUT)
+    run.log_artifact(artifact)
+
+    # Push to Hub
+    if push_to_hub:
+        from huggingface_hub import HfApi
+        HUB_ID = "YongkangZOU/evoxtral-rl"
+        print(f"Pushing to HuggingFace Hub: {HUB_ID}")
+        try:
+            api = HfApi(token=os.environ.get("HF_TOKEN"))
+            api.create_repo(HUB_ID, repo_type="model", exist_ok=True)
+            api.upload_folder(
+                folder_path=RL_OUTPUT,
+                repo_id=HUB_ID,
+                repo_type="model",
+                commit_message=f"RL adapter (RAFT): lr={learning_rate}, ep={num_epochs}",
+            )
+            print(f"Pushed to {HUB_ID}")
+        except Exception as e:
+            print(f"Hub push failed: {e}")
+
+    output_vol.commit()
+    wandb.finish()
+    print("RL finetuning complete!")
+    return train_result.metrics
+
+
+@app.local_entrypoint()
+def main(
+    num_samples: int = 4,
+    temperature: float = 0.7,
+    lr: float = 5e-5,
+    epochs: int = 1,
+    push_to_hub: bool = True,
+    finetune_only: bool = False,
+):
+    if not finetune_only:
+        print("Step 1: Generating and scoring completions...")
+        gen_results = generate_and_score.remote(
+            num_samples=num_samples,
+            temperature=temperature,
+        )
+        print(f"Generation results: {gen_results}")
+
+    print("\nStep 2: RL finetuning on curated data...")
+    ft_results = rl_finetune.remote(
+        learning_rate=lr,
+        num_epochs=epochs,
+        push_to_hub=push_to_hub,
+    )
+    print(f"RL finetune results: {ft_results}")
+    print("\nDone! Run eval with: modal run scripts/train_modal.py --eval-only")
+    print("(Update adapter_path in evaluate() to /output/evoxtral-rl)")

training/scripts/serve_modal.py

@@ -4,14 +4,18 @@ Swagger UI: https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/d
 
 Usage:
     # Deploy:
-    modal deploy scripts/serve_modal.py
+    modal deploy training/scripts/serve_modal.py
 
     # Test locally:
-    modal serve scripts/serve_modal.py
+    modal serve training/scripts/serve_modal.py
 
-    # Call the API:
+    # Call the API (JSON response):
     curl -X POST https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/transcribe \
         -F "file=@audio.wav"
+
+    # Call the streaming API (Server-Sent Events):
+    curl -N -X POST https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run/transcribe/stream \
+        -F "file=@audio.wav"
 """
 
 import modal
@@ -34,6 +38,7 @@ image = (
         "sentencepiece",
         "fastapi",
         "python-multipart",
+        "sse-starlette",
         gpu="A10G",
     )
     .env({"HF_HUB_CACHE": "/cache/huggingface"})
@@ -43,7 +48,50 @@ app = modal.App("evoxtral-api", image=image)
 hf_cache = modal.Volume.from_name("evoxtral-hf-cache", create_if_missing=True)
 
 MODEL_ID = "mistralai/Voxtral-Mini-3B-2507"
-ADAPTER_ID = "YongkangZOU/evoxtral-lora"
+ADAPTER_ID = "YongkangZOU/evoxtral-rl"
+
+
+def _decode_audio(audio_bytes):
+    """Decode audio bytes to float32 numpy array at 16kHz.
+
+    Uses librosa (backed by ffmpeg) so all common formats work:
+    WAV, FLAC, MP3, MP4, M4A, WebM, OGG, etc.
+    """
+    import numpy as np
+    import librosa
+    import tempfile
+    import os
+
+    # librosa needs a file path (uses ffmpeg under the hood for non-WAV)
+    with tempfile.NamedTemporaryFile(suffix=".audio", delete=False) as f:
+        f.write(audio_bytes)
+        tmp_path = f.name
+    try:
+        audio_array, sr = librosa.load(tmp_path, sr=16000, mono=True)
+    finally:
+        os.unlink(tmp_path)
+
+    return audio_array.astype(np.float32)
+
+
+def _prepare_inputs(processor, audio_array, language, device):
+    """Prepare model inputs from audio array."""
+    import torch
+
+    inputs = processor.apply_transcription_request(
+        language=language,
+        audio=[audio_array],
+        format=["WAV"],
+        model_id=MODEL_ID,
+        return_tensors="pt",
+    )
+    inputs = {
+        k: v.to(device, dtype=torch.bfloat16)
+        if v.dtype in (torch.float32, torch.float16, torch.bfloat16)
+        else v.to(device)
+        for k, v in inputs.items()
+    }
+    return inputs
 
 
 @app.cls(
@@ -73,23 +121,30 @@ class EvoxtralModel:
     @modal.asgi_app()
     def web(self):
         import torch
+        import json
+        import asyncio
         import numpy as np
+        from threading import Thread
         from fastapi import FastAPI, UploadFile, File, Form, HTTPException
-        from fastapi.responses import JSONResponse
+        from fastapi.responses import JSONResponse, StreamingResponse
+        from transformers import TextIteratorStreamer
 
         web_app = FastAPI(
             title="Evoxtral API",
             description=(
                 "Expressive tagged transcription powered by Voxtral-Mini-3B + LoRA. "
                 "Upload audio and get transcriptions with inline expressive tags like "
-                "[sighs], [laughs], [whispers], etc."
+                "[sighs], [laughs], [whispers], etc.\n\n"
+                "**Endpoints:**\n"
+                "- `POST /transcribe` — Returns full transcription as JSON\n"
+                "- `POST /transcribe/stream` — Streams tokens via Server-Sent Events (SSE)"
             ),
-            version="1.0.0",
+            version="2.0.0",
         )
 
         @web_app.get("/health", summary="Health check")
         async def health():
-            return {"status": "ok", "model": "evoxtral-lora", "base": MODEL_ID}
+            return {"status": "ok", "model": "evoxtral-rl", "base": MODEL_ID}
 
         @web_app.post(
             "/transcribe",
@@ -100,39 +155,16 @@ class EvoxtralModel:
             file: UploadFile = File(..., description="Audio file (WAV, MP3, FLAC, etc.)"),
             language: str = Form("en", description="Language code (e.g. 'en', 'fr', 'es')"),
         ):
-            import librosa
-            import soundfile as sf
-            import io
-
             audio_bytes = await file.read()
             if not audio_bytes:
                 raise HTTPException(status_code=400, detail="Empty audio file")
 
-            # Decode audio
             try:
-                audio_array, sr = sf.read(io.BytesIO(audio_bytes))
-                audio_array = audio_array.astype(np.float32)
-                if audio_array.ndim > 1:
-                    audio_array = audio_array.mean(axis=1)
-                if sr != 16000:
-                    audio_array = librosa.resample(audio_array, orig_sr=sr, target_sr=16000)
+                audio_array = _decode_audio(audio_bytes)
             except Exception as e:
                 raise HTTPException(status_code=400, detail=f"Failed to decode audio: {e}")
 
-            # Run inference
-            inputs = self.processor.apply_transcription_request(
-                language=language,
-                audio=[audio_array],
-                format=["WAV"],
-                model_id=MODEL_ID,
-                return_tensors="pt",
-            )
-            inputs = {
-                k: v.to(self.model.device, dtype=torch.bfloat16)
-                if v.dtype in (torch.float32, torch.float16, torch.bfloat16)
-                else v.to(self.model.device)
-                for k, v in inputs.items()
-            }
+            inputs = _prepare_inputs(self.processor, audio_array, language, self.model.device)
 
             with torch.no_grad():
                 output_ids = self.model.generate(**inputs, max_new_tokens=512, do_sample=False)
@@ -145,7 +177,61 @@ class EvoxtralModel:
             return {
                 "transcription": transcription,
                 "language": language,
-                "model": "evoxtral-lora",
+                "model": "evoxtral-rl",
             }
 
+        @web_app.post(
+            "/transcribe/stream",
+            summary="Transcribe audio with streaming (SSE)",
+            response_description="Server-Sent Events stream of transcription tokens",
+        )
+        async def transcribe_stream(
+            file: UploadFile = File(..., description="Audio file (WAV, MP3, FLAC, etc.)"),
+            language: str = Form("en", description="Language code (e.g. 'en', 'fr', 'es')"),
+        ):
+            audio_bytes = await file.read()
+            if not audio_bytes:
+                raise HTTPException(status_code=400, detail="Empty audio file")
+
+            try:
+                audio_array = _decode_audio(audio_bytes)
+            except Exception as e:
+                raise HTTPException(status_code=400, detail=f"Failed to decode audio: {e}")
+
+            inputs = _prepare_inputs(self.processor, audio_array, language, self.model.device)
+
+            streamer = TextIteratorStreamer(
+                self.processor.tokenizer,
+                skip_prompt=True,
+                skip_special_tokens=True,
+            )
+
+            generate_kwargs = dict(
+                **inputs,
+                max_new_tokens=512,
+                do_sample=False,
+                streamer=streamer,
+            )
+
+            thread = Thread(target=lambda: self.model.generate(**generate_kwargs))
+            thread.start()
+
+            async def event_generator():
+                full_text = ""
+                for token_text in streamer:
+                    if token_text:
+                        full_text += token_text
+                        yield f"data: {json.dumps({'token': token_text})}\n\n"
+                yield f"data: {json.dumps({'done': True, 'transcription': full_text, 'language': language, 'model': 'evoxtral-rl'})}\n\n"
+
+            return StreamingResponse(
+                event_generator(),
+                media_type="text/event-stream",
+                headers={
+                    "Cache-Control": "no-cache",
+                    "Connection": "keep-alive",
+                    "X-Accel-Buffering": "no",
+                },
+            )
+
         return web_app

training/scripts/train_modal.py

@@ -361,14 +361,14 @@ def train(
     timeout=3600,
     memory=32768,
 )
-def evaluate(adapter_path: str = "/output/evoxtral-lora"):
+def evaluate(adapter_path: str = "/output/evoxtral-lora", eval_name: str = "finetuned"):
     """Run Evoxtral-Bench evaluation (base vs finetuned)."""
     import torch
     import wandb
     import weave
     import json
     from pathlib import Path
-    from jiwer import wer as compute_wer
+    from jiwer import wer as compute_wer, cer as compute_cer_jiwer
     from datasets import load_from_disk, Audio
     from transformers import VoxtralForConditionalGeneration, AutoProcessor
     from peft import PeftModel
@@ -403,9 +403,24 @@ def evaluate(adapter_path: str = "/output/evoxtral-lora"):
     ds = ds.cast_column("audio", Audio(sampling_rate=16000))
     test_ds = ds["test"]
 
+    def compute_cer(ref, hyp):
+        """Character Error Rate."""
+        if not ref.strip():
+            return 0.0
+        return compute_cer_jiwer(ref, hyp)
+
     def run_model_eval(model, processor, model_name):
         model.eval()
-        results = {"wer": [], "tag_f1": []}
+        results = {
+            "wer": [], "cer": [],
+            "tag_f1": [], "tag_precision": [], "tag_recall": [],
+            "tag_hallucination_rate": [],
+            "emphasis_f1": [],
+        }
+        per_tag_tp = Counter()  # true positives per tag type
+        per_tag_fp = Counter()  # false positives (predicted but not in ref)
+        per_tag_fn = Counter()  # false negatives (in ref but not predicted)
+        all_predictions = []
 
         for i in range(min(len(test_ds), 50)):  # eval on up to 50 samples
             row = test_ds[i]
@@ -428,22 +443,108 @@ def evaluate(adapter_path: str = "/output/evoxtral-lora"):
             input_len = inputs["input_ids"].shape[1]
             prediction = processor.tokenizer.decode(output_ids[0][input_len:], skip_special_tokens=True)
 
-            # Compute metrics
+            # --- Text metrics ---
             ref_plain = strip_tags(tagged_text)
             pred_plain = strip_tags(prediction)
             if ref_plain.strip():
                 results["wer"].append(compute_wer(ref_plain, pred_plain))
-            results["tag_f1"].append(compute_tag_f1(prediction, tagged_text))
+                results["cer"].append(compute_cer(ref_plain, pred_plain))
+
+            # --- Tag metrics (precision, recall, F1) ---
+            pred_tags = Counter(extract_tags(prediction))
+            ref_tags = Counter(extract_tags(tagged_text))
+
+            common = pred_tags & ref_tags
+            tp = sum(common.values())
+            fp = sum(pred_tags.values()) - tp
+            fn = sum(ref_tags.values()) - tp
+
+            prec = tp / (tp + fp) if (tp + fp) > 0 else (1.0 if not ref_tags else 0.0)
+            rec = tp / (tp + fn) if (tp + fn) > 0 else (1.0 if not pred_tags else 0.0)
+            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else (1.0 if not ref_tags and not pred_tags else 0.0)
+
+            results["tag_precision"].append(prec)
+            results["tag_recall"].append(rec)
+            results["tag_f1"].append(f1)
+
+            # Tag hallucination rate
+            ref_tag_set = set(ref_tags.keys())
+            hallucinated = sum(v for k, v in pred_tags.items() if k not in ref_tag_set)
+            hall_rate = hallucinated / sum(pred_tags.values()) if pred_tags else 0.0
+            results["tag_hallucination_rate"].append(hall_rate)
+
+            # Per-tag breakdown
+            for tag in set(list(pred_tags.keys()) + list(ref_tags.keys())):
+                matched = min(pred_tags.get(tag, 0), ref_tags.get(tag, 0))
+                per_tag_tp[tag] += matched
+                per_tag_fp[tag] += max(0, pred_tags.get(tag, 0) - matched)
+                per_tag_fn[tag] += max(0, ref_tags.get(tag, 0) - matched)
+
+            # Emphasis F1
+            pred_emph = Counter([m.group(0).lower() for m in re.finditer(r'\b[A-Z]{2,}\b', prediction)])
+            ref_emph = Counter([m.group(0).lower() for m in re.finditer(r'\b[A-Z]{2,}\b', tagged_text)])
+            emph_common = sum((pred_emph & ref_emph).values())
+            emph_total_p = sum(pred_emph.values())
+            emph_total_r = sum(ref_emph.values())
+            if emph_total_p == 0 and emph_total_r == 0:
+                emph_f1 = 1.0
+            elif emph_total_p == 0 or emph_total_r == 0:
+                emph_f1 = 0.0
+            else:
+                ep = emph_common / emph_total_p
+                er = emph_common / emph_total_r
+                emph_f1 = 2 * ep * er / (ep + er) if (ep + er) > 0 else 0.0
+            results["emphasis_f1"].append(emph_f1)
+
+            # Store prediction for W&B table
+            all_predictions.append({
+                "sample_idx": i,
+                "reference": tagged_text,
+                "prediction": prediction,
+                "wer": results["wer"][-1] if results["wer"] else None,
+                "tag_f1": f1,
+                "tag_hallucination_rate": hall_rate,
+            })
 
             if i < 5:
                 print(f"\n[{model_name} Sample {i}]")
                 print(f"  Reference: {tagged_text[:100]}...")
                 print(f"  Predicted: {prediction[:100]}...")
 
-        avg_wer = sum(results["wer"]) / max(len(results["wer"]), 1)
-        avg_tag_f1 = sum(results["tag_f1"]) / max(len(results["tag_f1"]), 1)
-        print(f"\n{model_name} Results: WER={avg_wer:.4f}, Tag F1={avg_tag_f1:.4f}")
-        return {"avg_wer": avg_wer, "avg_tag_f1": avg_tag_f1}
+        # Compute averages
+        def avg(lst):
+            return sum(lst) / max(len(lst), 1)
+
+        avg_metrics = {f"avg_{k}": avg(v) for k, v in results.items()}
+
+        # Per-tag F1 breakdown
+        per_tag_f1 = {}
+        for tag in sorted(per_tag_tp.keys() | per_tag_fp.keys() | per_tag_fn.keys()):
+            tp = per_tag_tp[tag]
+            fp = per_tag_fp[tag]
+            fn = per_tag_fn[tag]
+            p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
+            r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
+            f = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
+            per_tag_f1[tag] = {"precision": round(p, 3), "recall": round(r, 3), "f1": round(f, 3),
+                               "support": tp + fn}
+
+        avg_metrics["per_tag_f1"] = per_tag_f1
+        avg_metrics["predictions"] = all_predictions
+
+        print(f"\n{model_name} Results:")
+        print(f"  WER:                  {avg_metrics['avg_wer']:.4f}")
+        print(f"  CER:                  {avg_metrics['avg_cer']:.4f}")
+        print(f"  Tag F1:               {avg_metrics['avg_tag_f1']:.4f}")
+        print(f"  Tag Precision:        {avg_metrics['avg_tag_precision']:.4f}")
+        print(f"  Tag Recall:           {avg_metrics['avg_tag_recall']:.4f}")
+        print(f"  Tag Hallucination:    {avg_metrics['avg_tag_hallucination_rate']:.4f}")
+        print(f"  Emphasis F1:          {avg_metrics['avg_emphasis_f1']:.4f}")
+        print(f"\n  Per-tag breakdown:")
+        for tag, m in sorted(per_tag_f1.items(), key=lambda x: -x[1]["support"]):
+            print(f"    [{tag}]: F1={m['f1']:.3f} P={m['precision']:.3f} R={m['recall']:.3f} (n={m['support']})")
+
+        return avg_metrics
 
     # ── Evaluate base model ──
     wandb.init(project="evoxtral", name="eval-base", job_type="evaluation", tags=["eval", "base"])
@@ -453,7 +554,26 @@ def evaluate(adapter_path: str = "/output/evoxtral-lora"):
         MODEL_ID, dtype=torch.bfloat16, device_map="auto",
     )
     base_results = run_model_eval(base_model, processor, "BASE")
-    wandb.log({"eval/wer": base_results["avg_wer"], "eval/tag_f1": base_results["avg_tag_f1"]})
+    wandb.log({
+        "eval/wer": base_results["avg_wer"],
+        "eval/cer": base_results["avg_cer"],
+        "eval/tag_f1": base_results["avg_tag_f1"],
+        "eval/tag_precision": base_results["avg_tag_precision"],
+        "eval/tag_recall": base_results["avg_tag_recall"],
+        "eval/tag_hallucination_rate": base_results["avg_tag_hallucination_rate"],
+        "eval/emphasis_f1": base_results["avg_emphasis_f1"],
+    })
+    # Log per-tag breakdown as table
+    tag_table = wandb.Table(columns=["tag", "f1", "precision", "recall", "support"])
+    for tag, m in base_results["per_tag_f1"].items():
+        tag_table.add_data(tag, m["f1"], m["precision"], m["recall"], m["support"])
+    wandb.log({"eval/per_tag_breakdown": tag_table})
+    # Log predictions table
+    pred_table = wandb.Table(columns=["idx", "reference", "prediction", "wer", "tag_f1", "hallucination_rate"])
+    for p in base_results["predictions"]:
+        pred_table.add_data(p["sample_idx"], p["reference"], p["prediction"],
+                           p["wer"], p["tag_f1"], p["tag_hallucination_rate"])
+    wandb.log({"eval/predictions": pred_table})
     wandb.finish()
 
     # Free base model memory
@@ -462,21 +582,45 @@ def evaluate(adapter_path: str = "/output/evoxtral-lora"):
 
     # ── Evaluate finetuned model ──
     if Path(adapter_path).exists():
-        wandb.init(project="evoxtral", name="eval-finetuned", job_type="evaluation", tags=["eval", "finetuned"])
+        wandb.init(project="evoxtral", name=f"eval-{eval_name}", job_type="evaluation", tags=["eval", eval_name])
         print("Loading finetuned model...")
         base_model = VoxtralForConditionalGeneration.from_pretrained(
             MODEL_ID, dtype=torch.bfloat16, device_map="auto",
         )
         ft_model = PeftModel.from_pretrained(base_model, adapter_path)
         ft_results = run_model_eval(ft_model, processor, "FINETUNED")
-        wandb.log({"eval/wer": ft_results["avg_wer"], "eval/tag_f1": ft_results["avg_tag_f1"]})
+        wandb.log({
+            "eval/wer": ft_results["avg_wer"],
+            "eval/cer": ft_results["avg_cer"],
+            "eval/tag_f1": ft_results["avg_tag_f1"],
+            "eval/tag_precision": ft_results["avg_tag_precision"],
+            "eval/tag_recall": ft_results["avg_tag_recall"],
+            "eval/tag_hallucination_rate": ft_results["avg_tag_hallucination_rate"],
+            "eval/emphasis_f1": ft_results["avg_emphasis_f1"],
+        })
+        # Log per-tag breakdown as table
+        tag_table = wandb.Table(columns=["tag", "f1", "precision", "recall", "support"])
+        for tag, m in ft_results["per_tag_f1"].items():
+            tag_table.add_data(tag, m["f1"], m["precision"], m["recall"], m["support"])
+        wandb.log({"eval/per_tag_breakdown": tag_table})
+        # Log predictions table
+        pred_table = wandb.Table(columns=["idx", "reference", "prediction", "wer", "tag_f1", "hallucination_rate"])
+        for p in ft_results["predictions"]:
+            pred_table.add_data(p["sample_idx"], p["reference"], p["prediction"],
+                               p["wer"], p["tag_f1"], p["tag_hallucination_rate"])
+        wandb.log({"eval/predictions": pred_table})
         wandb.finish()
 
-        print(f"\n{'='*50}")
+        print(f"\n{'='*60}")
         print(f"COMPARISON: Base vs Finetuned")
-        print(f"{'='*50}")
-        print(f"WER:    {base_results['avg_wer']:.4f} → {ft_results['avg_wer']:.4f}")
-        print(f"Tag F1: {base_results['avg_tag_f1']:.4f} → {ft_results['avg_tag_f1']:.4f}")
+        print(f"{'='*60}")
+        print(f"WER:                {base_results['avg_wer']:.4f} → {ft_results['avg_wer']:.4f}")
+        print(f"CER:                {base_results['avg_cer']:.4f} → {ft_results['avg_cer']:.4f}")
+        print(f"Tag F1:             {base_results['avg_tag_f1']:.4f} → {ft_results['avg_tag_f1']:.4f}")
+        print(f"Tag Precision:      {base_results['avg_tag_precision']:.4f} → {ft_results['avg_tag_precision']:.4f}")
+        print(f"Tag Recall:         {base_results['avg_tag_recall']:.4f} → {ft_results['avg_tag_recall']:.4f}")
+        print(f"Tag Hallucination:  {base_results['avg_tag_hallucination_rate']:.4f} → {ft_results['avg_tag_hallucination_rate']:.4f}")
+        print(f"Emphasis F1:        {base_results['avg_emphasis_f1']:.4f} → {ft_results['avg_emphasis_f1']:.4f}")
     else:
         print(f"No adapter found at {adapter_path}, skipping finetuned eval")
         ft_results = None
@@ -494,8 +638,12 @@ def main(
     batch_size: int = 2,
     push_to_hub: bool = True,
     eval_only: bool = False,
+    eval_rl: bool = False,
 ):
-    if eval_only:
+    if eval_rl:
+        results = evaluate.remote(adapter_path="/output/evoxtral-rl", eval_name="rl")
+        print(results)
+    elif eval_only:
         results = evaluate.remote()
         print(results)
     else:

web/src/app/api/speech-to-text/route.ts

@@ -20,12 +20,17 @@ export async function POST(req: Request) {
       );
     }
 
-    const MODEL_URL = process.env.MODEL_URL || "http://127.0.0.1:8000";
+    const MODAL_API_URL =
+      process.env.MODAL_API_URL ??
+      "https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run";
 
-    // Forward the formData to the Python backend
-    const response = await fetch(`${MODEL_URL}/transcribe`, {
+    // Forward the formData to the Modal API (non-streaming)
+    const upstream = new FormData();
+    const originalName = (audioFile as File).name || "audio.wav";
+    upstream.append("file", audioFile, originalName);
+    const response = await fetch(`${MODAL_API_URL}/transcribe`, {
       method: "POST",
-      body: formData,
+      body: upstream,
       // Signal timeout after 5 minutes
       signal: AbortSignal.timeout(5 * 60 * 1000),
     });

web/src/app/api/transcribe-stream/route.ts

@@ -0,0 +1,72 @@
+/**
+ * SSE proxy: forwards audio to Modal /transcribe/stream and pipes tokens back.
+ * Browser cannot call Modal directly (no CORS), so this Next.js route acts as relay.
+ */
+
+const MODAL_API_URL =
+  process.env.MODAL_API_URL ??
+  "https://yongkang-zou1999--evoxtral-api-evoxtralmodel-web.modal.run";
+
+export async function POST(req: Request) {
+  const formData = await req.formData();
+  const audioFile = formData.get("audio") as Blob | null;
+
+  if (!audioFile) {
+    return new Response(JSON.stringify({ error: "Audio file is required" }), {
+      status: 400,
+      headers: { "Content-Type": "application/json" },
+    });
+  }
+
+  const MAX_UPLOAD_BYTES = 100 * 1024 * 1024;
+  if (audioFile.size > MAX_UPLOAD_BYTES) {
+    return new Response(
+      JSON.stringify({ error: `File exceeds ${MAX_UPLOAD_BYTES / 1024 / 1024}MB limit` }),
+      { status: 413, headers: { "Content-Type": "application/json" } }
+    );
+  }
+
+  // Forward to Modal streaming endpoint, preserving original filename for format detection
+  const upstream = new FormData();
+  const originalName = (audioFile as File).name || "audio.wav";
+  upstream.append("file", audioFile, originalName);
+
+  const language = formData.get("language") as string | null;
+  if (language) upstream.append("language", language);
+
+  try {
+    const res = await fetch(`${MODAL_API_URL}/transcribe/stream`, {
+      method: "POST",
+      body: upstream,
+      signal: AbortSignal.timeout(5 * 60 * 1000),
+    });
+
+    if (!res.ok) {
+      const errText = await res.text().catch(() => "Upstream error");
+      return new Response(
+        JSON.stringify({ error: errText }),
+        { status: res.status, headers: { "Content-Type": "application/json" } }
+      );
+    }
+
+    // Pipe the SSE stream through to the client
+    return new Response(res.body, {
+      headers: {
+        "Content-Type": "text/event-stream",
+        "Cache-Control": "no-cache",
+        Connection: "keep-alive",
+        "X-Accel-Buffering": "no",
+      },
+    });
+  } catch (error: unknown) {
+    const isTimeout =
+      error instanceof Error &&
+      (error.name === "TimeoutError" || error.name === "AbortError");
+    return new Response(
+      JSON.stringify({
+        error: isTimeout ? "Transcription timed out" : "Failed to reach Modal API",
+      }),
+      { status: isTimeout ? 504 : 502, headers: { "Content-Type": "application/json" } }
+    );
+  }
+}

web/src/app/studio/page.tsx

@@ -28,8 +28,6 @@ import { cn } from "@/lib/utils"
 import { MagnifyingGlass } from "@phosphor-icons/react"
 import NextImage from "next/image"
 
-const API_BASE = process.env.NEXT_PUBLIC_API_URL ?? "http://localhost:3000"
-
 // --- Constants ---
 const SPEAKER_COLORS = [
   { avatar: "bg-blue-400", track: "bg-blue-200" },
@@ -669,12 +667,13 @@ function StudioContent() {
   // Per-segment DOM element refs for auto-scroll
   const segmentRefs = useRef<Map<number, HTMLDivElement>>(new Map())
 
-  const [session, setSession] = useState(() => sessionId ? getSession(sessionId) : null)
+  const [session, setSession] = useState<ReturnType<typeof getSession>>(null)
   const [activeId, setActiveId] = useState<number>(1)
   const [isPlaying, setIsPlaying] = useState(false)
   const [currentTime, setCurrentTime] = useState(0)
   const [isProcessing, setIsProcessing] = useState(false)
   const [processError, setProcessError] = useState<string | null>(null)
+  const [streamingText, setStreamingText] = useState<string | null>(null)
 
   // Sync session state with sessionId param
   useEffect(() => {
@@ -688,8 +687,8 @@ function StudioContent() {
   }, [sessionId])
 
   // Automatic processing for pending sessions.
-  // Uses async job polling: POST returns {job_id} immediately, then GET /api/job/:id
-  // until done — avoids HF Spaces ~3 min proxy timeout during long CPU inference.
+  // Uses Modal streaming API: tokens arrive via SSE for live display,
+  // then the full transcription is converted to a DiarizeResult.
   useEffect(() => {
     if (!session || processingRef.current || processError) return
 
@@ -698,58 +697,75 @@ function StudioContent() {
       const process = async () => {
         setIsProcessing(true)
         setProcessError(null)
+        setStreamingText("")
         try {
-          // 1. Submit job — server responds immediately with job_id (202)
           const formData = new FormData()
           formData.append("audio", session.file!, session.filename)
 
-          const submitRes = await fetch(`${API_BASE}/api/transcribe-diarize`, {
+          const res = await fetch("/api/transcribe-stream", {
             method: "POST",
             body: formData,
           })
 
-          if (!submitRes.ok) {
-            const errData = await submitRes.json().catch(() => ({}))
-            throw new Error(errData.error ?? "Submit failed")
+          if (!res.ok) {
+            const errData = await res.json().catch(() => ({}))
+            throw new Error(errData.error ?? "Transcription failed")
           }
 
-          const submitJson = await submitRes.json() as { job_id?: string } & Partial<DiarizeResult>
-
-          let data: DiarizeResult
-
-          if (submitJson.job_id) {
-            // New async format: poll for result
-            const job_id = submitJson.job_id
-            const POLL_INTERVAL = 3000
-            const MAX_POLLS = 60 * 20  // 60 min max
-            let polls = 0
-
-            data = await new Promise<DiarizeResult>((resolve, reject) => {
-              const tick = async () => {
-                polls++
-                if (polls > MAX_POLLS) {
-                  reject(new Error("Processing timed out after 60 minutes"))
-                  return
+          // Consume SSE stream
+          const reader = res.body!.getReader()
+          const decoder = new TextDecoder()
+          let fullText = ""
+          let buffer = ""
+
+          while (true) {
+            const { done, value } = await reader.read()
+            if (done) break
+
+            buffer += decoder.decode(value, { stream: true })
+            const lines = buffer.split("\n")
+            buffer = lines.pop() ?? ""
+
+            for (const line of lines) {
+              if (!line.startsWith("data: ")) continue
+              try {
+                const payload = JSON.parse(line.slice(6))
+                if (payload.token) {
+                  fullText += payload.token
+                  setStreamingText(fullText)
                 }
-                try {
-                  const pollRes = await fetch(`${API_BASE}/api/job/${job_id}`)
-                  const pollData = await pollRes.json()
-                  if (pollData.status === "done") {
-                    resolve(pollData.data as DiarizeResult)
-                  } else if (pollData.status === "error") {
-                    reject(new Error(pollData.error ?? "Processing failed"))
-                  } else {
-                    setTimeout(tick, POLL_INTERVAL)
-                  }
-                } catch (e) {
-                  reject(e)
+                if (payload.done) {
+                  fullText = payload.transcription ?? fullText
+                  setStreamingText(fullText)
                 }
+              } catch {
+                // skip malformed SSE lines
               }
-              setTimeout(tick, POLL_INTERVAL)
-            })
-          } else {
-            // Fallback: old proxy / local dev returned result directly
-            data = submitJson as DiarizeResult
+            }
+          }
+
+          // Get audio duration from the media element
+          const mediaDuration = mediaRef.current?.duration || 0
+
+          // Derive emotion from the first bracket tag in transcription
+          const firstTagMatch = fullText.match(/\[([^\]]+)\]/)
+          const firstTag = firstTagMatch ? getTagEntry(firstTagMatch[1]) : null
+
+          // Build DiarizeResult from plain transcription
+          const data: DiarizeResult = {
+            segments: fullText.trim() ? [{
+              id: 1,
+              speaker: "SPEAKER_00",
+              start: 0,
+              end: mediaDuration || 30,
+              text: fullText.trim(),
+              emotion: firstTag?.emotion ?? "Neutral",
+              valence: firstTag?.valence ?? 0,
+              arousal: firstTag?.arousal ?? 0,
+            }] : [],
+            duration: mediaDuration || 30,
+            text: fullText.trim(),
+            filename: session.filename,
           }
 
           updateSession(session.id, data)
@@ -758,9 +774,11 @@ function StudioContent() {
           if (updated?.data.segments && updated.data.segments.length > 0) {
             setActiveId(updated.data.segments[0].id)
           }
+          setStreamingText(null)
         } catch (e) {
           processingRef.current = false
           setProcessError(e instanceof Error ? e.message : "Request failed")
+          setStreamingText(null)
         } finally {
           setIsProcessing(false)
         }
@@ -850,7 +868,9 @@ function StudioContent() {
             {isProcessing && (
               <Badge variant="secondary" className="bg-blue-500/10 text-blue-500 hover:bg-blue-500/10 border-blue-500/20 gap-2 font-medium px-3 h-8">
                 <div className="size-2 rounded-full bg-blue-500 animate-pulse" />
-                Analysing Speech...
+                {streamingText !== null && streamingText.length > 0
+                  ? "Streaming..."
+                  : "Connecting to Evoxtral..."}
               </Badge>
             )}
 
@@ -889,15 +909,37 @@ function StudioContent() {
         <div className="flex-1 overflow-hidden flex flex-col min-w-0 relative">
           {isProcessing && segments.length === 0 && (
             <div className="absolute inset-0 z-20 bg-background/80 backdrop-blur-sm flex flex-col items-center justify-center p-8 text-center animate-in fade-in duration-500">
-              <div className="size-16 mb-6 relative">
-                <div className="absolute inset-0 border-4 border-muted rounded-full" />
-                <div className="absolute inset-0 border-4 border-primary border-t-transparent rounded-full animate-spin" />
-                <Waveform size={24} className="absolute inset-0 m-auto text-primary animate-pulse" />
-              </div>
-              <h2 className="text-xl font-bold mb-2">Analysing Audio</h2>
-              <p className="text-muted-foreground text-sm max-w-xs mx-auto">
-                Voxtral is currently transcribing and identifying speakers. This should only take a moment...
-              </p>
+                {streamingText !== null && streamingText.length > 0 ? (
+                /* Live streaming tokens */
+                <div className="max-w-lg w-full space-y-4">
+                  <div className="flex items-center justify-center gap-2 mb-4">
+                    <div className="size-2 rounded-full bg-emerald-500 animate-pulse" />
+                    <span className="text-sm font-semibold text-emerald-600">Streaming transcription...</span>
+                  </div>
+                  <div className="bg-muted/40 rounded-xl p-6 text-left max-h-64 overflow-y-auto">
+                    <p className="text-[15px] leading-relaxed font-medium tracking-tight whitespace-pre-wrap">
+                      {streamingText}
+                      <span className="inline-block w-[2px] h-[1em] bg-primary animate-pulse ml-0.5 align-text-bottom" />
+                    </p>
+                  </div>
+                  <p className="text-muted-foreground text-xs">
+                    Powered by Evoxtral — expressive tagged transcription
+                  </p>
+                </div>
+              ) : (
+                /* Initial loading spinner before first token arrives */
+                <>
+                  <div className="size-16 mb-6 relative">
+                    <div className="absolute inset-0 border-4 border-muted rounded-full" />
+                    <div className="absolute inset-0 border-4 border-primary border-t-transparent rounded-full animate-spin" />
+                    <Waveform size={24} className="absolute inset-0 m-auto text-primary animate-pulse" />
+                  </div>
+                  <h2 className="text-xl font-bold mb-2">Transcribing Audio</h2>
+                  <p className="text-muted-foreground text-sm max-w-xs mx-auto">
+                    Connecting to Evoxtral... tokens will stream in real-time.
+                  </p>
+                </>
+              )}
             </div>
           )}