docs/research/references.md

# Evoxtral — Research References & Sources

References for the technical report covering expressive tagged transcription, ASR evaluation, and post-training methods.

## Core Model

- **Voxtral: An Audio-Language Model** — Mistral AI, 2025. Voxtral Mini and Small are open-weights audio chat models trained with SFT + DPO + Online DPO. Online DPO delivers crisper grounding, fewer hallucinations, and more helpful responses.
  - Paper: https://arxiv.org/abs/2507.13264
  - Model: https://huggingface.co/mistralai/Voxtral-Mini-3B-2507

## RL / Post-Training for ASR

- **Group Relative Policy Optimization for Speech Recognition** — Proposes GRPO with rule-based rewards for LLM-based ASR. Achieved 18% relative WER reduction on AMI-IHM and 27.9% on AMI-SDM compared to SFT-adapted models.
  - Paper: https://arxiv.org/abs/2509.01939

- **Advancing Speech Understanding in Speech-Aware Language Models with GRPO** — Applies GRPO to large audio language models, investigating different rule-based reward functions and RL data construction strategies.
  - Paper: https://arxiv.org/abs/2509.16990

- **Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering** — Demonstrates RL (GRPO) achieves state-of-the-art on audio QA tasks, outperforming SFT baselines.
  - Paper: https://arxiv.org/abs/2503.11197

- **Explore the Reinforcement Learning for LLM-based ASR and TTS System** — Surveys RL applications to speech models, noting that while RL has enhanced text-based LLMs, application to ASR/TTS remains underexplored.
  - Paper: https://arxiv.org/abs/2509.18569

## ASR Evaluation Metrics

- **Speech Recognition Accuracy: Production Metrics & Optimization 2025** — Deepgram. Covers WER, CER, Keyword Recall Rate (KRR), Real-Time Factor (RTF), and end-to-end latency. Production systems need blended metrics depending on use case.
  - Source: https://deepgram.com/learn/speech-recognition-accuracy-production-metrics

- **Moving Beyond Word Error Rate to Evaluate ASR in Clinical Samples** — Argues WER alone is insufficient; error type, meaning, and context matter. A single substitution can drastically change intent with the same WER penalty.
  - Paper: https://www.sciencedirect.com/science/article/pii/S0165178125003385

- **Measuring the Accuracy of Automatic Speech Recognition Solutions** — ACM survey on ASR evaluation methodology, limitations of WER/CER, and alternative metrics.
  - Paper: https://dl.acm.org/doi/10.1145/3636513

- **On the Robust Approximation of ASR Metrics** — ACL 2025. Novel label-free approach for approximating ASR performance using multimodal embeddings.
  - Paper: https://arxiv.org/abs/2502.12408

- **ProfASR-Bench: A Professional-talk ASR Dataset for High-Stakes Applications** — Evaluation suite supporting conventional metrics plus entity-aware scores and slice-wise reporting by accent and gender.
  - Paper: https://arxiv.org/abs/2512.23686

## Expressive Speech & TTS

- **ElevenLabs v3 Text-to-Speech** — TTS model supporting inline expressive audio tags for emotions, non-verbal sounds, and delivery cues. Tag set used as target vocabulary for Evoxtral.
  - Docs: https://elevenlabs.io/docs/api-reference/text-to-speech

## Fine-Tuning Frameworks

- **ms-swift** — ModelScope framework supporting SFT/DPO/GRPO for 600+ LLMs and 300+ MLLMs. AAAI 2025.
  - GitHub: https://github.com/modelscope/ms-swift

- **OpenRLHF** — Scalable agentic RL framework based on Ray, supporting PPO, DAPO, REINFORCE++, and more.
  - GitHub: https://github.com/OpenRLHF/OpenRLHF

- **PEFT (Parameter-Efficient Fine-Tuning)** — HuggingFace library for LoRA and other adapter methods.
  - GitHub: https://github.com/huggingface/peft

## Evaluation Methodology

- **jiwer** — Python library for WER, CER, and other ASR metrics based on edit distance.
  - GitHub: https://github.com/jitsi/jiwer

## Our Benchmark: Evoxtral-Bench

Metrics computed by `scripts/train_modal.py::evaluate()`:

| Metric | Description | Direction |
|--------|-------------|-----------|
| **WER** | Word Error Rate on plain text (tags stripped) | lower = better |
| **CER** | Character Error Rate on plain text | lower = better |
| **Tag F1** | F1 score for tag extraction (multiset intersection) | higher = better |
| **Tag Precision** | Fraction of predicted tags that match reference | higher = better |
| **Tag Recall** | Fraction of reference tags captured by prediction | higher = better |
| **Tag Hallucination Rate** | Fraction of predicted tags not in reference at all | lower = better |
| **Emphasis F1** | F1 for CAPITALIZED emphasis words | higher = better |
| **Per-Tag F1** | Breakdown by individual tag type (e.g. [laughs], [sighs]) | higher = better |

### Why These Metrics

- **WER + CER**: Standard ASR quality. CER is more granular and catches character-level errors WER misses.
- **Tag Precision vs Recall**: F1 alone hides whether the model hallucinates tags (low precision) or misses them (low recall). Judges care about this distinction.
- **Tag Hallucination Rate**: Critical for downstream TTS — hallucinated tags produce wrong prosody/effects.
- **Per-Tag Breakdown**: Shows which expressive cues the model handles well vs struggles with, informing future data collection.
- **Emphasis F1**: CAPS words (e.g., "BELIEVE", "NEVER") carry prosodic emphasis in ElevenLabs v3; measuring this separately tracks delivery accuracy.