greenadntan/md-asr-v0-report

🇲🇩 md-ASR-v0 — Moldova-Specific ASR: Evaluation & Prototype Path

Meta AI Research Intern Report | Chișinău, April 2026

Overview

This repository contains the complete evaluation and prototype design for a Moldova-specific Automatic Speech Recognition (ASR) system that handles:

• 🇷🇴 Romanian/Moldovan with local accent and correct diacritics (ă â î ș ț)
• 🇷🇺 Russian as spoken in Moldova
• 🔄 Code-switched ro↔ru speech (the primary challenge)
• 🟡 Gagauz and Ukrainian (best-effort)

Key Findings

Finding	Detail
Best zero-shot bilingual model	Whisper large-v3 (8.24% WER ro, 4.17% WER ru on FLEURS)
Best Romanian-only model	Parakeet Ro 110M (3.29% WER on CV-21, not publicly released)
Critical data gap	0h of open ro↔ru code-switched speech; 0h of Gagauz audio
Recommendation	Fine-tune Whisper small + LoRA adapters + language-ID routing
Projected deployment	~107MB on-device, ~250ms latency on Android
Projected improvement	>55% WER reduction vs Whisper small on code-switched Moldovan audio

Repository Contents

File	Description
report/md_asr_v0_report.md	6-page comprehensive report with tables, benchmarks, and recommendations
report/eval_toolkit.py	Evaluation toolkit: WER, CER, Diacritics F1, Code-Switch Accuracy, NE-F1, md-ASR Score
report/colab_demo.py	Colab demo: upload 30s audio, get side-by-side comparison with language tags

Candidate Models Evaluated

Model	FLEURS-ro	FLEURS-ru	Size	On-Device
Whisper large-v3	8.24%	4.17%	3GB	❌
SeamlessM4T-v2	8.87%	7.43%	4.6GB	❌
Canary-1B-v2	6.61%	6.90%	2GB	❌
MMS-1B-all	27.9%	30.3%	1.9GB	❌
md-ASR-v0 (projected)	~12%	~10%	107MB	✅

Quick Start — Evaluation Toolkit

from eval_toolkit import compute_wer, compute_diacritics_f1, compute_md_asr_score

# Romanian WER with diacritics-preserving normalization
result = compute_wer("Am mers la Primăria Chișinău", "Am mers la Primaria Chisinau", lang="ro")
print(f"WER: {result['wer']:.2%}")  # Catches diacritics errors

# Diacritics F1
diac = compute_diacritics_f1("Primăria Chișinău", "Primaria Chisinau")
print(f"Diacritics F1: {diac['diacritics_f1']:.2%}")

# Composite md-ASR Score
score = compute_md_asr_score(wer_ro=0.12, wer_ru=0.10, cs_accuracy=0.65, diacritics_f1=0.92, ne_f1=0.60)
print(f"md-ASR Score: {score['md_asr_score']:.4f}")

Citation

@misc{md-asr-v0,
  title={md-ASR-v0: Moldova-Specific Automatic Speech Recognition — Evaluation & Prototype Path},
  author={Meta AI Moldova Research},
  year={2026},
  note={Internal research prototype report}
}

License

Apache 2.0