license: cc-by-nc-4.0
language:
- multilingual
library_name: coreml
tags:
- forced-alignment
- mms
- wav2vec2
- ctc
- coreml
- lyrics
- ios
pipeline_tag: automatic-speech-recognition
MMS-FA Multilingual Forced Aligner (CoreML)
On-device CTC forced aligner for word-level lyric timestamps, multilingual (1000+ languages via romanization). Alignment half of a WhisperX-style pipeline: a transcript is aligned to audio by Viterbi-decoding CTC emissions.
Converted from torchaudio.pipelines.MMS_FA (Meta MMS ctc_alignment_mling_uroman).
⚠️ LICENSE: CC-BY-NC-4.0 (non-commercial). Inherited from Meta's MMS release. Hosting/attribution is fine; shipping in a commercial app requires resolving the non-commercial restriction first. For commercial production use the Apache-2.0
wav2vec2-base960h-aligner-coreml(English) instead, or obtain a commercial MMS license / a differently-licensed multilingual model.
I/O contract
| input | waveform float32 [1, 960000] — a fixed 60 s window @ 16 kHz mono |
| output | emissions float32 [1, 2999, 28] — log-probs; 20 ms / frame (stride 320) |
| dict | mms_fa_dict.json = {label: index}, blank -=0, then lowercase romanized a–z + apostrophe (no word-separator) |
Text must be romanized (uroman) before tokenization for non-Latin scripts. Long audio: overlapping 60 s windows (hop 50 s), center-crop, single CTC Viterbi over concatenated emissions. A synthetic star (garbage) token wraps the transcript to absorb instrumental intros/outros — align on a separated vocal stem for heavily-produced tracks. Precision FLOAT16; target iOS 17.