| --- |
| license: cc-by-nc-4.0 |
| language: |
| - multilingual |
| library_name: coreml |
| tags: |
| - forced-alignment |
| - mms |
| - wav2vec2 |
| - ctc |
| - coreml |
| - lyrics |
| - ios |
| pipeline_tag: automatic-speech-recognition |
| --- |
| |
| # MMS-FA Multilingual Forced Aligner (CoreML) |
|
|
| On-device CTC **forced aligner** for word-level lyric timestamps, multilingual |
| (1000+ languages via romanization). Alignment half of a WhisperX-style pipeline: |
| a transcript is aligned to audio by Viterbi-decoding CTC emissions. |
|
|
| Converted from `torchaudio.pipelines.MMS_FA` (Meta MMS `ctc_alignment_mling_uroman`). |
|
|
| > ⚠️ **LICENSE: CC-BY-NC-4.0 (non-commercial).** Inherited from Meta's MMS |
| > release. Hosting/attribution is fine; **shipping in a commercial app requires |
| > resolving the non-commercial restriction first.** For commercial production |
| > use the Apache-2.0 `wav2vec2-base960h-aligner-coreml` (English) instead, or |
| > obtain a commercial MMS license / a differently-licensed multilingual model. |
|
|
| ## I/O contract |
|
|
| | | | |
| |---|---| |
| | input | `waveform` float32 `[1, 960000]` — a fixed **60 s** window @ 16 kHz mono | |
| | output | `emissions` float32 `[1, 2999, 28]` — log-probs; **20 ms / frame** (stride 320) | |
| | dict | `mms_fa_dict.json` = `{label: index}`, blank `-`=0, then lowercase romanized a–z + apostrophe (no word-separator) | |
|
|
| Text must be **romanized** (uroman) before tokenization for non-Latin scripts. |
| Long audio: overlapping 60 s windows (hop 50 s), center-crop, single CTC Viterbi |
| over concatenated emissions. A synthetic star (garbage) token wraps the |
| transcript to absorb instrumental intros/outros — align on a **separated vocal |
| stem** for heavily-produced tracks. Precision FLOAT16; target iOS 17. |
|
|
