|
Download README.md from jsbeaudry/Routyo: direct link, hf CLI and curl.
- Browser
- Download file 7.6 kB
-
https://huggingface.co/jsbeaudry/Routyo/resolve/main/README.md
- Command line
-
hf download hf://jsbeaudry/Routyo/README.md
-
curl -L -o README.md https://huggingface.co/jsbeaudry/Routyo/resolve/main/README.md
7.6 kB
| license: apache-2.0 | |
| language: | |
| - ht | |
| - en | |
| - fr | |
| - es | |
| pipeline_tag: audio-classification | |
| base_model: Cactus-Compute/whistle | |
| tags: | |
| - language-identification | |
| - spoken-language-identification | |
| - haitian-creole | |
| - kreyol | |
| - whistle | |
| - on-device | |
| # Routyo: spoken-language router for Haitian Creole, English, French and Spanish | |
| Routyo listens to a sentence and says which language it is: **Haitian Creole (ht), English (en), French (fr) or | |
| Spanish (es)**, with a rate for each. It is made to sit in front of small speech and language models: it sends each | |
| sentence to the right one, so Kreyòl and the other three languages can be mixed in one conversation without any | |
| button to press. | |
| It works with any voice and any accent. A Haitian speaking English is English; a foreigner speaking Kreyòl is Kreyòl. | |
| It runs on CPU (about 35 ms per sentence on a Mac) and weighs about 50 MB in all. | |
| Built by a Haitian team, on one Mac, for the 12+ million people who speak Kreyòl and for anyone who moves between these four | |
| languages. | |
| ## Results | |
| Whole sentences, four-way accuracy. Nothing below was used in training: other speakers, other voices, other | |
| sentences. "Pair" asks Kreyòl against one other language only, as an interpreter app does. | |
| Whisper large-v3-turbo is scored on the same clips with its language guess limited to the same four languages. | |
| | test set | clips | **Routyo** (+ word check) | Routyo, sound only | Whisper large-v3-turbo | | |
| |---|---|---|---|---| | |
| | real speech, other people | 1,751 | **99.7 %** (pair 99.8) | 99.7 % | 91.7 % | | |
| | synthetic voices never heard in training (Haitian, foreign, native) | 2,130 | **96.6 %** (pair 98.5) | 95.2 % | 51.5 % | | |
| | earlier synthetic Haitian voices, held out | 628 | **96.7 %** (pair 97.9) | 97.1 % | 64.6 % | | |
| | one Haitian speaker, live on a phone | 47 | **93.6 %** (pair 97.9) | 89.4 % | 74.5 % | | |
| On Kreyòl itself the gap is widest. Kreyòl spoken in foreign voices: Routyo 97.3 %, Whisper 27.9 %. Haitian voices | |
| speaking English, French or Spanish: Routyo 95.1 %, Whisper 31.9 %. | |
| From only the first 1.5 s of speech (sound only): real speech 92.7 % (pair 96.2), unseen voices 87.4 % (pair 93.4). | |
| Speed on an Apple-silicon Mac, 200 real sentences of 8.7 s on average: | |
| | | per sentence | size | | |
| |---|---|---| | |
| | **Routyo**, sound only (CPU) | **35 ms** | ~50 MB | | |
| | Routyo + word check (CPU) | ~0.3 s | same | | |
| | Whisper large-v3-turbo language detection (Mac GPU) | 0.8 s | 1.5 GB | | |
| ## Use it | |
| ```bash | |
| pip install cactus-compute numpy soundfile soxr huggingface_hub | |
| hf download jsbeaudry/Routyo --local-dir routyo | |
| python routyo/routyo.py clip.wav # all four languages | |
| python routyo/routyo.py clip.wav --words # with the word check (more accurate, ~0.3 s) | |
| python routyo/routyo.py clip.wav --pair ht,en # Kreyòl against English only | |
| python routyo/routyo.py clip.wav --first 1.5 # decide from the first 1.5 s of speech | |
| ``` | |
| ```python | |
| import sys; sys.path.insert(0, "routyo") | |
| from routyo import Routyo, load_audio | |
| r = Routyo() # downloads the stock Whistle encoder on first use | |
| r.rates(load_audio("clip.wav")) # [{"language": "en", "name": "English", "rate": 0.987}, ...] | |
| r.rates(samples_16k, pair=("ht", "fr")) # 16 kHz mono float32 samples work too | |
| ``` | |
| Runs on the [Cactus](https://github.com/cactus-compute/cactus) engine (`cactus-compute`, tested with 4.1.0), | |
| which runs Whistle on CPU. The results above were measured with its earlier version (`cactus-needle`, engine | |
| 3.3.3); on 400 test clips the two engines pick the same language 397 times, with the same accuracy. | |
| ## How it works | |
| | piece | what | | |
| |---|---| | |
| | encoders (frozen) | the stock [Whistle](https://huggingface.co/Cactus-Compute/whistle) encoder (`whistle.cact`, downloaded from its own repo) and a Kreyòl-tuned Whistle (`whistle-kreyol.cact`, in this repo). Each gives 512 numbers per 80 ms through the engine's `embed` call; Routyo joins them (1,024 per frame). | | |
| | head (`routyo.npz`) | per-frame layers (1,024 → 256 → 256, ReLU), then attentive mean and spread over time beside the plain mean and spread, then 4 languages. Three such heads, trained from different seeds, are averaged (log-probabilities). Each is temperature-calibrated. | | |
| | word check (optional, `word_check.json`) | Whistle transcribes the sentence forced into each language (the Kreyòl encoder for ht, the stock one for en/fr/es); the mean word probability `wp[k]` joins the sound score: `score_k = (log p[k] + β·wp[k] + c_k) / T`. β, c and T were fitted on validation data only. | | |
| `routyo.json` holds the same head as plain arrays with the combining rule, for porting to any runtime (it is what a | |
| phone app loads). | |
| The Kreyòl-tuned encoder is Whistle fine-tuned on about 145 hours of transcribed speech, mostly Haitian Creole plus | |
| some French, English and Spanish (FLEURS) so it keeps those languages. As a transcriber it reaches 14.6 % word error | |
| rate on our Kreyòl test set; here only its encoder frames and its forced Kreyòl transcription are used. | |
| ## Training data | |
| **Real speech**: Kreyòl 12.4 h (radio, phone recordings, read speech, Bible readings, studio recordings, online | |
| videos), English 6.3 h, Spanish 3.9 h, French 3.1 h (FLEURS, MInDS-14, VoxPopuli, including accented English). | |
| **Synthetic speech**, made for this model so that no voice and no accent gives the language away: | |
| | kind | engine | voices | gives | | |
| |---|---|---|---| | |
| | Haitian voices | Fatiman V3 (our Kreyòl text-to-speech) | 13 consented Haitian voices (10 for training, 3 kept for test) | Haitian-accented English, French, Spanish, and Kreyòl | | |
| | foreign voices speaking Kreyòl | Fatiman V3 cloning catalogue voices | 102 voices with US, UK, French, Spanish, Mexican, German and Brazilian accents (87 train, 15 test) | Kreyòl in foreign voices | | |
| | native voices | [gradium/phonon-40m-multilingual](https://huggingface.co/gradium/phonon-40m-multilingual) | the same catalogue voices | native English, French and Spanish, and Kreyòl read with a foreign accent | | |
| The texts are everyday sentences written in all four languages with the same meaning, so the topic never tells the | |
| language. Every synthetic clip had to be read back correctly by Whistle in its own language (character error rate | |
| ≤ 0.35): 16,935 of 20,300 clips passed (15.9 h). No voice of a data contributor was cloned. | |
| ## Limits | |
| - Four languages only. Speech in any other language is still put in one of the four. | |
| - Short clips are the hardest: under 2 s, and the first-1.5 s numbers, are lower than whole sentences. | |
| - The live test is one speaker; the unseen-voice test uses synthetic voices. | |
| - The hardest cases: Haitian voices speaking English (91.7 %) and French read by a synthetic voice (91.1 %). | |
| French and Kreyòl share many words and sounds. | |
| ## Files | |
| | file | size | what | | |
| |---|---|---| | |
| | `routyo.py` | 8 KB | inference: audio → language rates (command line and Python) | | |
| | `routyo.npz` | 4.3 MB | the three heads (numpy arrays) | | |
| | `routyo.json` | 22 MB | the same heads as JSON, for other runtimes | | |
| | `whistle-kreyol.cact` | 29 MB | the Kreyòl-tuned Whistle (Cactus `.cact` format) | | |
| | `word_check.json` | 1 KB | word-check weights | | |
| The stock `whistle.cact` (17 MB) is fetched from [Cactus-Compute/whistle](https://huggingface.co/Cactus-Compute/whistle). | |
| ## License and credits | |
| Apache 2.0, like the Whistle base model and the Cactus engine by Cactus Compute. | |
| Thanks to Cactus Compute for Whistle and the Cactus engine, and to Gradium for Phonon-40m-multilingual (CC BY-NC 4.0), which | |
| voiced part of the synthetic training audio. | |