Spaces:
Running
Running
| title: README | |
| emoji: 🗣 | |
| colorFrom: indigo | |
| colorTo: pink | |
| sdk: static | |
| pinned: false | |
| # TinyAya — Turkish ⇄ Hindi Speech-to-Speech Translation | |
| Can a **text-pretrained multilingual model** be taught to translate *speech to | |
| speech* — for a language pair with almost no parallel speech data? | |
| That is what this organization holds: a LoRA-adapted **Cohere2** backbone driving | |
| a **frozen Moshi depth decoder** over **Mimi** audio codes, with a text | |
| "inner-monologue" running alongside the audio stream. Everything is here — the | |
| text corpus, the synthetic speech, the encoded tokens, the model, and every | |
| training checkpoint. | |
| ## What v0.3 actually achieved | |
| The full-corpus run covered **76,250 steps / 2.07 epochs** on a Cloud TPU v6e-16, | |
| reaching a best validation composite of **2.8199** at step 76,000. Read honestly, | |
| capability arrived in a clear order — **language identity → text translation → | |
| audio synthesis** — and the run bought the first two: | |
| | what we measured (`@best`, 500 held-out rows, greedy) | hi→tr | tr→hi | | |
| |---|---|---| | |
| | free-run **text** chrF++ — the inner-monologue | **25.7** | **25.1** | | |
| | generated-audio **ASR-chrF++** | 3.7 | 9.6 | | |
| | **ground-truth-audio ceiling** for the same pipeline | 92.1 | 86.6 | | |
| | BLASER-2.0 QE (ASR-free speech semantics, 1–5) | 2.53 | 2.49 | | |
| **The model learns the translation mapping; intelligible audio synthesis is the | |
| frontier.** The gap between 3.7/9.6 and the 92/87 ceiling is not a broken | |
| pipeline — that ceiling was measured through the same codec and ASR — it is the | |
| frozen depth decoder. BLASER-2.0 sees more meaning in the generated speech than | |
| a transcript recovers, which says the audio carries real signal that is simply | |
| not yet intelligible. | |
| We publish the number next to its ceiling every time. A low score with no ceiling | |
| beside it is unreadable. | |
| ## Start here | |
| | | | | |
| |---|---| | |
| | 🎧 **Model** | [`tr-hi-s2st-v0.3`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3) — ~89-checkpoint suite + playable audio | | |
| | 📊 **Full results** | [v0.3 evaluation report](https://github.com/tiny-aya-simultaneous-translation/model/blob/main/docs/v0.3-eval-report.md) | | |
| | 🗂 **Training corpus** | [`tr-hi-mimi-encoded`](https://huggingface.co/datasets/tiny-aya-translate/tr-hi-mimi-encoded) | | |
| | 💻 **Code** | [github.com/tiny-aya-simultaneous-translation](https://github.com/tiny-aya-simultaneous-translation) | | |
| | ✍️ **Write-up** | [Adapting Moshi for Low-Resource Speech Translation](https://labscommunity.cohere.com/blog/2026/adapting-moshi-low-resource-speech-translation/) | | |
| | 📈 **Training run** | [W&B `xzcb60bl`](https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/xzcb60bl) · [emergence report](https://wandb.ai/cataluna84/tinyaya-stage2-tpu/reports/TinyAya-v0.3-Emergence-and-Data-Efficiency--VmlldzoxNzU1OTU1NQ==) | | |
| ## How the data becomes the model | |
| ``` | |
| tr-hi-parallel-text text triples (en pivot -> tr / hi) | |
| | TTS | |
| tr-hi-parallel-speech-v2 synthetic speech + round-trip ASR quality signals | |
| | Mimi encode | |
| tr-hi-mimi-encoded 8 codebooks @ 12.5 Hz + word-level alignments | |
| | Stage-2 training | |
| tr-hi-s2st-v0.3 the released model | |
| ``` | |
| ## Also in this organization | |
| - **Model history** — [`v0.1`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.1) | |
| and [`v0.2`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.2) are kept | |
| as honest history, including the disclosed mistakes: v0.2 trained on the wrong | |
| dataset via a launcher default, and both trained audio-only because a filename | |
| check wrongly concluded the corpus had no text alignments. It ships 840,426 of | |
| them. | |
| - **Evaluation sets** — [`fleurs-tr-hi-mimi-encoded`](https://huggingface.co/datasets/tiny-aya-translate/fleurs-tr-hi-mimi-encoded) | |
| (real human speech; **acoustic shift only** — its texts overlap training and it | |
| must never be called held-out text), | |
| [`lahaja-eval`](https://huggingface.co/datasets/tiny-aya-translate/lahaja-eval) (Hindi, dialect-labelled), | |
| [`cv-tr-eval`](https://huggingface.co/datasets/tiny-aya-translate/cv-tr-eval) (Turkish). | |
| - **Codec research** — [`dualcodec-hindi-25hz`](https://huggingface.co/tiny-aya-translate/dualcodec-hindi-25hz) | |
| and [`dualcodec-turkish-25hz`](https://huggingface.co/tiny-aya-translate/dualcodec-turkish-25hz): | |
| can a neural codec fine-tuned on a low-resource language beat the stock | |
| multilingual one? Directly relevant, since codec quality is what caps v0.3. | |
| ## Caveats worth knowing before you use this | |
| Training speech is **entirely synthetic TTS**, and the model is measurably | |
| distribution-bound — on real human speech the text score collapses from ~25 to | |
| ~8 chrF++. Translation references are machine-translated, so they carry their own | |
| ceiling. This is a **research artifact** for low-resource S2ST, not a production | |
| translator. | |
| ## Licensing | |
| Released **weights are CC-BY-NC-4.0** — non-commercial, inherited from | |
| [`CohereLabs/tiny-aya-base`](https://huggingface.co/CohereLabs/tiny-aya-base). | |
| The training and evaluation **code is Apache-2.0**. Moshi/Mimi components are | |
| CC-BY-4.0. Corpus text derives from FLORES (**CC BY-SA 4.0 — share-alike | |
| propagates**), OPUS-100 and TTS output; check each dataset card before | |
| redistributing. | |
| Compute for the v0.3 run was provided by **[Google's TPU Research Cloud (TRC)](https://sites.research.google/trc/)**. | |