README.md

---
title: README
emoji: 🗣
colorFrom: indigo
colorTo: pink
sdk: static
pinned: false
---

# TinyAya — Turkish ⇄ Hindi Speech-to-Speech Translation

Can a **text-pretrained multilingual model** be taught to translate *speech to
speech* — for a language pair with almost no parallel speech data?

That is what this organization holds: a LoRA-adapted **Cohere2** backbone driving
a **frozen Moshi depth decoder** over **Mimi** audio codes, with a text
"inner-monologue" running alongside the audio stream. Everything is here — the
text corpus, the synthetic speech, the encoded tokens, the model, and every
training checkpoint.

## What v0.3 actually achieved

The full-corpus run covered **76,250 steps / 2.07 epochs** on a Cloud TPU v6e-16,
reaching a best validation composite of **2.8199** at step 76,000. Read honestly,
capability arrived in a clear order — **language identity → text translation →
audio synthesis** — and the run bought the first two:

| what we measured (`@best`, 500 held-out rows, greedy) | hi→tr | tr→hi |
|---|---|---|
| free-run **text** chrF++ — the inner-monologue | **25.7** | **25.1** |
| generated-audio **ASR-chrF++** | 3.7 | 9.6 |
| **ground-truth-audio ceiling** for the same pipeline | 92.1 | 86.6 |
| BLASER-2.0 QE (ASR-free speech semantics, 1–5) | 2.53 | 2.49 |

**The model learns the translation mapping; intelligible audio synthesis is the
frontier.** The gap between 3.7/9.6 and the 92/87 ceiling is not a broken
pipeline — that ceiling was measured through the same codec and ASR — it is the
frozen depth decoder. BLASER-2.0 sees more meaning in the generated speech than
a transcript recovers, which says the audio carries real signal that is simply
not yet intelligible.

We publish the number next to its ceiling every time. A low score with no ceiling
beside it is unreadable.

## Start here

| | |
|---|---|
| 🎧 **Model** | [`tr-hi-s2st-v0.3`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3) — ~89-checkpoint suite + playable audio |
| 📊 **Full results** | [v0.3 evaluation report](https://github.com/tiny-aya-simultaneous-translation/model/blob/main/docs/v0.3-eval-report.md) |
| 🗂 **Training corpus** | [`tr-hi-mimi-encoded`](https://huggingface.co/datasets/tiny-aya-translate/tr-hi-mimi-encoded) |
| 💻 **Code** | [github.com/tiny-aya-simultaneous-translation](https://github.com/tiny-aya-simultaneous-translation) |
| ✍️ **Write-up** | [Adapting Moshi for Low-Resource Speech Translation](https://labscommunity.cohere.com/blog/2026/adapting-moshi-low-resource-speech-translation/) |
| 📈 **Training run** | [W&B `xzcb60bl`](https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/xzcb60bl) · [emergence report](https://wandb.ai/cataluna84/tinyaya-stage2-tpu/reports/TinyAya-v0.3-Emergence-and-Data-Efficiency--VmlldzoxNzU1OTU1NQ==) |

## How the data becomes the model

```
tr-hi-parallel-text          text triples (en pivot -> tr / hi)
        |  TTS
tr-hi-parallel-speech-v2     synthetic speech + round-trip ASR quality signals
        |  Mimi encode
tr-hi-mimi-encoded           8 codebooks @ 12.5 Hz + word-level alignments
        |  Stage-2 training
tr-hi-s2st-v0.3              the released model
```

## Also in this organization

- **Model history** — [`v0.1`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.1)
  and [`v0.2`](https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.2) are kept
  as honest history, including the disclosed mistakes: v0.2 trained on the wrong
  dataset via a launcher default, and both trained audio-only because a filename
  check wrongly concluded the corpus had no text alignments. It ships 840,426 of
  them.
- **Evaluation sets** — [`fleurs-tr-hi-mimi-encoded`](https://huggingface.co/datasets/tiny-aya-translate/fleurs-tr-hi-mimi-encoded)
  (real human speech; **acoustic shift only** — its texts overlap training and it
  must never be called held-out text),
  [`lahaja-eval`](https://huggingface.co/datasets/tiny-aya-translate/lahaja-eval) (Hindi, dialect-labelled),
  [`cv-tr-eval`](https://huggingface.co/datasets/tiny-aya-translate/cv-tr-eval) (Turkish).
- **Codec research** — [`dualcodec-hindi-25hz`](https://huggingface.co/tiny-aya-translate/dualcodec-hindi-25hz)
  and [`dualcodec-turkish-25hz`](https://huggingface.co/tiny-aya-translate/dualcodec-turkish-25hz):
  can a neural codec fine-tuned on a low-resource language beat the stock
  multilingual one? Directly relevant, since codec quality is what caps v0.3.

## Caveats worth knowing before you use this

Training speech is **entirely synthetic TTS**, and the model is measurably
distribution-bound — on real human speech the text score collapses from ~25 to
~8 chrF++. Translation references are machine-translated, so they carry their own
ceiling. This is a **research artifact** for low-resource S2ST, not a production
translator.

## Licensing

Released **weights are CC-BY-NC-4.0** — non-commercial, inherited from
[`CohereLabs/tiny-aya-base`](https://huggingface.co/CohereLabs/tiny-aya-base).
The training and evaluation **code is Apache-2.0**. Moshi/Mimi components are
CC-BY-4.0. Corpus text derives from FLORES (**CC BY-SA 4.0 — share-alike
propagates**), OPUS-100 and TTS output; check each dataset card before
redistributing.

Compute for the v0.3 run was provided by **[Google's TPU Research Cloud (TRC)](https://sites.research.google/trc/)**.