README.md

---
license: apache-2.0
language:
- hi
- en
tags:
- automatic-speech-recognition
- whisper
- hindi
- hinglish
- code-switching
- speech-recognition
pipeline_tag: automatic-speech-recognition
---

# Tara: Frontier Hindi Transcription Model

> Watch the launch video [here](https://youtu.be/R4-p5JuaBG4). Try via API [here](https://router.trelis.com/models).

**Tara is a frontier automatic speech-recognition model for Hindi and mixed-code (Hinglish)
transcription.** On the AI4Bharat Vistaar Hindi benchmark suite it achieves state-of-the-art
aggregate accuracy, outperforming leading commercial Hindi ASR systems on the 7-benchmark
Vistaar mean, while natively handling Hindi–English code-switched speech through a dedicated **mixed-code
mode** that renders English words in Latin script and Hindi in Devanagari, the way real
Hinglish is written.

## Highlights

- **State-of-the-art Vistaar Hindi aggregate**: 12.06 WER mean over the 7 Vistaar sets, ahead
  of Sarvam Saaras-v3 (12.32), with wins on Kathbath, GramVaani, IndicTTS and CommonVoice-hi.
- **Native code-switching**: 8.37 WER on Code-Switch FLEURS (CS-FLEURS) Hindi–English read code-switch via Tara's
  mixed-code mode, competitive with the best commercial systems.
- **Robust across domains**: read speech, noisy speech, telephony (GramVaani 21.03 vs Sarvam
  23.00), spontaneous conversation (IndicVoices), and accented adult/child speech (HiACC).
- **Bilingual**: retains strong English (6.68 WER CommonVoice-en, 4.55 FLEURS-en).
- **Standard tooling**: loads with 🤗 Transformers exactly like `openai/whisper-large-v3`.

## Usage

```python
import librosa
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

repo = "Trelis/tara"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16).to("cuda")

tk = processor.tokenizer
hi, en, mc = (tk.convert_tokens_to_ids(t) for t in ("<|hi|>", "<|en|>", "<|mixedcode|>"))
trn, nts = (tk.convert_tokens_to_ids(t) for t in ("<|transcribe|>", "<|notimestamps|>"))

audio_16k, _ = librosa.load("clip.wav", sr=16000, mono=True)
feats = processor(audio_16k, sampling_rate=16000,
                  return_tensors="pt").input_features.to("cuda", torch.bfloat16)

# Example 1: pure Hindi
out = model.generate(input_features=feats,
                     forced_decoder_ids=[(1, hi), (2, trn), (3, nts)],
                     max_new_tokens=444)
print(tk.decode(out[0], skip_special_tokens=True))

# Example 2: Hindi-English mixed-code, inject <|mixedcode|> right after the language token.
# Language auto-detection also works: generate one step unforced and the FIRST generated
# token is the language token; then inject <|mixedcode|> after it and continue.
out = model.generate(input_features=feats,
                     forced_decoder_ids=[(1, hi), (2, mc), (3, trn), (4, nts)],
                     max_new_tokens=444)
print(tk.decode(out[0], skip_special_tokens=True))
```

The mixed-code mode (the `<|mixedcode|>` prefix above) conditions generation only: on pure-Hindi
audio it neither degrades accuracy nor forces transliteration; on mixed-code audio it renders
English words in Latin script.

## Evaluation

Evaluation code, the exact text normalizer, and Tara's per-utterance predictions for every
benchmark below are published at [TrelisResearch/tara](https://github.com/TrelisResearch/tara),
so all numbers can be reproduced or re-scored under alternative normalizers.

**Protocol.** All numbers are corpus WER after light text normalization* (Unicode NFC plus
punctuation removal; nukta and all vowel and nasal marks preserved). All systems are scored
on clips ≤ 30 s with identical references. Commercial-system results are measured by us
under the same protocol; they are not vendor-reported figures.

### Vistaar Hindi benchmark (WER ↓)

| Benchmark | **Tara** | Sarvam Saaras-v3 | ElevenLabs Scribe-v2 |
|---|--:|--:|--:|
| [Kathbath](https://huggingface.co/datasets/Trelis/vistaar-hi-kathbath-test) (clean read) | **9.34** | 9.71 | 9.60 |
| [Kathbath-hard](https://huggingface.co/datasets/Trelis/vistaar-hi-kathbath_noisy-test) (noisy) | 10.82 | **10.55** | 11.11 |
| [MUCS](https://huggingface.co/datasets/Trelis/vistaar-hi-mucs-test) | 10.79 | **9.69** | 10.93 |
| [GramVaani](https://huggingface.co/datasets/Trelis/vistaar-hi-gramvaani-test) (telephony) | **21.03** | 23.00 | 26.94 |
| [IndicTTS](https://huggingface.co/datasets/Trelis/vistaar-hi-indictts-test) | **9.46** | 10.38 | 13.17 |
| [CommonVoice-hi](https://huggingface.co/datasets/Trelis/vistaar-hi-commonvoice-test) | **12.51** | 12.88 | 13.44 |
| [FLEURS-hi](https://huggingface.co/datasets/Trelis/vistaar-hi-fleurs-test) | 10.47 | **10.05** | 11.33 |
| **Mean (7 Vistaar sets)** | **12.06** | 12.32 | 13.79 |
| [IndicVoices-500](https://huggingface.co/datasets/Trelis/indicvoices-500-hi-eval) (spontaneous, non-Vistaar) | 16.51 | **15.29** | 27.46 |

IndicVoices-500 is a 500-sample spontaneous-speech control from the IndicVoices validation
split; it is not part of the Vistaar mean.

### Code-switching (Hinglish) benchmarks (WER ↓)

Tara and Sarvam are measured in their code-mixed modes.

| Benchmark | **Tara** | Sarvam Saaras-v3 | ElevenLabs Scribe-v2 |
|---|--:|--:|--:|
| [CoSHE-500](https://huggingface.co/datasets/Trelis/CoSHE-500) (conversational CS) | 14.41 | **11.25** | 12.40 |
| [Code-Switch FLEURS hi-en](https://huggingface.co/datasets/Trelis/cs-fleurs-hineng-read-test) (read CS) | 8.37 | 16.47 | **7.57** |
| [Hi-accent adult (HiACC)](https://huggingface.co/datasets/Trelis/hiacc-adult-test-eval) | 12.93 | 13.16 | **12.87** |
| [Hi-accent child (HiACC)](https://huggingface.co/datasets/Trelis/hiacc-child-test-eval) | 10.69 | **10.10** | 11.66 |

### English (WER ↓)

Scored with the standard Whisper English normalizer.

| Benchmark | **Tara** | Sarvam | Scribe-v2 |
|---|--:|--:|--:|
| [CommonVoice-en](https://huggingface.co/datasets/Trelis/cv-en-scripted-test-500) | 6.68 | 8.68 | **5.28** |
| [FLEURS-en](https://huggingface.co/datasets/Trelis/fleurs-en-test) | 4.55 | 4.36 | **2.93** |

\* Normalization: `unicodedata.normalize("NFC")`, lowercasing, then removal of punctuation
and symbols (`। , . ? ! " : ; - – — “ ” ( ) [ ] < > / ~ % ₹ $ …`), invisible formatting
characters (zero-width joiner/space) and the Unicode replacement character; apostrophes are
kept. The ≤30 s rule excludes 2 clips on GramVaani, 2 on IndicTTS and 1 on FLEURS-hi; no
other set has any. Measurement error is small: re-runs across hardware and precision agree
to within 0.1 WER. There is also slight noise in the reference labels (for example
inconsistent nukta spelling: both हज़ार and हजार appear as references within GramVaani, and
both ज़्यादा and ज्यादा within MUCS), but this should not affect the numbers by much.

## Limitations

- **Mode selection**: peak accuracy comes from picking the mode per clip (Hindi, mixed-code, or
  English). When the language mix is unknown, **the Hindi mixed-code mode is a safe default**: on
  pure-Hindi audio it produces pure-Hindi transcripts with no measured accuracy loss, and on mixed
  audio it handles the code-switching. Automatic language detection is also supported: let the
  model generate the language token and inject mixed-code after it (see Usage).
- **Clip length**: evaluated on clips ≤ 30 s; longer audio should be chunked (standard Whisper
  practice).
- Hindi–English only; other Indic languages are out of scope for this release.

## Intended use

Transcription of Hindi and Hindi–English code-switched speech: voice assistants, contact-center
analytics, media captioning, and speech data pipelines.

## Model details

- **Architecture**: Whisper large-v3 (encoder–decoder, 1.55B params) + mixed-code mode
- **Languages**: Hindi (hi), English (en), Hindi–English code-switch
- **Sample rate**: 16 kHz input
- **I/O**: ≤30 s audio per window → text
- **License**: Apache 2.0

## Attribution

We thank [Gram Vaani](https://gramvaani.org) for permission to use the
[Gram Vaani ASR Challenge 2022 Corpus](https://www.openslr.org/118/) in training Tara. Gram Vaani
builds community-anchored voice based engagement platforms ('Mobile Vaani' clubs) that give underserved and
marginalised communities a channel to access information and express themselves.

## License

This model is released under the **Apache License 2.0**.

## Citation

If you use Tara in your work, please cite:

```bibtex
@misc{trelis2026tara,
  title  = {Tara: Frontier Hindi Transcription Model},
  author = {{Trelis Research}},
  year   = {2026},
  url    = {https://huggingface.co/Trelis/tara}
}
```