Dragonfly · Telugu + Hindi + English speech recognition

Dragonfly is a fine-tune of openai/whisper-large-v3-turbo (809M parameters) for Telugu, Hindi and English, including speech that mixes them ("Tinglish", "Hinglish").

The model writes the way people type mixed speech. English words stay in Latin script inside Telugu or Hindi sentences, and sentences get punctuation:

Language	Example output
Telugu	ఈ సమయంలో iraqలో తీసుకునే చర్యల వలన వర్గాల మధ్య యుద్ధం…
Hindi	file new और text document option पर click करें.
English	to the north and within easy reach is the romantic and fascinating town of cintra…

Usage

import torch
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="balasriharsha/dragonfly",
    torch_dtype=torch.float16,
    device="cuda",  # or "mps" on Apple silicon, "cpu"
)

# Pass the main language of the audio: "telugu", "hindi" or "english".
print(asr("audio.wav", generate_kwargs={"language": "telugu", "task": "transcribe"})["text"])

# Long recordings (> 30 s): let the pipeline chunk them.
print(asr("long.wav", chunk_length_s=30, batch_size=8, generate_kwargs={"language": "hindi"})["text"])

Setting language is recommended. Without it Whisper guesses the language from the first 30 seconds, and code-mixed audio can be guessed wrong.

Short clips (about 3 s or less) can occasionally loop and repeat a phrase. All the benchmark numbers below were produced with this guard, which only changes clips that loop:

out = asr(audio, generate_kwargs={
    "language": "telugu",
    "temperature": (0.0, 0.2, 0.4, 0.6, 0.8),  # retry with sampling only if the greedy output loops
    "compression_ratio_threshold": 2.4,
    "logprob_threshold": None, "no_speech_threshold": None,
})

whisper.cpp (Mac, CPU, on-device)

ggml-dragonfly-q5_0.bin (547 MB) runs with whisper.cpp. On 100 Telugu and 100 Hindi clips it matches the full model within 0.1 point of CER.

whisper-cli -m ggml-dragonfly-q5_0.bin -l te -nt -bs 1 -bo 1 -nf -f audio.wav

Use -nf (no temperature fallback). whisper.cpp's fallback uses token entropy to detect loops, and it mistakes normal Telugu text for a loop. That re-decodes good audio with random sampling and raised Telugu CER from 5.8% to 9.2% in our test. Long Telugu sentences also need more than whisper.cpp's default 220 tokens per segment: raise the n_max = whisper_n_text_ctx(ctx)/2 - 4 limit in src/whisper.cpp to the full context.

Benchmarks

Word error rate (WER %, lower is better). Every system was run by us on the same clips, in November 2026. The Sarvam Saaras V4 numbers come from Sarvam's public API.

Language	Test set	Clips	Dragonfly	Saaras V4	Whisper large-v3-turbo
Telugu	IndicVoices + FLEURS (held-out conversational)	496	22.1	23.1	97.8
Telugu	IndicContextEval, L1	980	29.0	20.5	91.0
Telugu	Kathbath test (read speech)	500	22.0	not measured	90.2
Hindi	IndicVoices-hi (conversational)	300	19.5	13.0	45.8
Hindi	FLEURS-hi (read speech)	300	17.6	7.8	27.3
Hindi	MUCS 2021 (Hindi-English code-mixed)	300	26.8	51.2	60.4
English	FLEURS-en	300	6.1	4.6	4.8
English	Svarah (Indian-accented English)	300	15.4	not measured	7.3

How the numbers were made

• Normalisation: lowercase, punctuation removed and the Indic NLP normaliser applied before scoring.
• Hindi rows: both reference and output are converted to one (Latin) script first. The references write English words in Devanagari, while this model writes them in Latin, and that difference is not a recognition error.
• IndicContextEval: scored with the paper's method, with punctuation stripped. The model writes punctuation and the benchmark does not, so with punctuation counted this model's WER is 38.7%.
• Decoding: greedy, with the loop guard shown above. The same 300 clips were drawn at random (seed 0) from each Hindi and English test set; the Telugu sets use fixed clip lists.

Punctuation F1 (. , ?, on correctly recognised words):

• FLEURS-en: 86.5, the same as Whisper large-v3-turbo (86.3) and Saaras V4 (86.3).
• Telugu held-out set: 87.0.

Known limitations

• Numbers come out as words ("thirty nine", "one six nine eight zero four"), not digits. Add an inverse-text-normalisation step if you need digits. This is most of the Svarah gap.
• Indian names in English speech are sometimes written in Telugu or Devanagari script (for example మహారాష్ట్ర inside an English sentence). This affects about 8% of Svarah clips.
• English accent coverage: the English training data is mostly US/UK read speech (LibriTTS-R), so Indian-accented English is weaker than stock Whisper.
• Read Hindi and IndicContextEval Telugu: on these, Saaras V4 is clearly more accurate.
• No timestamps: the model was trained without timestamp tokens.
• Languages: only Telugu, Hindi and English were trained. Other languages may be degraded compared with the base model.

Training

• Start point: a Telugu fine-tune of whisper-large-v3-turbo (itself trained on the Telugu data below).
• Run: 29,092 steps, one epoch over 931k clips; effective batch 32; learning rate 1e-5 with 500 warm-up steps and linear decay; bf16. Trained on one NVIDIA L40S, about 25 hours.
• Labels: each clip's transcript is prefixed with its own language token.

Language	Data	Hours (approx.)
Telugu	IndicVoices (473 h), Kathbath (142 h), Vaani (108 h), FLEURS (8 h)	~730
Hindi	IndicVoices (787 h), MUCS 2021 (83 h), FLEURS (6 h)	~875
English	LibriTTS-R (about half of train-clean-100 + 360), FLEURS en_us	~130

How the labels were made

• Script style: English words inside Telugu and Hindi transcripts were converted to Latin script with an LLM (DeepSeek), using consistent spellings.
• Hindi punctuation: restored with a punctuation model trained on FLEURS and Sangraha text (CC BY 4.0).
• English labels: lowercased, keeping . , ?.

Licence and attribution

The model weights are released under the MIT licence, like the base model. They were trained on:

• IndicVoices and Kathbath (AI4Bharat);
• Vaani (ARTPARK, IISc and Google);
• FLEURS (Google);
• LibriTTS-R;
• MUCS 2021 Hindi-English, OpenSLR SLR104 (CC BY-SA 4.0);
• Sangraha (AI4Bharat).

Svarah (AI4Bharat) and IndicContextEval were used only for evaluation. Please cite and credit these datasets when you use the model.