Whisper Hebrish - Whisper Fine Tuned For Code-Switching English With Hebrew

Version 2 (2026-10-17).

A fine-tune of openai/whisper-large-v3-turbo for English speech that contains Hebrew words, the way English-speaking immigrants in Israel talk ("I need to renew my teudat zehut before we go to the makolet"). The output is English. Hebrew words are written in Latin script using one canonical romanization. It does not transcribe Hebrew.

v1 (2025-11-17) is kept at the v1 tag.

What changed from v1

	v1	v2
Data	516 desktop clips (~0.4 h)	`Read-Aloud-Voice-Sentences`, mostly phone takes, 1,929 training rows / 215 min
Romanization	ad hoc	canonical spellings, one per term
English share	none	29% monolingual English (own takes plus public replay), to avoid forgetting English
Test split	same sessions as training	term-disjoint: the Hebrew terms in the test set never appear in training

Results (term-disjoint test set, greedy decoding)

Scored on phone takes only, excluding the 30 test rows that were imported from v1's training data. v1 has 0.0% WER on those rows because it was trained on them.

Metric	Stock turbo	v1	v2
Code-switching WER (raw), 125 utterances	14.7%	12.1%	6.7%
Code-switching WER (normalized)	9.8%	7.1%	4.9%
Recall on unseen Hebrew terms (109 occurrences, 70 terms)	36.7%	48.6%	53.2%
Exact-case recall on unseen terms	19.3%	31.2%	44.0%
Recall on Hebrew terms also seen in training (112 occurrences)	43.8%	50.0%	76.8%
Own English WER (normalized), 33 utterances	4.0%	4.1%	2.9%
LibriSpeech-PC test-clean sample WER (normalized), 200 utterances	1.8%	2.2%	2.2%
Long-form WER (normalized), 6 utterances	15.4%	20.7%	9.2%

Normalized WER lowercases, turns hyphens into spaces and drops punctuation. Raw WER also counts casing and punctuation. The per-utterance hypotheses are in evals/.

Training

Full fine-tune, 10 epochs, learning rate 1e-5 (linear, 10% warmup), effective batch 32.
bf16 autocast with fp32 master weights. One A100-80GB on Modal, 25.5 minutes.
The best checkpoint was chosen by validation WER: step 427 of 610.
Target column: text (romanized). The dataset revision and full config are in training/config.json and training/result.json.

Usage

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="danielrosehill/Whisper-Hebrish")
print(asr("clip.wav", generate_kwargs={"language": "en", "task": "transcribe"})["text"])

Force language="en". With auto-detection Whisper may switch to Hebrew script.

whisper.cpp (GGML) / desktop

whisper.cpp builds are in ggml/, converted with whisper.cpp v1.9.5 convert-h5-to-ggml.py (128 mel bins, 32 encoder / 4 decoder layers):

File	Size
`ggml-whisper-hebrish-v2-large-v3-turbo-f16.bin`	1.62 GB
`ggml-whisper-hebrish-v2-large-v3-turbo-q8_0.bin` (recommended)	874 MB
`ggml-whisper-hebrish-v2-large-v3-turbo-q5_0.bin`	574 MB

On 10 phone clips from the term-disjoint test set, all three gave output identical to the PyTorch model with greedy decoding (ggml/verify.json).

ffmpeg -i input.m4a -ar 16000 -ac 1 -c:a pcm_s16le clip.wav
whisper-cli -m ggml-whisper-hebrish-v2-large-v3-turbo-q8_0.bin -l en -f clip.wav

Always pass -l en. For AMD GPUs build whisper.cpp with -DGGML_HIP=ON -DGPU_TARGETS=<your gfx> (ROCm) or -DGGML_VULKAN=ON.

Limitations

It was trained on one speaker (an Irish-born English speaker in Israel), so other accents are untested.
It writes only the canonical romanization of each term. Other common spellings (e.g. "teudat zeut") are counted as errors.
The test set is small: 125 code-switching utterances.