mlboydaisuke/EmbeddingGemma-2-Transcribe-Reader

EmbeddingGemma 2 Transcribe — reader v2

A 19.6 M-parameter reader that turns the audio soft tokens of Google's EmbeddingGemma 2 into English words: each 40 ms token is mapped into the model's 768-d text-embedding space and read as the closest of 200,000 words, or as blank, with greedy CTC decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained. Reader v2 is trained on the three LibriSpeech training sets (960 h): test-clean WER 4.21 %, test-other 10.42 %.

The idea comes from Shahir's post (2026-10-13) and the Embedding Gemma Transcribe Mac app (eg2.haku.sh), whose code and weights are not public. This reader is an independent reproduction, trained from scratch on LibriSpeech. It is not affiliated with Google.

Reader v1 (4 layers, 13.3 M parameters, trained on train-clean-100 alone; test-clean 6.81 %, test-other 15.10 %) is revision 1af9c5f5 of this repository.

These weights are framework-neutral (a PyTorch state dict and a NumPy table). The converted builds that run them on a device:

Build	Repository
Core AI (iOS 27 / macOS 27): the EmbeddingGemma 2 audio encoder and this reader with its word table, as .aimodel bundles	mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI
LiteRT (Android), with reader v1	mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT

Files

File	Bytes	SHA-256	What
reader.pt	78,323,637	d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2	the reader: a PyTorch state_dict, float32
word_table.npy	307,200,128	b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8	float16 [200000, 768], one L2-normalized row per word (the file of v1)
vocab.txt	1,737,588	bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68	the 200,000 words, one per line; the line number is the word index (the file of v1)
reader_config.json	2,080	3aede371f7fc1b69bd948a3fce6f861e44eedb237d7b6dce360063ea007a90ed	the contract for ports: input, output, logits, decoding, the window policy, the WER
RELEASE.md	15,575	9d0ac2a5d246d8b35f6456891e9a2ae2f199ef44a51087d1fa367f4cd1f35994	the release note: WER details, the reader's output on the 20 fixture utterances, training log, weaknesses

SHA256SUMS lists every file. RELEASE.md is the release note of the reader/ directory of the eg2-transcribe repository; the paths in it (train/, fixtures/) are relative to that directory.

Contract

• Input: EmbeddingGemma 2 audio soft tokens, float32 [T, 512], 25 per second: the embed_audio output of google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b (model.get_audio_features(input_features, input_features_mask).pooler_output; transformers 5.19.0 with the default attention, not eager, whose audio mask is inverted in that version).
• Body: LayerNorm → Linear 512→512, 6 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU, rotary positions with base 10000), LayerNorm → Linear 512→768. Attention covers the whole input window in both directions. 19,573,762 parameters.
• Output per token: a unit vector in the 768-d text space and a blank logit.
• Logits: word w scores scale × cos(vector, word_table[w]), with scale = exp(log_scale) = 216.351 (learned); blank is class 200,000 and scores blank_logit.
• Decoding: per token, blank where blank_logit > top-1 word logit (a tie reads the word), else the top-1 word; merge repeats, drop blanks. No language model, no punctuation, no casing.
• Long audio: 30 s encoder windows (750 tokens), each starting 96 tokens (3.84 s) before the end of the previous one, keeping only its new tokens and dropping its last token: the encoder is causal, so the start of a window lacks its left context and the last token of a 30 s cut misses one input frame. reader_config.json → window.
• Weights file: reader.pt keys body.* (the reader body), blank.weight [1, 768], blank.bias [1], log_scale; the module is Reader(arch, n_vocab, k, layers) of reader/train/model.py in the eg2-transcribe repository (arch a, layers 6; reader_config.json → architecture.layers).

WER

Greedy, no language model, every utterance of each set. Reference and hypothesis are lower-cased with runs of spaces collapsed; words outside the vocabulary stay in the references and count as errors.

Set	Utterances	Reference words	WER	Substitutions	Deletions	Insertions	v1 WER
dev-clean (model selection)	2,703	54,402	3.75 %	1,621	210	211	6.57 %
dev-other	2,864	50,948	10.26 %	4,109	679	441	14.93 %
test-clean	2,620	52,576	4.21 %	1,739	247	226	6.81 %
test-other	2,939	52,343	10.42 %	4,249	655	551	15.10 %
fixtures test-clean-20	20	541	3.70 %	18	1	1	4.81 %

Long audio, in the 30 s windows of the contract: The Adventure of the Speckled Band (LibriVox, 3,257.4 s, 125 windows), scored against the Project Gutenberg text (9,889 words; the LibriVox intro and outro are in the audio only): 3.81 % through the float32 PyTorch path, 3.80 % through the Core AI build (the encoder on the GPU and this reader in float16 on the Neural Engine of an Apple M4 Max, two windows in flight). Reader v1: 5.78 % and 5.80 %.

Training

• Data: LibriSpeech train-clean-100, train-clean-360 and train-other-500 (CC BY 4.0), 281,241 utterances, 961.6 h, read as one set per epoch. The 19,755 utterances with a word outside the vocabulary were left out: 261,486 utterances, 890.2 h, 80,121,094 tokens. Soft tokens computed once in float32 (TF32 off), stored as float16.
• dev-clean chose the reader body and the epoch; test-clean and test-other were decoded once, after training.
• Objective: CTC over the 200,000 words and blank (torch.nn.functional.ctc_loss, float32, zero_infinity, mean of loss / target length).
• Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, 500 warmup steps, then cosine to zero at step 163,430; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 weights and activations, the training matmuls in TF32 (A100); the dev, test and fixture WER are decoded in full float32.
• 9 epochs, 147,087 steps; the best dev-clean WER came at epoch 7 (these weights); early stop after 2 epochs without a better dev-clean WER.
• Hardware: one NVIDIA A100-SXM4-80GB (Google Colab), torch 2.11.0+cu130, CUDA 13.0; 4.38 h of training plus 2 min of dev decoding.
• Body chosen on dev-clean between two bodies trained the same way: 4 layers (13.3 M parameters, 3.88 % at its best epoch 8) and 6 layers (19.6 M, 3.75 % at epoch 7, these weights).

The command of the released run is in RELEASE.md.

Word table

• vocab.txt: every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by code point, apostrophes kept.
• word_table.npy: the EmbeddingGemma 2 text embedding of each word with the prompt task: search result | query: (sentence-transformers, float32, then stored as float16 and L2-normalized). Each of 100 random words, embedded again, comes back at rank 1. Both files are those of v1.
• Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %). These are always errors.

Limits

• By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 1.8 % for words of 1–2 tokens (up to 80 ms), 2.0 % for 3–5 tokens and 6.1 % for 6 or more.
• The most frequent substitutions on test-clean are function words (and → in 27, in → and 22, a → the 16); homophones are 12.9 % of all substitutions, which a reader without a language model cannot tell apart.
• English read speech only (audiobooks); trained on utterances of up to 29.7 s and measured on utterances of up to 35.2 s; longer audio goes through the 30 s windows above.

License and credits

• This repository: Apache-2.0 (LICENSE); see NOTICE.
• LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on train-clean-100, train-clean-360 and train-other-500.
• LibriSpeech lexicon: https://www.openslr.org/11, public domain.
• EmbeddingGemma 2 (google/embeddinggemma-2, Google DeepMind): Apache 2.0. It produces the soft tokens the reader reads and the word table. Deployments must adhere to the Gemma Prohibited Use Policy.