| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: automatic-speech-recognition |
| base_model: google/embeddinggemma-2 |
| base_model_relation: adapter |
| datasets: |
| - openslr/librispeech_asr |
| tags: [embeddinggemma, speech-recognition, ctc, word-embeddings, on-device, core-ai, litert] |
| --- |
| |
| # EmbeddingGemma 2 Transcribe — reader v2 |
|
|
| A 19.6 M-parameter reader that turns the audio soft tokens of Google's |
| [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) into English words: each 40 ms token is mapped |
| into the model's 768-d text-embedding space and read as the closest of 200,000 words, or as blank, with greedy CTC |
| decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained. Reader v2 is |
| trained on the three LibriSpeech training sets (960 h): test-clean WER 4.21 %, test-other 10.42 %. |
|
|
| The idea comes from Shahir's post (2026-10-13) and the *Embedding Gemma Transcribe* Mac app (eg2.haku.sh), whose |
| code and weights are not public. This reader is an independent reproduction, trained from scratch on LibriSpeech. It |
| is not affiliated with Google. |
|
|
| Reader v1 (4 layers, 13.3 M parameters, trained on train-clean-100 alone; test-clean 6.81 %, test-other 15.10 %) is |
| revision `1af9c5f5` of this repository. |
|
|
| These weights are framework-neutral (a PyTorch state dict and a NumPy table). The converted builds that run them on |
| a device: |
|
|
| | Build | Repository | |
| |---|---| |
| | Core AI (iOS 27 / macOS 27): the EmbeddingGemma 2 audio encoder and this reader with its word table, as `.aimodel` bundles | [mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI](https://huggingface.co/mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI) | |
| | LiteRT (Android), with reader v1 | [mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT](https://huggingface.co/mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT) | |
|
|
| ## Files |
|
|
| | File | Bytes | SHA-256 | What | |
| |---|---:|---|---| |
| | `reader.pt` | 78,323,637 | `d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2` | the reader: a PyTorch `state_dict`, float32 | |
| | `word_table.npy` | 307,200,128 | `b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8` | float16 `[200000, 768]`, one L2-normalized row per word (the file of v1) | |
| | `vocab.txt` | 1,737,588 | `bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68` | the 200,000 words, one per line; the line number is the word index (the file of v1) | |
| | `reader_config.json` | 2,080 | `3aede371f7fc1b69bd948a3fce6f861e44eedb237d7b6dce360063ea007a90ed` | the contract for ports: input, output, logits, decoding, the window policy, the WER | |
| | `RELEASE.md` | 15,575 | `9d0ac2a5d246d8b35f6456891e9a2ae2f199ef44a51087d1fa367f4cd1f35994` | the release note: WER details, the reader's output on the 20 fixture utterances, training log, weaknesses | |
|
|
| `SHA256SUMS` lists every file. `RELEASE.md` is the release note of the `reader/` directory of the [eg2-transcribe](https://github.com/john-rocky/eg2-transcribe) |
| repository; the paths in it (`train/`, `fixtures/`) are relative to that directory. |
|
|
| ## Contract |
|
|
| - **Input**: EmbeddingGemma 2 audio soft tokens, float32 `[T, 512]`, 25 per second: the `embed_audio` output of |
| `google/embeddinggemma-2` at revision `914f7f89142e33e77833254d9c9b90c3cef7303b` |
| (`model.get_audio_features(input_features, input_features_mask).pooler_output`; transformers 5.19.0 with the |
| default attention, not `eager`, whose audio mask is inverted in that version). |
| - **Body**: LayerNorm → Linear 512→512, 6 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU, |
| rotary positions with base 10000), LayerNorm → Linear 512→768. Attention covers the whole input window in both |
| directions. 19,573,762 parameters. |
| - **Output per token**: a unit vector in the 768-d text space and a blank logit. |
| - **Logits**: word `w` scores `scale × cos(vector, word_table[w])`, with `scale = exp(log_scale) = 216.351` |
| (learned); blank is class 200,000 and scores `blank_logit`. |
| - **Decoding**: per token, blank where `blank_logit > top-1 word logit` (a tie reads the word), else the top-1 word; |
| merge repeats, drop blanks. No language model, no punctuation, no casing. |
| - **Long audio**: 30 s encoder windows (750 tokens), each starting 96 tokens (3.84 s) before the end of the previous |
| one, keeping only its new tokens and dropping its last token: the encoder is causal, so the start of a window lacks |
| its left context and the last token of a 30 s cut misses one input frame. `reader_config.json` → `window`. |
| - **Weights file**: `reader.pt` keys `body.*` (the reader body), `blank.weight [1, 768]`, `blank.bias [1]`, |
| `log_scale`; the module is `Reader(arch, n_vocab, k, layers)` of `reader/train/model.py` in the eg2-transcribe |
| repository (arch `a`, `layers` 6; `reader_config.json` → `architecture.layers`). |
|
|
| ## WER |
|
|
| Greedy, no language model, every utterance of each set. Reference and hypothesis are lower-cased with runs of spaces |
| collapsed; words outside the vocabulary stay in the references and count as errors. |
|
|
| | Set | Utterances | Reference words | WER | Substitutions | Deletions | Insertions | v1 WER | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | dev-clean (model selection) | 2,703 | 54,402 | 3.75 % | 1,621 | 210 | 211 | 6.57 % | |
| | dev-other | 2,864 | 50,948 | 10.26 % | 4,109 | 679 | 441 | 14.93 % | |
| | test-clean | 2,620 | 52,576 | 4.21 % | 1,739 | 247 | 226 | 6.81 % | |
| | test-other | 2,939 | 52,343 | 10.42 % | 4,249 | 655 | 551 | 15.10 % | |
| | fixtures `test-clean-20` | 20 | 541 | 3.70 % | 18 | 1 | 1 | 4.81 % | |
|
|
| Long audio, in the 30 s windows of the contract: *The Adventure of the Speckled Band* (LibriVox, 3,257.4 s, 125 |
| windows), scored against the Project Gutenberg text (9,889 words; the LibriVox intro and outro are in the audio only): |
| 3.81 % through the float32 PyTorch path, 3.80 % through the Core AI build (the encoder on the GPU and this reader in |
| float16 on the Neural Engine of an Apple M4 Max, two windows in flight). Reader v1: 5.78 % and 5.80 %. |
|
|
| ## Training |
|
|
| - Data: LibriSpeech train-clean-100, train-clean-360 and train-other-500 (CC BY 4.0), 281,241 utterances, 961.6 h, |
| read as one set per epoch. The 19,755 utterances with a word outside the vocabulary were left out: 261,486 |
| utterances, 890.2 h, 80,121,094 tokens. Soft tokens computed once in float32 (TF32 off), stored as float16. |
| - dev-clean chose the reader body and the epoch; test-clean and test-other were decoded once, after training. |
| - Objective: CTC over the 200,000 words and blank (`torch.nn.functional.ctc_loss`, float32, `zero_infinity`, mean of |
| loss / target length). |
| - Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, 500 warmup steps, then cosine to zero at |
| step 163,430; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 weights and activations, |
| the training matmuls in TF32 (A100); the dev, test and fixture WER are decoded in full float32. |
| - 9 epochs, 147,087 steps; the best dev-clean WER came at epoch 7 (these weights); early stop after 2 epochs without |
| a better dev-clean WER. |
| - Hardware: one NVIDIA A100-SXM4-80GB (Google Colab), torch 2.11.0+cu130, CUDA 13.0; 4.38 h of training plus 2 min of |
| dev decoding. |
| - Body chosen on dev-clean between two bodies trained the same way: 4 layers (13.3 M parameters, 3.88 % at its best |
| epoch 8) and 6 layers (19.6 M, 3.75 % at epoch 7, these weights). |
|
|
| The command of the released run is in `RELEASE.md`. |
|
|
| ## Word table |
|
|
| - `vocab.txt`: every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by |
| code point, apostrophes kept. |
| - `word_table.npy`: the EmbeddingGemma 2 text embedding of each word with the prompt `task: search result | query: ` |
| (sentence-transformers, float32, then stored as float16 and L2-normalized). Each of 100 random words, embedded |
| again, comes back at rank 1. Both files are those of v1. |
| - Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %). These are always |
| errors. |
|
|
| ## Limits |
|
|
| - By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 1.8 % for words of 1–2 |
| tokens (up to 80 ms), 2.0 % for 3–5 tokens and 6.1 % for 6 or more. |
| - The most frequent substitutions on test-clean are function words (and → in 27, in → and 22, a → the 16); homophones |
| are 12.9 % of all substitutions, which a reader without a language model cannot tell apart. |
| - English read speech only (audiobooks); trained on utterances of up to 29.7 s and measured on utterances of up to |
| 35.2 s; longer audio goes through the 30 s windows above. |
|
|
| ## License and credits |
|
|
| - This repository: Apache-2.0 (`LICENSE`); see `NOTICE`. |
| - LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain |
| audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on |
| train-clean-100, train-clean-360 and train-other-500. |
| - LibriSpeech lexicon: https://www.openslr.org/11, public domain. |
| - EmbeddingGemma 2 (`google/embeddinggemma-2`, Google DeepMind): Apache 2.0. It produces the soft tokens the reader |
| reads and the word table. Deployments must adhere to the |
| [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). |
|
|
