mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT

EmbeddingGemma 2 Transcribe — LiteRT

The audio encoder of EmbeddingGemma 2 and a small trained reader as LiteRT files, for English speech to text on an Android phone: the encoder, kept frozen, turns every 40 ms of audio into a vector, and the reader reads each vector as one of 200,000 English words or a blank. Audio, model and transcript stay on the phone.

The Android app that runs these files is eg2-transcribe on GitHub (android/). The reader's weights, training code and evaluation are in the companion repository mlboydaisuke/EmbeddingGemma-2-Transcribe-Reader on Hugging Face.

The idea comes from Shahir's post (@TslShahir, 2026-10-13) and the Embedding Gemma Transcribe Mac app. This is an independent project, not affiliated with Google.

Files

File	Bytes	SHA-256	What
eg2_audio_f16safe_fp16w_w30.tflite	589,273,696	f0f6c37ee62b51372ab28013617aad53793a84aaddd96e1e844d0cea0eb2f11d	the encoder for 30 s windows (signature audio_30s); the CPU and the GPU run this one
eg2_audio_f16safe_fp16w_w10.tflite	589,015,376	1405986b353c3152e4a677d9446a5694a0f8fd0b13b794d26c892ef85295188c	the encoder for 10 s windows (audio_10s); the NPU runs this one
reader_w30.tflite	386,726,384	129d6aa2d829dc4418675d01659ccc6fc42e33b84c2486f36faa9407d45c8716	reader v2, the word table and the choice of the best word in one graph, 750 tokens (reader_30s)
reader_w10.tflite	386,470,384	128119442ca776deb5499f20189eac86c05060bd774e14fcb93d8bfd9e8c97fb	the same for 250 tokens (reader_10s)
vocab.txt	1,737,588	bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68	the 200,000 words, one per line; the line number is the word index

On the phone. The app reads the two encoder files from its files directory and reader_w30.tflite, reader_w10.tflite and vocab.txt from the reader directory inside it; the app's README gives the adb commands, and the way to pick an encoder file on the phone instead.

Encoder. The audio tower and embed_audio of google/embeddinggemma-2 (revision 914f7f89), exported with litert-torch, one window length per file. Inputs: mel [1, F, 128], the log-mel spectrogram of transformers' Gemma4AudioFeatureExtractor (16 kHz audio, a 20 ms frame every 10 ms, 128 mel bins), and mask [1, F], 1 for a real frame; F = 3,000 for 30 s, 1,000 for 10 s. Outputs: soft [1, F / 4, 512], one vector per 40 ms, and mask [1, F / 4]. Every RMSNorm is rescaled so that its sums stay inside the float16 range, which the GPU at float16 and the NPU need; in float32 the rescaled graph gives the same numbers bit for bit. The weights of the fully connected and convolution layers are stored as float16 and turned back into float32 inside the graph, so a CPU, or a GPU at fp32 precision, computes in float32 with rounded weights.

Reader. Reader v2 (a 6-layer transformer of 19.6 M parameters trained with CTC on LibriSpeech's three training sets, 960 h; 890 h in training after leaving out the utterances with a word outside the vocabulary), the word table (the EmbeddingGemma 2 text embedding of each of the 200,000 words, float16 as released) and an argmax, in one graph. Inputs: soft [1, T, 512] and mask [1, T], with T = 750 or 250. Outputs per token: ids (int32, the best word), best (its score, 216.351 × the cosine to the word's text embedding) and blank (the blank's score). A token is its best word unless the blank scores higher, a run of the same word is one word, and blanks are dropped. The reader itself is float32; the table is used in float32 from its float16 values. Reader v2 keeps reader v1's inputs, outputs and file names, so the app runs either; reader v1 (4 layers, 13.3 M parameters, LibriSpeech train-clean-100) is this repository at revision 54d0e902.

Windows

The encoder runs on fixed windows and sees nothing before a window starts, so the tokens at the start of a window come out wrong. The windows follow the reader's contract: 30 s, 96 tokens (3.84 s) of overlap on the left, 1 extra token on the right.

• Window k starts at token 653 k (750 − 96 − 1). Window 0 keeps its tokens 0 to 748, every later window its tokens 96 to 748, and the window that reaches the end of the recording keeps through its last token.
• The log-mel is computed on the whole recording and each window is a slice of its frames. A window cut out of exactly 30 s of audio (480,000 samples) would give 2,999 frames, and its last token would miss its last frame.
• The reader reads the 750 tokens of a window as one block, both ways in time; the kept tokens of all windows make the transcript.

The encoder does 750 / 653 = 1.149× the work of one pass. The 10 s windows of the NPU keep 153 new tokens of 250. With these windows the app's word or blank equals the PyTorch reader's on all 81,436 tokens of the 54-minute recording below.

Performance

The Android app (debug build, LiteRT 2.2.0 CompiledModel) on a 54 min 17 s LibriVox recording (The Adventure of the Speckled Band, read by Kirsten Ferreri), in 125 windows of 30 s:

Phone	Encoder and reader on	Encoder per 30 s window	Reader per 30 s window	54 min 17 s of audio	Speed	WER
Pixel 8a (Tensor G3)	GPU fp32	1,510 ms	989 ms	300.8 s	10.8× real time	3.812 %
Pixel 8a (Tensor G3), an 8.1 s recording: auto window, 10 s files	GPU fp32	426.5 ms (one 10 s window)	218.9 ms (one 10 s window)	0.70 s for the 8.1 s	11.6× real time	— (words equal to the reader's release output)
Galaxy S26 (SM8850)	GPU fp32	318 ms	787 ms	140.8 s	23.1× real time	3.812 %
Galaxy S26 (SM8850), an 8.1 s recording: auto window, 10 s files	GPU fp32	104.1 ms (one 10 s window)	100.1 ms (one 10 s window)	0.26 s for the 8.1 s	31.4× real time	— (words equal to the reader's release output)

Pixel 8a: medians over the 125 windows; thermal status 2 from the second minute to the end of the run. The time covers log-mel, encoder, reader and words; reading the FLAC file came on top (5.4 s, the app's own FLAC reader). The app peaked at 5,267,760 kB of memory (VmHWM). The WER counts 377 errors in the 9,889 words of the book text, after lower-casing and collapsing spaces; the PyTorch reader on the same windows gives the same 3.812 %. The recording starts and ends with the LibriVox intro and outro, which the book text does not have, and a reader of an audiobook can depart from the printed words, so a WER on this recording is indicative only. The 8.1 s row is the app's bundled example (LibriSpeech test-clean 1284-1180-0000), which the app runs on the two 10 s files (a recording of up to 9.96 s): one of the 20 test-clean utterances of the accuracy check below, thermal status 1 at the end of that run. With reader v1 (this repository at revision 54d0e902) and an earlier build of the app, the same phone gave 1,401 ms and 839 ms per window, 276.8 s for the recording and a WER of 5.784 %.

Galaxy S26: medians over the 125 windows, in the same build of the app with the same five files (each file's SHA-256 on the phone equal to the table above). The run started on a rested phone (thermal status 0, battery 35.9 °C after a 9-minute wait) and the phone throttled as it ran: thermal status 1 at 25 s and 2 from 45 s to the end, the GPU's clock limit from 1,300 MHz down to 826 MHz and the fastest CPU cores from 4.7 GHz to 1.4 GHz by the end. Reading the FLAC file came on top (2.8 s). The app peaked at 4,472,048 kB (VmHWM). The WER is the same 377 errors, with the PyTorch reader's word or blank on all 81,436 tokens. On this GPU the reader's 30 s window costs 787 ms against 100 ms for its 10 s window (the 8.1 s row), so the app's choice of 30 s windows for a long recording is not the faster one on this phone. The 8.1 s row is the bundled example through the app's Try button, one run on a rested phone.

The encoder alone on the Galaxy S26, measured with an earlier build of the app, before the reader was in (2026-10-13); one call writes the window, runs the encoder and reads the result:

Encoder on	Window	Time per window	Conditions
GPU fp32	30 s	290.3 ms	one window, thermal status 1; over the 54 minutes (thermal status 3 at the end) the median was 572.7 ms
CPU, 4 threads	30 s	558.1 ms	one window, thermal status 1
NPU (the Hexagon HTP, through LiteRT's Qualcomm JIT compiler)	10 s	94.7 ms	median of 5 windows, thermal status 2; the whole graph runs on the HTP; compiling on the phone took 121.6 s once, later starts read the compiled graph from the app's cache

The NPU runs, but it computes in float16 and misses the strict per-token check at quiet moments: over the full 54-minute recording, 37 of its 81,434 tokens fell under a cosine of 0.99 against the float32 encoder run on the whole recording at once (0.45 at the lowest), none of them digital silence. It is kept in the app as an option; GPU fp32 is the default, and the NPU has not been measured with the reader.

Accuracy

• Reader v2 alone (its release notes; float32 reference soft tokens, greedy decoding, no language model): WER 4.21 % on LibriSpeech test-clean, 10.42 % on test-other (reader v1: 6.81 % and 15.10 %).
• These files in the app on the Pixel 8a: on 20 LibriSpeech test-clean utterances, GPU fp32 and CPU both give 20 of 20 transcripts equal to the reader's outputs listed in its release notes, with the same word or blank as the PyTorch reader on all 5,036 tokens; on the 54-minute recording, the WER above, with the same word or blank as the PyTorch reader on all 81,436 tokens.

Known weaknesses

• Words that sound alike: 12.9 % of the reader's substitutions on test-clean are homophones (one pronunciation in the lexicon, such as "o" and "oh" or "meter" and "metre"), which a reader without a language model cannot tell apart. The most frequent substitutions are short function words: "and" and "in" for each other, "a" and "the" for each other, "an" read as "and".
• Long words: words of 6 or more tokens (240 ms or longer) are read wrong more often than short ones, 6.1 % against 1.8 % for words of 1 or 2 tokens (2,518 words of 100 test-clean utterances, word spans from a wav2vec 2.0 forced alignment).
• No punctuation and no capital letters; trained on read English audiobook speech (LibriSpeech) only.
• float16 arithmetic (the GPU's default precision, the NPU) moves some quiet tokens away from the float32 result; the app runs GPU fp32 unless another backend is picked.
• Memory: with the encoder and the reader both on the GPU the app peaked at 5,267,760 kB on the 54-minute recording and 5,142,000 kB on the 20 utterances (VmHWM, Pixel 8a, 8 GB); on the Galaxy S26 (12 GB) 4,472,048 kB and 4,343,156 kB. While the app loaded its models for the long recording, the phone's available memory (MemAvailable) fell to 873,848 kB; once, right after a long run on the CPU, it fell under 800,000 kB, where the device check stopped the app.

Conversion

The encoder files come from google/embeddinggemma-2 at revision 914f7f89142e33e77833254d9c9b90c3cef7303b; the reader files from reader v2 of mlboydaisuke/EmbeddingGemma-2-Transcribe-Reader at revision de224fd8fa4ed4519fdddd9aa4154432d46a1bd3 (reader.pt with SHA-256 d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2), with the reader's model code from eg2-transcribe at commit 6bb4b7f. Both were converted with litert-torch 0.10.0 (main of google-ai-edge/litert-torch at 0a81bf6, installed editable: the released 0.9.4 lacks the Gemma 4 audio classes), transformers 5.19.0 and torch 2.12.0, and the float16 weights were stored with ai-edge-quantizer-nightly 0.10.0.dev20261013. The scripts and every step are in hf-to-litertlm scripts/eg2_transcribe/; one command per kind of file:

python scripts/eg2_transcribe/export_encoder.py export --model hf/embeddinggemma-2 --fixtures eg2-transcribe/reader/fixtures --out out/eg2
python scripts/eg2_transcribe/export_reader.py export --reader-dir hf/reader --reader-code eg2-transcribe/reader/train --fixtures eg2-transcribe/reader/fixtures --out out/eg2

On 2026-10-15 a rebuild of the two encoder files with these scripts, from the sources and versions above, gave them byte for byte, and the two reader files are the reader command's output from reader v2 the same day; the SHA-256 of all four are the ones in the Files table.

License and credits

• These files: Apache-2.0 (LICENSE, NOTICE); the app's code is Apache-2.0 as well.
• EmbeddingGemma 2 (Google DeepMind): Apache-2.0; use must follow the Gemma Prohibited Use Policy.
• LibriSpeech (the reader's training data and the test utterances): V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0.
• LibriSpeech lexicon (the words of vocab.txt): https://www.openslr.org/11, public domain.
• The 54-minute recording: LibriVox, The Adventures of Sherlock Holmes, section 8, read by Kirsten Ferreri (https://archive.org/details/adventures_holmes), public domain; its reference text is from Project Gutenberg eBook #1661 (public domain in the USA). Neither is in this repository.