jgalamba/manga-ocr-kvcache-tflite

manga-ocr — KV-cache + fp16 (LiteRT/TFLite)

On-device manga text-line recognizer: a first-party conversion of kha-white/manga-ocr-base to a KV-cached decoder with fp16 weights, for LiteRT/TFLite. The KV-cache makes autoregressive decoding ~30x faster per token on-device than a no-cache re-run decoder, and fp16 roughly halves the size — at byte-identical output.

Provenance: A conversion of kha-white's Apache-2.0 weights, produced with ai-edge-torch / litert_torch. Conversion recipe: see scripts/phase6c_b3_conversion/ in the project repo.

Files

file	precision	size	purpose
encoder_int8.tflite	int8 (dyn-range)	84 MB	ViT-base encoder (recommended ship — smallest, lossless)
encoder_fp16.tflite	fp16	164 MB	ViT-base encoder (on-device-proven alternative)
decoder_cache_fp16.tflite	fp16	47 MB	KV-cache init/step decoder (ship)
encoder.tflite	fp32	328 MB	encoder reference
decoder_cache.tflite	fp32	94 MB	decoder reference
mocr2025_vocab.csv	—	tiny	char id→token map (shared with mocr_2025)

Ship config: encoder_int8.tflite + decoder_cache_fp16.tflite (131 MB) — the int8 dynamic-range encoder is lossless (88.4% / CER 0.116, identical to fp16) and ~2.2× faster than fp16 on host (935 vs 2036 ms/crop), with fp32 I/O so it drops in for the fp16 encoder; encoder_fp16.tflite stays as the on-device-proven alternative (int8 on-device latency/heap is the last gate). Or encoder_fp16.tflite + decoder_cache_fp16.tflite (211 MB). The cache decoder takes token IDs directly — no external embeddings file is needed (unlike the no-cache variant).

Architecture & I/O

• Encoder (signature serving_default): in args_0 [1,3,224,224] f32, NCHW RGB, normalized (x/255 − 0.5) / 0.5; out output_0 [1,197,768].
• Decoder — 2 layers × 12 heads × 64 dim, vocab 6144, BOS=2 / EOS=3 / PAD=0; two signatures:
  • init(args_0 enc[1,197,768] f32, args_1 startTok[1,1] i64) → output_0 logits [1,6144], output_1..2 selfK [1,12,1,64], output_3..4 selfV, output_5..6 crossK [1,12,197,64], output_7..8 crossV.
  • step(args_0 enc, args_1 tok[1,1] i64, args_2 pos[1] i64, args_3..6 selfK/V buffer[1,12,256,64], args_7..10 crossK/V[1,12,197,64]) → output_0 logits, output_1..2 newSelfK [1,12,1,64], output_3..4 newSelfV.

Decode loop

enc = encoder(img); init(enc, [[2]]) → logits0 + KV; copy the length-1 self-K/V into a 256-length buffer at index 0; cur = argmax(logits0). Then for pos = 1, 2, …: step(enc, cur, pos, selfK, selfV, crossK, crossV); write the returned length-1 self-K/V at buffer index pos; stop at EOS (or a maxNewTokens cap to bound runaways); emit ids ≥ 5 and detokenize via mocr2025_vocab.csv. Reference implementations: MangaOcrCacheRunner.java (Android) and b3_validate_cache.py (Python) in the conversion repo.

Validation

• Java/Python decode parity: 16/16 exact (CER 0.000) on-device (Samsung SM-A536U1).
• Quality (host, full tier-A, n=317): 88.4% char-acc / CER 0.116 — identical to the no-cache decode (the KV-cache is a pure compute optimization → lossless); fp16 == fp32 accuracy.
• On-device decode ~31 ms/token; the fixed ViT-base encoder (~1.3 s) is the bottleneck.
• Encoder int8 (dynamic-range): lossless (88.4% / CER 0.116) at 84 MB (from 164 MB fp16), ~2.2× faster than fp16 on host (935 vs 2036 ms/crop); on-device latency/heap pending.

License & attribution

Apache License 2.0 — see LICENSE and NOTICE. Derived from manga-ocr by Maciej Budyś (kha-white), Apache-2.0 (github.com/kha-white/manga-ocr). manga-ocr was trained on the Manga109-s dataset (academic use; http://www.manga109.org/en/) — acknowledged as a courtesy; these files do not contain or redistribute Manga109 data.