README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- image-to-text
base_model:
- kha-white/manga-ocr-base
---
# manga-ocr — ExecuTorch (Japanese text out of a picture of Japanese text)

Hand one line of Japanese to this and get the characters back. Vertical or horizontal,
furigana and all, printed or laid over artwork. It **generates** the characters rather
than classifying them: a ViT-base encoder reads the crop once, and a two-layer BERT
decoder emits one character at a time until it stops.

The shelf's other OCR is docTR, which reads Latin script in two stages — find the boxes,
then read each crop. This one is the second stage for Japanese, and it is the only shelf
model that reads vertical text.

```
manga_ocr_encoder  pixel_values (1, 3, 224, 224) fp32
    -> encoder_hidden (1, 197, 768) fp32
manga_ocr_decoder  input_ids (1, L) int64, encoder_hidden (1, 197, 768) fp32
    -> logits (1, 6144) fp32      # the next character, at the last position only
```

- **Source**: [kha-white/manga-ocr-base](https://huggingface.co/kha-white/manga-ocr-base),
  trained on Manga109s — 111.0M parameters, 6,144-character vocabulary
- **License**: Apache-2.0
- **Reference implementation**: [kha-white/manga_ocr](https://github.com/kha-white/manga_ocr)

## Files

Pick one row. The two halves must match: the encoder's hidden state is what the decoder
was measured against.

| build | encoder | decoder | encoder speed | decoder speed |
|---|---|---|---|---|
| **Core ML + XNNPACK int8** | `manga_ocr_encoder_coreml_all.pte` **172.1 MB** | `manga_ocr_decoder_xnnpack_int8.pte` **45.0 MB** | **4.2 ms** | 2.5 ms |
| XNNPACK int8 | `manga_ocr_encoder_xnnpack_int8.pte` 88.9 MB | `manga_ocr_decoder_xnnpack_int8.pte` 45.0 MB | 25.5 ms | 2.5 ms |
| XNNPACK fp32 | `manga_ocr_encoder_xnnpack_fp32.pte` 343.3 MB | `manga_ocr_decoder_xnnpack_fp32.pte` 117.4 MB | 29.5 ms | 3.2 ms |
| XNNPACK fp16 | `manga_ocr_encoder_xnnpack_fp16.pte` 173.1 MB | `manga_ocr_decoder_xnnpack_fp16.pte` 58.8 MB | 71.2 ms | 3.1 ms |

Host numbers, on a loaded machine, decoder timed at a 3-character prefix — read them as a
lower bound and as ratios rather than as device latency. Eager fp32 on the same host is
29.9 ms for the encoder and 4.6 ms for the decoder. XNNPACK takes 82.9% of the encoder
and 51.9% of the decoder; Core ML takes all of the encoder.

**fp16 is the row nothing should pick.** Its encoder is the size of the Core ML build and
less than half the speed of fp32, because XNNPACK inserts a cast at every boundary it
cannot fuse. It is listed because it was built and measured, not because it wins anything.

**int8 is the row to pick on Android**: half of fp16 on the encoder and three-quarters of
it on the decoder, **faster than eager PyTorch on both**, and no character different from
eager anywhere it was measured. What that costs is margin, not characters — see below.

## Running it

**1. Preprocess — the checkpoint's recipe, not PIL's defaults.**

```python
image = image.convert("L").convert("RGB").resize((224, 224), processor.resample)
mean = np.array(processor.image_mean, dtype=np.float32)   # 0.5, 0.5, 0.5
std = np.array(processor.image_std, dtype=np.float32)     # 0.5, 0.5, 0.5
x = (np.asarray(image, dtype=np.float32) / 255.0 - mean) / std
pixel_values = torch.from_numpy(np.ascontiguousarray(x.transpose(2, 0, 1)))[None]
```

Three details that are each load-bearing. **Greyscale and back to RGB** — the reference
pipeline does this, so a colour page and a scan of the same panel give the same tensor.
**`resample` from the config, which is bilinear** — `Image.resize` defaults to bicubic,
and a filter the model was not trained under changes what it reads. **The aspect ratio is
squashed to a square on purpose**: 224x224 regardless of whether the line is a wide
horizontal strip or a tall vertical one, which is what the model saw in training.

**2. The encoder, once per crop. The decoder, once per character.**

```python
hidden = encoder.execute([pixel_values])[0]
ids = torch.tensor([[2]])                  # decoder_start_token_id
for _ in range(300):
    token = int(decoder.execute([ids, hidden])[0][0].argmax())
    if token == 3:                         # eos_token_id
        break
    ids = torch.cat([ids, torch.tensor([[token]])], dim=1)
```

**3. Decode with `vocab.txt` — there is no tokenizer to install.** The checkpoint names
`BertJapaneseTokenizer`, whose MeCab step exists only to split *text* into words, and this
model never reads text. `subword_tokenizer_type` is `character` and the vocabulary holds
no `##` continuations, so a character is a line of `vocab.txt` and decoding is a lookup
and a join:

```python
vocab = open("vocab.txt", encoding="utf-8").read().splitlines()
text = "".join(vocab[i] for i in ids[0, 1:].tolist()
               if vocab[i] not in {"[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"})
```

Worth knowing because transformers 5.x cannot instantiate that tokenizer class at all —
it has dropped the slow tokenizers. Reading the vocabulary directly is not a workaround,
it is what an app on the device does anyway.

## Why there is no KV cache

The decoder is **two** layers over a 6,144-entry vocabulary. Re-running the whole prefix
each step is cheap enough that a cache would buy less than it costs: it would turn one
method with a dynamic length into a cache protocol the caller has to hold, allocate and
reset, and it would pin a maximum length into the file.

The decoder also **returns only the last row of logits**. The other L-1 rows are the model
re-deriving characters the caller already has; dropping them keeps 6,144 floats crossing
the boundary per step instead of L times that.

## Greedy, not beam search

The checkpoint's own generation config asks for 4 beams. Greedy is what the graph is
shaped for and what every number here was measured with; on the horizontal gate image
greedy and beam-4 read the same string, and the 84-line set never needed more than greedy
to match eager. Beam search remains possible from outside — the decoder is a pure function
of `(prefix, hidden)`, so a caller can run it once per beam per step — but what it buys on
harder input than this has not been measured.

## Core ML: the encoder yes, the decoder no

**The encoder is the whole speed story**: 100% delegated, 4.2 ms against 29.9 ms for eager
on the same host, at 172.1 MB. Worst-output correlation against fp32 eager is 0.999984.

**The decoder is not shipped for Core ML, and the reason is specific.** It converts, it
delegates 100%, and — once eight zero-element constants that transformers' BERT attention
leaves behind are pruned — it loads. What it will not do is run at a length other than the
one it was exported at: `execute() failed with error 0x32`. A decoder whose prefix grows by
one character per step is exactly the case that needs the dynamic axis, so the decoder
stays on XNNPACK, where it runs at every length from 1 to 300. This is not "Core ML rejects
dynamic shapes" — it accepts the export and pins the shape.

## What was measured

The gate is **the string that came out**, not correlation. The encoder's output is a hidden
state nobody reads, and the decoder's output is one row of logits whose argmax is all that
survives — a correlation of 0.999 and a different character are the same file.

**84 generated lines**: 12 sentences x 4 typefaces (Hiragino gothic W3 and W6, rounded
gothic, mincho) x both orientations x 7 degradations, from clean through gaussian blur,
sensor noise, low contrast, heavy smear, heavy grain, and a near-contrastless JPEG at
quality 12.

The set is generated deterministically, so the numbers below reproduce run to run.

| build | lines identical to eager | mean CER vs eager |
|---|---|---|
| XNNPACK fp32 | 84 / 84 | 0.0000 |
| XNNPACK fp16 | 84 / 84 | 0.0000 |
| XNNPACK int8 | 84 / 84 | 0.0000 |

Eager itself gets 82 of 84 exactly right (CER 0.0022 against the rendered truth), so the
model is not perfect on this set and every build reproduces it character for character
anyway — its two mistakes included.

### How close it came

84 lines agreeing is 84 samples. What generalises is how much room each decision had. A
character changes only if a build moves the winning logit past its best rival, so this is
measured at **every one of the 1,008 decoding steps**: the winner's lead in the build's
own logits, as a fraction of the lead the fp32 reference left it. 1.0 is untouched, 0.0
is a tie, below 0.0 is a changed character.

| build | median | worst step | steps changed |
|---|---|---|---|
| XNNPACK fp32 | 1.0000 | 1.0000 | 0 / 1008 |
| XNNPACK fp16 | 0.9999 | 0.9820 | 0 / 1008 |
| XNNPACK int8 | 0.9999 | **0.6513** | 0 / 1008 |

**int8's worst moment still kept 65% of the margin**, with the median step untouched at
0.9999. That is the number behind the recommendation, not the 84 identical strings.

The same steps also carry a worst-case bound — the reference gap over twice the largest
error anywhere in the 6,144 logits, which assumes that error landed exactly where it would
do the most damage. Its minimum is **1.16 for int8**, above 1.0, so on this set a changed
character is not merely absent but arithmetically impossible. Both numbers are printed by
`convert/check_manga_ocr.py`.

### What a whole line costs

The decoder re-reads the prefix each step, so its cost grows with the characters already
emitted. Measured per call, median of 20:

| prefix | fp32 | fp16 | int8 |
|---|---|---|---|
| 1 | 2.36 ms | 2.27 ms | 2.18 ms |
| 8 | 3.37 ms | 3.45 ms | 2.90 ms |
| 32 | 3.94 ms | 4.88 ms | 4.26 ms |
| 64 | 5.31 ms | 6.94 ms | 4.99 ms |

**Sixty-four times the prefix costs 2.3 times the step**, which is what makes the missing
cache a fair trade. Summed over the prefixes a real line walks through, the decoder side of
a **12-character line is 33 ms on int8** (39 ms fp32); add one encoder run — 4.2 ms on
Core ML, 25.5 ms on XNNPACK int8 — for the whole read.

### What this does not say

- **These are rendered lines, not photographed manga.** The set varies what a renderer can
  vary; it has no screentone, no handwriting, no speech-bubble clutter, no page curl. Read
  it as how far a build has drifted from eager, not as an accuracy score for the model.
- **The degradations never broke eager.** Its two mistakes are both on the clean and
  lightly-noised rows; heavy smear, heavy grain and the quality-12 JPEG produced none. So
  the set never reached a regime where the model itself starts failing, and cannot say
  what the builds do there.
- **Nothing here ran on a phone.** Every number is host CPU, and the machine was busy.

## Source

Converted with [executorch-convert](https://github.com/john-rocky) — `convert/export_manga_ocr.py`
for the two files, `convert/check_manga_ocr.py` for everything measured above.
ExecuTorch 1.4.0, PyTorch 2.13.0.