validation_report.md

# Tokenizer Validation Report

Model: `geez_en_shared.model`, vocab size: 120000

## Per-language metrics (sampled)

| Language | Lines sampled | Fertility (tok/word) | Compression (chars/tok) | UNK rate | Roundtrip failure rate |
|---|---|---|---|---|---|
| amharic | 2000 | 1.389 | 3.594 | 0.0 | 0.009 |
| english | 2000 | 1.237 | 5.0 | 0.0 | 0.022 |
| geez | 1813 | 1.2 | 2.833 | 0.0 | 0.0 |
| tigre | 2000 | 1.285 | 3.181 | 0.0 | 0.0 |
| tigrinya | 2000 | 1.376 | 3.446 | 0.0 | 0.008 |

**Note on roundtrip failure rate:** these are not reconstruction bugs. Spot-checking
actual mismatches shows they're SentencePiece's default normalizer doing its job —
e.g. a real ellipsis character `…` gets normalized to three dots `...`, and a
leading BOM (``) gets stripped. `decode(encode(x))` reflects the *normalized*
form, not raw input bytes; this is standard SentencePiece behavior, not data loss.

## Morphologically hard words (from the MoVoC paper)


**amharic**

- `አልሰበሩም` -> ['▁አል', 'ሰበሩ', 'ም'] (3 pieces, roundtrip OK)
- `የማይሰብረው` -> ['▁የማይ', 'ሰብ', 'ረው'] (3 pieces, roundtrip OK)
- `መምህርነት` -> ['▁መምህር', 'ነት'] (2 pieces, roundtrip OK)

**tigrinya**

- `ኣይትከውንን` -> ['▁ኣይት', 'ከውን', 'ን'] (3 pieces, roundtrip OK)
- `ምሕዳራት` -> ['▁ምሕዳ', 'ራት'] (2 pieces, roundtrip OK)

## Bilin (byn) byte-fallback check

- probe: `ብሊን ቋንቋ` -> 2 pieces, roundtrip OK
- No Bilin corpus exists to train on (see SOURCES.md) — this only confirms byte_fallback keeps unseen Ge'ez-script text lossless, not that Bilin gets meaningful subword coverage.