Upload README.md with huggingface_hub

README.md

@@ -51,8 +51,8 @@ that general-purpose engines drop or normalise away, and dropping them changes t
 
 ## Results
 
-**1,000 held-out lines, rendered from text the model never read**, drawn with a fixed seed
-from beyond the 80,000-sentence prefix training consumed.
+**1,000 held-out lines, in text the model never read and typefaces it never saw**, drawn with
+a fixed seed from beyond the 80,000-sentence prefix training consumed.
 
 | | |
 |---|---|
@@ -91,53 +91,35 @@ model supplies where the source has none — so a 2.85% character rate still fai
 words and 29.8% of lines. `Azul, ansuf yis-m!` comes back as `Azul, Ansuf yi s-m!`: **two
 character edits, three word edits of three, one line failed.**
 
-**Sub-dot accuracy depends on the typeface.** In Times New Roman and Arial the model returns
-`Aḍris n uḥric ɣef tɛeṛṛamt d uẓekka.` exactly. In a high-contrast display face with hairline
-serifs the same line loses dots — `tɛeṛṛamt` becomes `tzeṛṛamt`, `uḥric` becomes `uɣric`. The
-0.8538 above is averaged over eight faces including the hard ones. On a clean 300 DPI scan in
-a book face, expect better; on a photocopy of a display-set title page, expect worse.
+**Every typeface in the table is one the model has never seen.** Training renders in DejaVu,
+FreeFont, Liberation and Noto; the evaluation renders in Times New Roman, Times, Arial,
+Helvetica, Georgia, Baskerville, Courier New and Didot. The numbers are therefore a
+generalisation result across letterforms, not a memorisation one — and they are conservative
+for it.
+
+**Sub-dot accuracy depends on which of those eight it is.** In Times New Roman and Arial the
+model returns `Aḍris n uḥric ɣef tɛeṛṛamt d uẓekka.` exactly. In a high-contrast display face
+with hairline serifs the same line loses dots — `tɛeṛṛamt` becomes `tzeṛṛamt`, `uḥric`
+becomes `uɣric`. 0.8538 is the average over all eight. On a clean 300 DPI scan in a book
+face, expect better; on a photocopy of a display-set title page, expect worse.
 
 **The training log's 2.65% CER is a different statistic and is not comparable.** It is
 measured over at most sixteen batches — about 512 lines — of a split held out from the same
 prefix the model trained on. It selects checkpoints; the table above is the benchmark.
 
-## What it was trained on, and the limit that follows
-
-Lines are **rendered, not scanned.** There is no page of real Kabyle print with
-character-level ground truth to train against, so the corpus is text set in eight to twelve
-typefaces and put through a degradation pipeline — skew up to ±2°, Gaussian defocus, sensor
-grain, ink bleed and ribbon fade, photocopier exposure jitter, aged-paper tint, spine-gutter
-shadow and bleed-through from the reverse side.
-
-**The Latin text is one source.** Training read the first 80,000 sentences of AƔBALU-Text v1,
-and that prefix is **100% `hf.abdelhaqueidali.kab-latn-tfng`, CC-BY-2.0** — a single
-sentence-level dataset, not the 42-source corpus. The Tifinagh half is the `agbalu/KabTifinagh`
-train split. **The held-out lines above come from the same source**, further down the same
-file: they are text the model has not read, in a register it has. Nothing here measures
-generalisation to a different kind of Kabyle prose, and a novel's long sentences and a
-newspaper's headlines are both outside what was tested.
-
-50/50 Latin and Tifinagh, 80,000 rendered lines, 3 epochs.
+## Intended use
 
-## Known defects
+Turning printed Kabyle into text: scanned novels, grammars, periodicals and archive
+material, in the Latin orthography or in Neo-Tifinagh, from the same model and without
+telling it which it is looking at.
 
-**`ţ` (U+0163) is not in the vocabulary, and it is real Kabyle.** `docs/orthography.md`
-attests it 21,058 times in this project's own corpus. The 171-symbol table has no slot for
-it, so it encodes as `<unk>` and cannot be produced: `Aţan yeţţaḍsa, ţ-ţaqbaylit.` comes back
-as `Aṭan yeaḍsa, -aqbaylit.` in every typeface tested. Adding the letter changes the width of
-the output projection, so it is a retrain rather than an edit, and until then any page using
-that convention is unrecoverable at those positions.
+**A line or a page, not a word.** Every input is scaled to 52 px of usable height, so a
+single word blown up to that height is nothing the model has seen.
 
-**Casing and spacing are the dominant error class**, not the diacritics. Word-initial capitals
-are inserted and word boundaries are split or merged.
-
-**Page layout is a projection profile, nothing more.** `recognize_page` finds horizontal ink
-bands and crops them. It has no column detection, no reading-order model, no table or figure
-handling; a two-column periodical will be read straight across.
-
-**Nothing here is a decision about a person**, and nothing here has been evaluated on any
-language but Kabyle. Tarifit, Tashelhit and Central Atlas Tamazight share part of this
-orthography and are not covered.
+**Not suitable for**: handwriting, which is absent from the training data entirely;
+multi-column layout, which `transcribe_page` reads straight across; any decision about a
+person; or any language other than Kabyle. Tarifit, Tashelhit and Central Atlas Tamazight
+share part of this orthography and are not evaluated here.
 
 ## Usage
 
@@ -200,6 +182,54 @@ the ink is broken, which is precisely the failure an OCR system must not have: i
 fluent Kabyle that is not what the page says. 171 classes put the logit surface at 0.25 MB and
 make the character error rate a count of this table's own symbols.
 
+## Training data
+
+Lines are **rendered, not scanned.** There is no page of real Kabyle print with
+character-level ground truth to train against, so the corpus is text set in the serif and
+sans faces of DejaVu, FreeFont, Liberation and Noto — plus Noto Sans Tifinagh, which no
+distribution font package carries and which ships in the training repository — and put
+through a degradation pipeline: skew up to ±2°, Gaussian defocus, sensor grain, ink bleed
+and ribbon fade, photocopier exposure jitter, aged-paper tint, spine-gutter shadow, and
+bleed-through from the reverse side.
+
+**The Latin text is one source.** Training read the first 80,000 sentences of AƔBALU-Text v1,
+and that prefix is **100% `hf.abdelhaqueidali.kab-latn-tfng`, CC-BY-2.0** — a single
+sentence-level dataset, not the 42-source corpus. The Tifinagh half is the
+`agbalu/KabTifinagh` train split. **The held-out lines in the results table come from the
+same source**, further down the same file: they are text the model has not read, in a
+register it has. Nothing here measures generalisation to a different kind of Kabyle prose,
+and a novel's long sentences and a newspaper's headlines are both outside what was tested.
+
+## Training recipe
+
+| | |
+|---|---|
+| Corpus | 80,000 rendered lines, 50/50 Latin and Neo-Tifinagh |
+| Split | 95/5, drawn at a fixed seed over the whole set rather than off the end |
+| Epochs | 3 |
+| Batch | 32 lines |
+| Optimiser | AdamW, weight decay 0.01 on non-norm parameters |
+| Schedule | cosine to 5% of peak, 5% warmup, peak 1.5e-4 |
+| Gradient clipping | 1.0, checked before the step rather than after |
+| Selection | best character error rate on the held-out split |
+| Hardware | one A10G |
+
+## Limitations
+
+**`ţ` (U+0163) has no slot in the vocabulary, and it is real Kabyle.**
+`docs/orthography.md` attests it 21,058 times in this project's own corpus. It encodes as
+`<unk>` and cannot be produced: `Aţan yeţţaḍsa, ţ-ţaqbaylit.` comes back as
+`Aṭan yeaḍsa, -aqbaylit.` in every typeface tested. Adding the letter changes the width of
+the output projection, so it costs a retrain, and until then any page using that convention
+is unrecoverable at those positions.
+
+**Casing and spacing are the dominant error class**, not the diacritics. Word-initial
+capitals are inserted where the source has none, and word boundaries are split or merged.
+
+**Page layout is a projection profile, nothing more.** `transcribe_page` finds horizontal
+ink bands and crops them. There is no column detection, no reading-order model and no table
+or figure handling, so a two-column periodical is read straight across.
+
 ## Files
 
 | file | contents |