Upload folder using huggingface_hub

README.md

@@ -0,0 +1,290 @@
+---
+language:
+- kab
+license: apache-2.0
+tags:
+- kabyle
+- taqbaylit
+- berber
+- amazigh
+- ocr
+- image-to-text
+- tifinagh
+- document-understanding
+- low-resource
+- ctc
+pipeline_tag: image-to-text
+metrics:
+- cer
+- wer
+- exact_match
+model-index:
+- name: Feraoun-Pro
+  results:
+  - task:
+      type: image-to-text
+      name: Line-level document OCR, Kabyle Latin
+    dataset:
+      type: agbalu/KabTifinagh
+      name: 25,484 held-out rendered lines, Latin
+    metrics:
+    - type: cer
+      value: 0.0734
+      name: Character error rate
+  - task:
+      type: image-to-text
+      name: Line-level document OCR, Kabyle Neo-Tifinagh
+    dataset:
+      type: agbalu/KabTifinagh
+      name: 4,516 held-out rendered lines, Neo-Tifinagh
+    metrics:
+    - type: cer
+      value: 0.0091
+      name: Character error rate
+---
+
+# Feraoun-Pro
+
+A 51M-parameter CTC recogniser that reads a **line of printed Kabyle** (Taqbaylit, `kab`)
+from a pixel strip and writes it out as text — in the Berber Latin orthography and in
+Neo-Tifinagh, from one checkpoint, without being told which script it is looking at, in a
+single forward pass.
+
+It exists because the Kabyle written record is on paper. Novels, grammars, periodicals and
+the archive scans that carry a century of prose are images, and general-purpose OCR was
+never trained on the language: the emphatic consonants `ḍ ḥ ṛ ṣ ṭ ẓ` are a base letter plus
+a sub-dot that other engines drop or normalise away, and dropping the dot changes the word.
+`ţ` — Boulifa's original transcription convention — has a slot in this vocabulary. It has
+never had one before.
+
+**What it adds over [`agbalu/Feraoun-36M`](https://huggingface.co/agbalu/Feraoun-36M),
+which also reads both scripts.** A 173-symbol table against 171 — `ţ` and `Ţ`, Boulifa's
+convention, and 33 Neo-Tifinagh codepoints against 31. A single CTC forward pass in place of
+one autoregressive decoder step per character. A variable-width canvas up to 1,024 px, so a
+long line is no longer squeezed to fit. 600,000 training lines against 80,000. The two
+models are separate artifacts on the same task and both stay published; Results says how to
+choose.
+
+## Results
+
+**30,000 held-out lines, rendered**, drawn at a fixed seed over the full 600,000-line pool —
+85% Latin, 15% Neo-Tifinagh, matching the training mix. The model saw none of these
+sentences during training.
+
+| | Latin (`n` = 25,484) | Neo-Tifinagh (`n` = 4,516) |
+|---|---|---|
+| Character error rate | **7.34%** | **0.91%** |
+
+**Line exact match over all 30,000 lines: 81.77%.**
+
+Three things the table does not say on its own.
+
+**`Feraoun-36M` publishes 2.85% Latin CER, 1.64% Neo-Tifinagh CER and 70.20% exact match,
+and none of those is comparable with the figures above.** The two models were scored on
+different held-out draws from different line sets, and neither has been run on the other's —
+so the pair is two measurements, not a ranking, and this card does not turn it into one.
+Choose by page: this model takes a line up to 1,024 px without squeezing it, has `ţ` in its
+alphabet, and decodes in one pass; the 36M model is smaller and is the one whose font
+generalisation has been tested.
+
+**The Tifinagh number is a retention result, not a difficulty result.** Neo-Tifinagh writes
+no sub-dots and no capitals and its lines are shorter, so sub-1% there is not a better score
+on the same task — it is an easier task, and the figure's value is that it proves the script
+survived training alongside Latin rather than fading.
+
+**Checkpoint selection was gated on both scripts at once.** The combined error
+`0.5 · Latin CER + 0.5 · Tifinagh CER` had to improve while neither script regressed past
+its baseline. A checkpoint that reads Tifinagh better by forgetting Latin is not selected,
+which is the failure this model's whole training design is built around.
+
+## What the numbers cover, and what they do not
+
+Every figure above is measured on rendered lines. What that leaves open, stated so you can
+plan around it rather than discover it:
+
+- **Real scanned pages are not in the evaluation.** Training and validation are both
+  synthetic renders, as is `Feraoun-36M`'s 2.85%. Performance on Adlis book scans — genuine
+  century-old print — is open for both models.
+- **The head-to-head against `Feraoun-36M` has not been run**, for the reason above.
+- **`ţ`/`Ţ` have vocabulary slots and no per-glyph score.** The validation pool holds too
+  few `ţ` lines to report one. The slots are what make the glyph representable at all; no
+  other Kabyle OCR system has them.
+- **Font generalisation is not quantified.** Training and evaluation draw from the same set
+  of container faces, so a typeface the model has never seen is outside what these numbers
+  describe. Expect it to be the axis that moves most.
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | **51,440,589**, plus 132,036 elements of buffer — the sinusoidal position table and the stem's batch-norm statistics — for 51,572,625 tensor elements in the weights file. The training checkpoint holds 60,158,014: the difference is an 8,585,389-parameter auxiliary decoder that supplies gradients and is not exported |
+| Stem | 5-layer convolutional front end, 64 px fixed height, variable width to 1,024 px |
+| Encoder | 16-layer pre-norm Transformer, 512 hidden, 8 heads |
+| Head | CTC over 173 classes, greedy decode |
+| Input | `(1, 3, 64, W)`, pixel range `[−1, 1]`, aspect ratio preserved |
+| Vocabulary | 173 classes — 4 special, Latin, Kabyle extended and sub-dot (`ḍ ḥ ṛ ṣ ṭ ẓ`), accented Latin, `ţ`/`Ţ`, digits, punctuation, 33 Neo-Tifinagh |
+
+**No decoder steps, no beam search, no key-value cache.** A line is one forward pass:
+convolutional stem, Transformer encoder, linear projection, CTC argmax. The auxiliary
+autoregressive decoder existed only to keep gradients reaching the early convolutional
+layers, and it is not in the published weights.
+
+**173 classes, not subwords.** A subword tokeniser completes vocabulary words where the ink
+is broken, which is precisely the failure an OCR system must not have: it produces fluent
+Kabyle that is not what the page says.
+
+## Intended use
+
+Turning printed Kabyle into text: scanned novels, grammars, periodicals and archive
+material, in the Latin orthography or in Neo-Tifinagh, from the same model.
+
+**Not suitable for**: handwriting, absent from the training data; multi-column layout, which
+`transcribe_page` reads straight across; any decision about a person; or any language other
+than Kabyle. Tarifit, Tashelhit and Central Atlas Tamazight share part of this orthography
+and are not evaluated here.
+
+## Usage
+
+`transformers`, `torch`, `pillow` and `numpy`, nothing else. The architecture is not one of
+the library's own, so the modelling code travels in this repository and
+`trust_remote_code=True` is what loads it.
+
+```python
+from PIL import Image
+from transformers import AutoModel
+
+model = AutoModel.from_pretrained("agbalu/Feraoun-Pro", trust_remote_code=True).eval()
+
+print(model.transcribe(Image.open("line_crop.png")))
+# 'Taqbaylit d tutlayt tayemmat nneɣ.'
+
+print(model.transcribe(Image.open("tifinagh_line.png")))
+# 'ⵜⴰⵇⴱⴰⵢⵍⵉⵜ ⴷ ⵜⵓⵜⵍⴰⵢⵜ ⵜⴰⵢⴻⵎⵎⴰⵜ ⵏⵏⴻⵖ.'
+```
+
+`transcribe` does the preprocessing as well as the decoding, and that matters: the model was
+fitted on lines scaled to 64 px of height with aspect ratio preserved, so a caller who builds
+the tensor themselves at another scale is measuring a different model.
+
+For a full page, `transcribe_page` segments on the horizontal ink profile first. That
+segmentation assumes a single column of roughly horizontal lines; it has not been evaluated
+on multi-column layouts, marginalia or skewed scans.
+
+## Training data
+
+Lines are **rendered, not scanned.** There is no page of real Kabyle print with
+character-level ground truth to train against.
+
+| | |
+|---|---|
+| Latin text | AƔBALU-Text v1, not publicly released |
+| Neo-Tifinagh text | [`agbalu/KabTifinagh`](https://huggingface.co/datasets/agbalu/KabTifinagh) `script_conversion` train split, CC-BY-2.0 |
+| Objective | CTC, plus an auxiliary autoregressive decoder whose gradients reach the encoder and which is **not** exported |
+
+The renderer composites text set at 24–38 pt in the faces available in the training container
+onto generated paper grain, then degrades by blur, exposure, JPEG artefacting and a slight
+skew. Height is fixed at 64 px; width follows the line up to 1,024 px, so glyphs are never
+squashed to a square canvas.
+
+**The degradation is deliberately modest**, and the model has not been shown genuinely hard
+inputs: no bleed-through from the reverse of the page, no ink starvation, no page-edge
+shadow, no show-through from facing pages. Expect real archival scans to be harder than
+anything in training.
+
+**The corpus composition is uneven.** AƔBALU-Text v1 is a 42-source mixture and the sources
+are not weighted equally. The line set is a seeded draw over the full pool rather than a
+prefix, which is what keeps it from being one source, but it does not flatten the source
+distribution: a novel's long sentences and a newspaper's headlines are both in the pool and
+both outside what a 30,000-line validation draw can fully represent.
+
+## Training recipe
+
+One A10G throughout, 600,000 rendered lines per stage, 570,000 train / 30,000 validation on
+a fixed seed, batch 64, one epoch per stage, cosine schedule with 5% warmup.
+
+| stage | mix | peak lr | what it does |
+|---|---|---|---|
+| 1 | Latin only | `2e-5` | builds the Latin recogniser and the sub-dot distinctions |
+| 2 | 60% Latin / 40% Tifinagh | `2e-5` | introduces the 33 Neo-Tifinagh codepoints |
+| 3 | 85% Latin / 15% Tifinagh | `5e-6` → `2.5e-7` | settles both scripts; 8,907 steps |
+
+**The staging is the design, not a schedule that happened to be split up.** Introducing
+Neo-Tifinagh at 40% costs Latin accuracy — that is where this model's 7.34% Latin CER comes
+from, and it is why stage 3 drops the Tifinagh share to a rehearsal fraction and the learning
+rate by 4×. The 15% rehearsal is enough to keep the Tifinagh codepoints from fading while the
+Latin weights settle. Every stage's checkpoint is gated on the combined two-script error with
+a per-script regression ceiling, so no stage can trade one script for the other.
+
+## Limitations
+
+**7.34% Latin CER on synthetic renders is not 7.34% on your documents.** The validation set
+is drawn from the same rendered distribution as training. Real scanned pages have ink bleed,
+page curvature, show-through and aging that the renderer approximates at modest levels.
+Whether the gap is one point or ten is not known.
+
+**Latin character accuracy is the weak axis.** It is the price of dual-script capacity in a
+model this size, and it is stated in Results rather than buried here.
+
+**Punctuation after dense glyph clusters is a consistent weakness.** In Neo-Tifinagh,
+`ⴰⵣⵓⵍ ⴼⵍⵍ-ⴰⵡⴻⵏ, ⴰⵎⴻⴽ ⵜⴻⵍⵍⴰⵎ?` came back as `ⴰⵣⵓⵍ ⴼⵍⵍ,ⴰⵡⴻⵏ, ⴰⵎⴻⴽ ⵜⴻⵍⵍⴰⵎ!` — the hyphen read
+as a comma and the closing `?` as `!`, both at marks that follow glyphs without a full
+inter-word space.
+
+**Whitespace boundaries are not reliable.** The validation ground truth is stripped, so
+trailing-space errors introduced by the renderer's post-processing do not appear in the CER.
+They will appear in your output if the downstream consumer is whitespace-sensitive.
+
+**Page layout is a projection profile, nothing more.** `transcribe_page` finds horizontal ink
+bands and crops them. There is no column detection, no reading-order model and no table or
+figure handling, so a two-column periodical is read straight across.
+
+## Files
+
+| file | contents |
+|---|---|
+| `model.safetensors` | the convolutional stem, the 16-layer encoder and the CTC head — 51,440,589 parameters and 132,036 elements of buffer, 206 MB fp32. `from_pretrained` reports no missing and no unexpected keys |
+| `config.json` | architecture and the `auto_map` that points at the code below |
+| `configuration_feraoun_pro.py`, `modeling_feraoun_pro.py` | the architecture and the preprocessing, importing only `torch`, `transformers`, `pillow` and `numpy` |
+| `tokenizer.json`, `tokenizer_config.json` | the 173-symbol table, id for id with the one the model was trained on |
+
+Training cannot be resumed from the published files: the optimizer, scheduler, step and epoch
+stay in the source checkpoint on the training volume and are not exported.
+
+## Reproduction
+
+```bash
+make modal-ocr-pro RUN=latin      EPOCHS=1 LR=2e-5   RATIO=0.0  LINES=600000
+make modal-ocr-pro RUN=dual       EPOCHS=1 LR=2e-5   RATIO=0.40 RESUME=latin
+make modal-ocr-pro RUN=feraoun-pro EPOCHS=1 LR=5e-6  RATIO=0.15 RESUME=dual
+make modal-ocr-pro-pull RUN=feraoun-pro
+make push REPO=feraoun-pro
+```
+
+## The name
+
+**Mouloud Feraoun** (1913–1962), the schoolteacher from Tizi Hibel who wrote *Le Fils du
+pauvre* and put Si Mohand's oral poetry onto the printed page — the direction this model
+reverses. He was assassinated by the OAS on 15 March 1962, three days before the Évian
+Accords were signed. The naming is homage and carries no endorsement.
+
+## Citation
+
+```bibtex
+@software{agbalu_feraoun_pro_2026,
+  title  = {Feraoun-Pro: dual-script CTC document OCR for Kabyle},
+  author = {AƔBALU},
+  year   = {2026},
+  url    = {https://huggingface.co/agbalu/Feraoun-Pro},
+  note   = {51,572,625 parameters; Latin CER 7.34\%, Neo-Tifinagh CER 0.91\%,
+            line exact match 81.77\%, over 30,000 held-out rendered lines}
+}
+```
+
+## Licence
+
+**Apache-2.0** on the weights and the code. The training text is CC-BY-2.0 on the Tifinagh
+side and a mixed corpus of uneven provenance on the Latin side. A permissive grant on the
+weights makes no claim about the text behind them, so the corpus composition is stated above
+rather than assumed.
+
+Part of [AƔBALU](https://huggingface.co/agbalu), a Kabyle NLP collection.

__init__.py

@@ -0,0 +1,6 @@
+"""Feraoun-Pro standalone hub package."""
+
+from .configuration_feraoun_pro import FeraounProConfig
+from .modeling_feraoun_pro import FeraounProForCTC, prepare_line_image, segment_page_into_lines
+
+__all__ = ["FeraounProConfig", "FeraounProForCTC", "prepare_line_image", "segment_page_into_lines"]

config.json

@@ -0,0 +1,197 @@
+{
+  "architectures": [
+    "FeraounProForCTC"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_feraoun_pro.FeraounProConfig",
+    "AutoModel": "modeling_feraoun_pro.FeraounProForCTC"
+  },
+  "bos_token_id": 1,
+  "d_model": 512,
+  "dim_feedforward": 2048,
+  "dropout": 0.1,
+  "eos_token_id": 2,
+  "max_width_frames": 256,
+  "model_type": "feraoun_pro",
+  "num_encoder_layers": 16,
+  "num_heads": 8,
+  "pad_token_id": 0,
+  "transformers_version": "5.12.1",
+  "unk_token_id": 3,
+  "vocab_size": 173,
+  "vocabulary": [
+    "<pad>",
+    "<s>",
+    "</s>",
+    "<unk>",
+    "a",
+    "b",
+    "c",
+    "d",
+    "e",
+    "f",
+    "g",
+    "h",
+    "i",
+    "j",
+    "k",
+    "l",
+    "m",
+    "n",
+    "o",
+    "p",
+    "q",
+    "r",
+    "s",
+    "t",
+    "u",
+    "v",
+    "w",
+    "x",
+    "y",
+    "z",
+    "A",
+    "B",
+    "C",
+    "D",
+    "E",
+    "F",
+    "G",
+    "H",
+    "I",
+    "J",
+    "K",
+    "L",
+    "M",
+    "N",
+    "O",
+    "P",
+    "Q",
+    "R",
+    "S",
+    "T",
+    "U",
+    "V",
+    "W",
+    "X",
+    "Y",
+    "Z",
+    "\u0263",
+    "\u0194",
+    "\u025b",
+    "\u0190",
+    "\u010d",
+    "\u010c",
+    "\u01e7",
+    "\u01e6",
+    "\u0163",
+    "\u0162",
+    "\u1e0d",
+    "\u1e0c",
+    "\u1e25",
+    "\u1e24",
+    "\u1e5b",
+    "\u1e5a",
+    "\u1e63",
+    "\u1e62",
+    "\u1e6d",
+    "\u1e6c",
+    "\u1e93",
+    "\u1e92",
+    "\u00e9",
+    "\u00e8",
+    "\u00ea",
+    "\u00eb",
+    "\u00e0",
+    "\u00e2",
+    "\u00ee",
+    "\u00ef",
+    "\u00f4",
+    "\u00f9",
+    "\u00fb",
+    "\u00e7",
+    "\u00c9",
+    "\u00c8",
+    "\u00ca",
+    "\u00cb",
+    "\u00c0",
+    "\u00c2",
+    "\u00ce",
+    "\u00cf",
+    "\u00d4",
+    "\u00d9",
+    "\u00db",
+    "\u00c7",
+    "0",
+    "1",
+    "2",
+    "3",
+    "4",
+    "5",
+    "6",
+    "7",
+    "8",
+    "9",
+    " ",
+    ".",
+    ",",
+    "?",
+    "!",
+    "\u00ab",
+    "\u00bb",
+    "-",
+    "\u2014",
+    ":",
+    ";",
+    "'",
+    "\"",
+    "/",
+    "(",
+    ")",
+    "[",
+    "]",
+    "{",
+    "}",
+    "%",
+    "*",
+    "+",
+    "=",
+    "_",
+    "~",
+    "\u2013",
+    "\u2026",
+    "\n",
+    "\t",
+    "\u2d30",
+    "\u2d31",
+    "\u2d33",
+    "\u2d37",
+    "\u2d39",
+    "\u2d3b",
+    "\u2d3c",
+    "\u2d3d",
+    "\u2d40",
+    "\u2d43",
+    "\u2d44",
+    "\u2d45",
+    "\u2d47",
+    "\u2d49",
+    "\u2d4a",
+    "\u2d4d",
+    "\u2d4e",
+    "\u2d4f",
+    "\u2d53",
+    "\u2d54",
+    "\u2d55",
+    "\u2d59",
+    "\u2d5a",
+    "\u2d5b",
+    "\u2d5c",
+    "\u2d5f",
+    "\u2d61",
+    "\u2d62",
+    "\u2d63",
+    "\u2d65",
+    "\u2d56"
+  ]
+}

configuration_feraoun_pro.py

@@ -0,0 +1,63 @@
+"""Configuration for the Feraoun-Pro document OCR model."""
+
+from __future__ import annotations
+
+from typing import Any, Final
+
+from transformers import PreTrainedConfig
+
+SYMBOLS: Final[str] = (
+    "abcdefghijklmnopqrstuvwxyz"
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+    "ɣƔɛƐčČǧǦţŢ"
+    "ḍḌḥḤṛṚṣṢṭṬẓẒ"
+    "éèêëàâîïôùûçÉÈÊËÀÂÎÏÔÙÛÇ"
+    "0123456789"
+    " .,?!«»-—:;'\"/()[]{}%*+=_~–…\n\t"
+    "ⴰⴱⴳⴷⴹⴻⴼⴽⵀⵃⵄⵅⵇⵉⵊⵍⵎⵏⵓⵔⵕⵙⵚⵛⵜⵟⵡⵢⵣⵥⵖ"
+)
+"""The 169 printable classes after the four special tokens, spelled out rather than
+imported: nothing under `hub/` may import `agbalu`, because these modules are loaded from
+the published repository where this package does not exist.
+`tests/unit/test_ocr_pro.py` pins them against `agbalu.ocr.vocabulary_pro`, so the two
+copies cannot drift apart silently."""
+
+DEFAULT_VOCABULARY: Final[tuple[str, ...]] = ("<pad>", "<s>", "</s>", "<unk>", *SYMBOLS)
+
+
+class FeraounProConfig(PreTrainedConfig):
+    model_type = "feraoun_pro"
+
+    def __init__(
+        self,
+        vocabulary: list[str] | tuple[str, ...] | None = None,
+        vocab_size: int = 173,
+        d_model: int = 512,
+        num_encoder_layers: int = 16,
+        num_heads: int = 8,
+        dim_feedforward: int = 2048,
+        max_width_frames: int = 256,
+        dropout: float = 0.1,
+        **kwargs: Any,
+    ) -> None:
+        # The symbol table ships in `config.json`: a CTC id means nothing without it, and a
+        # downloader who has to reconstruct it from this repository does not have a usable
+        # artifact. `vocab_size` stays a separate field because it is the width of
+        # `ctc_head` and a saved config carries it.
+        self.vocabulary = list(vocabulary) if vocabulary is not None else list(DEFAULT_VOCABULARY)
+        self.vocab_size = vocab_size
+        self.d_model = d_model
+        self.num_encoder_layers = num_encoder_layers
+        self.num_heads = num_heads
+        self.dim_feedforward = dim_feedforward
+        self.max_width_frames = max_width_frames
+        self.dropout = dropout
+
+        kwargs.setdefault("pad_token_id", 0)
+        kwargs.setdefault("bos_token_id", 1)
+        kwargs.setdefault("eos_token_id", 2)
+        kwargs.setdefault("unk_token_id", 3)
+        super().__init__(**kwargs)
+
+
+__all__ = ["DEFAULT_VOCABULARY", "SYMBOLS", "FeraounProConfig"]

export.stats.json

@@ -0,0 +1,100 @@
+{
+  "source": "artifacts/runs/feraoun-pro/best.pt",
+  "source_contents": [
+    "model",
+    "optimizer",
+    "rng",
+    "state"
+  ],
+  "dropped_state": [
+    "optimizer",
+    "rng"
+  ],
+  "dropped_tied": [],
+  "dropped_derived": [],
+  "dropped_training_only": [
+    "aux_decoder.decoder.layers.0.linear1.bias",
+    "aux_decoder.decoder.layers.0.linear1.weight",
+    "aux_decoder.decoder.layers.0.linear2.bias",
+    "aux_decoder.decoder.layers.0.linear2.weight",
+    "aux_decoder.decoder.layers.0.multihead_attn.in_proj_bias",
+    "aux_decoder.decoder.layers.0.multihead_attn.in_proj_weight",
+    "aux_decoder.decoder.layers.0.multihead_attn.out_proj.bias",
+    "aux_decoder.decoder.layers.0.multihead_attn.out_proj.weight",
+    "aux_decoder.decoder.layers.0.norm1.bias",
+    "aux_decoder.decoder.layers.0.norm1.weight",
+    "aux_decoder.decoder.layers.0.norm2.bias",
+    "aux_decoder.decoder.layers.0.norm2.weight",
+    "aux_decoder.decoder.layers.0.norm3.bias",
+    "aux_decoder.decoder.layers.0.norm3.weight",
+    "aux_decoder.decoder.layers.0.self_attn.in_proj_bias",
+    "aux_decoder.decoder.layers.0.self_attn.in_proj_weight",
+    "aux_decoder.decoder.layers.0.self_attn.out_proj.bias",
+    "aux_decoder.decoder.layers.0.self_attn.out_proj.weight",
+    "aux_decoder.decoder.layers.1.linear1.bias",
+    "aux_decoder.decoder.layers.1.linear1.weight",
+    "aux_decoder.decoder.layers.1.linear2.bias",
+    "aux_decoder.decoder.layers.1.linear2.weight",
+    "aux_decoder.decoder.layers.1.multihead_attn.in_proj_bias",
+    "aux_decoder.decoder.layers.1.multihead_attn.in_proj_weight",
+    "aux_decoder.decoder.layers.1.multihead_attn.out_proj.bias",
+    "aux_decoder.decoder.layers.1.multihead_attn.out_proj.weight",
+    "aux_decoder.decoder.layers.1.norm1.bias",
+    "aux_decoder.decoder.layers.1.norm1.weight",
+    "aux_decoder.decoder.layers.1.norm2.bias",
+    "aux_decoder.decoder.layers.1.norm2.weight",
+    "aux_decoder.decoder.layers.1.norm3.bias",
+    "aux_decoder.decoder.layers.1.norm3.weight",
+    "aux_decoder.decoder.layers.1.self_attn.in_proj_bias",
+    "aux_decoder.decoder.layers.1.self_attn.in_proj_weight",
+    "aux_decoder.decoder.layers.1.self_attn.out_proj.bias",
+    "aux_decoder.decoder.layers.1.self_attn.out_proj.weight",
+    "aux_decoder.embedding.weight",
+    "aux_decoder.lm_head.bias",
+    "aux_decoder.lm_head.weight"
+  ],
+  "tensors": 221,
+  "parameters": 51572625,
+  "files": [
+    {
+      "name": "README.md",
+      "bytes": 13969,
+      "sha256": "39d5914cdb53dcfd14aefcb7838e870cbd981600ac21ac3734934a0534b2007f"
+    },
+    {
+      "name": "__init__.py",
+      "bytes": 296,
+      "sha256": "9ac3b94df93a57b0dac01996acc6e5ef217e016f117fbc97fd05c473d7dbfc84"
+    },
+    {
+      "name": "config.json",
+      "bytes": 2507,
+      "sha256": "746ce438be8325339c8e5a0b0caddcbd20d1cab56386f13e0157d147ce3f65ab"
+    },
+    {
+      "name": "configuration_feraoun_pro.py",
+      "bytes": 2423,
+      "sha256": "32020487761b9375e0a6931cfdfd39551bccd8c50939a39108bdc845036b8b8f"
+    },
+    {
+      "name": "model.safetensors",
+      "bytes": 206313076,
+      "sha256": "dfc55395ed15b9d727eae16594ec7f87c8ef28fdc2e6d5f7bd6e88d7bbafe165"
+    },
+    {
+      "name": "modeling_feraoun_pro.py",
+      "bytes": 7590,
+      "sha256": "7c8018a171eb4647453140103bf919f98d2860339a74d819ef1869865a239563"
+    },
+    {
+      "name": "tokenizer.json",
+      "bytes": 4764,
+      "sha256": "350100049e4f193563065147c35fce2dbcc344cf98420e0fa209c4b668864c37"
+    },
+    {
+      "name": "tokenizer_config.json",
+      "bytes": 193,
+      "sha256": "444cfded3d0102faf5c8bb4706f540becd8103ce7344ddbbad8df2da0928fa52"
+    }
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfc55395ed15b9d727eae16594ec7f87c8ef28fdc2e6d5f7bd6e88d7bbafe165
+size 206313076

modeling_feraoun_pro.py

@@ -0,0 +1,207 @@
+"""Feraoun-Pro: Hybrid CNN-Transformer CTC Document OCR Model for Kabyle."""
+
+from __future__ import annotations
+
+import math
+from typing import TYPE_CHECKING, Final
+
+import torch
+from torch import Tensor, nn
+from transformers import PreTrainedModel
+
+from .configuration_feraoun_pro import FeraounProConfig
+
+if TYPE_CHECKING:
+    from PIL import Image
+
+TARGET_HEIGHT: Final[int] = 64
+MAX_WIDTH: Final[int] = 1024
+CROP_INK_THRESHOLD: Final[int] = 225
+LINE_INK_THRESHOLD: Final[int] = 200
+LINE_MARGIN: Final[int] = 4
+
+
+def crop_text_bbox(image: Image.Image, padding: int = LINE_MARGIN) -> Image.Image:
+    """Crop to the bounding box of the ink."""
+    import numpy as np
+
+    array = np.array(image.convert("L"), dtype=np.uint8)
+    ink = array < CROP_INK_THRESHOLD
+    if not ink.any():
+        return image
+
+    rows, columns = np.where(ink)
+    left = max(0, int(columns.min()) - padding)
+    right = min(image.width, int(columns.max()) + padding)
+    top = max(0, int(rows.min()) - padding)
+    bottom = min(image.height, int(rows.max()) + padding)
+    if right <= left or bottom <= top:
+        return image
+    return image.crop((left, top, right, bottom))
+
+
+def prepare_line_image(image: Image.Image) -> Tensor:
+    """Crop to the ink, scale the height to 64 px, normalise, and clamp the width."""
+    import numpy as np
+    from PIL import Image as PILImage
+
+    cropped = crop_text_bbox(image).convert("RGB")
+    width, height = cropped.size
+
+    scale = float(TARGET_HEIGHT) / max(height, 1)
+    new_width = max(16, min(MAX_WIDTH, int(width * scale)))
+    # Ensure divisible by 4 for CNN stem
+    new_width = int(math.ceil(new_width / 4.0) * 4)
+
+    resized = cropped.resize((new_width, TARGET_HEIGHT), resample=PILImage.Resampling.LANCZOS)
+    arr = np.array(resized, dtype=np.float32) / 255.0
+    arr = (arr - 0.5) / 0.5
+    return torch.from_numpy(arr).permute(2, 0, 1).unsqueeze(0)  # (1, 3, 64, W)
+
+
+def segment_page_into_lines(page: Image.Image, min_line_height: int = 15) -> list[Image.Image]:
+    """Segment page into line strips on horizontal ink profile."""
+    import numpy as np
+
+    array = np.array(page.convert("L"), dtype=np.uint8)
+    ink = (array < LINE_INK_THRESHOLD).astype(np.float32).sum(axis=1)
+    active = ink > 0.015 * array.shape[1]
+
+    lines: list[Image.Image] = []
+    start = 0
+    inside = False
+
+    for idx, is_ink in enumerate(active):
+        if is_ink and not inside:
+            start = idx
+            inside = True
+        elif not is_ink and inside:
+            inside = False
+            if idx - start >= min_line_height:
+                lines.append(
+                    page.crop(
+                        (0, max(0, start - 2), page.width, min(page.height, idx + 2))
+                    ).convert("RGB")
+                )
+
+    if inside and len(active) - start >= min_line_height:
+        lines.append(page.crop((0, start, page.width, len(active))).convert("RGB"))
+
+    return lines
+
+
+class CNNStem(nn.Module):
+    def __init__(self, d_model: int) -> None:
+        super().__init__()
+        self.block1 = nn.Sequential(
+            nn.Conv2d(3, 32, kernel_size=3, stride=(2, 1), padding=1, bias=False),
+            nn.BatchNorm2d(32),
+            nn.GELU(),
+        )
+        self.block2 = nn.Sequential(
+            nn.Conv2d(32, 64, kernel_size=3, stride=(2, 1), padding=1, bias=False),
+            nn.BatchNorm2d(64),
+            nn.GELU(),
+        )
+        self.block3 = nn.Sequential(
+            nn.Conv2d(64, 128, kernel_size=3, stride=(2, 2), padding=1, bias=False),
+            nn.BatchNorm2d(128),
+            nn.GELU(),
+        )
+        self.block4 = nn.Sequential(
+            nn.Conv2d(128, 256, kernel_size=3, stride=(2, 2), padding=1, bias=False),
+            nn.BatchNorm2d(256),
+            nn.GELU(),
+        )
+        self.proj = nn.Linear(256 * 4, d_model)
+
+    def forward(self, x: Tensor) -> Tensor:
+        x = self.block1(x)
+        x = self.block2(x)
+        x = self.block3(x)
+        x = self.block4(x)
+        b, c, h, w = x.shape
+        x = x.permute(0, 3, 1, 2).contiguous().view(b, w, c * h)
+        out: Tensor = self.proj(x)
+        return out
+
+
+class FeraounProForCTC(PreTrainedModel):
+    config_class = FeraounProConfig
+    base_model_prefix = "feraoun_pro"
+
+    def __init__(self, config: FeraounProConfig) -> None:
+        super().__init__(config)
+        self.config = config
+        self.stem = CNNStem(d_model=config.d_model)
+
+        # Build sinusoidal positional encodings
+        pe = torch.zeros(config.max_width_frames, config.d_model)
+        position = torch.arange(config.max_width_frames).unsqueeze(1)
+        div_term = torch.exp(
+            torch.arange(0, config.d_model, 2) * (-math.log(10000.0) / config.d_model)
+        )
+        pe[:, 0::2] = torch.sin(position * div_term)
+        pe[:, 1::2] = torch.cos(position * div_term)
+        self.register_buffer("pos_encoding", pe)
+
+        encoder_layer = nn.TransformerEncoderLayer(
+            d_model=config.d_model,
+            nhead=config.num_heads,
+            dim_feedforward=config.dim_feedforward,
+            dropout=config.dropout,
+            activation="gelu",
+            batch_first=True,
+            norm_first=True,
+        )
+        # The nested-tensor fast path does not apply to a pre-norm layer; torch infers that
+        # and warns. Declaring it keeps a downloader's console clean.
+        self.encoder = nn.TransformerEncoder(
+            encoder_layer, num_layers=config.num_encoder_layers, enable_nested_tensor=False
+        )
+        self.ctc_head = nn.Linear(config.d_model, config.vocab_size)
+        self.post_init()
+
+    def forward(self, pixel_values: Tensor) -> Tensor:
+        features = self.stem(pixel_values)
+        seq_len = features.size(1)
+        # `get_buffer` rather than the attribute: `nn.Module.__getattr__` is typed as
+        # `Tensor | Module`, and a registered buffer is always the former.
+        pe = self.get_buffer("pos_encoding")[:seq_len, :].unsqueeze(0).to(features.device)
+        encoded = self.encoder(features + pe)
+        logits: Tensor = self.ctc_head(encoded)
+        return logits
+
+    def decode_ctc(self, token_ids: list[int]) -> str:
+        """Collapse repeats, then drop blanks — the CTC rule, in that order.
+
+        Dropping blanks first merges the two characters a blank was separating, so a
+        geminate consonant comes back single.
+        """
+        vocabulary = self.config.vocabulary
+        blank = self.config.pad_token_id
+        special = {blank, self.config.bos_token_id, self.config.eos_token_id}
+        collapsed: list[int] = []
+        previous = -1
+        for token_id in token_ids:
+            if token_id != previous:
+                if token_id not in special:
+                    collapsed.append(token_id)
+                previous = token_id
+        return "".join(vocabulary[token_id] for token_id in collapsed if token_id < len(vocabulary))
+
+    @torch.no_grad()
+    def transcribe(self, image: Image.Image) -> str:
+        """Read one line crop. Preprocessing included, because it is half of this model."""
+        self.eval()
+        pixel_values = prepare_line_image(image).to(self.device)
+        logits = self(pixel_values=pixel_values)
+        return self.decode_ctc(torch.argmax(logits, dim=-1)[0].tolist())
+
+    @torch.no_grad()
+    def transcribe_page(self, page: Image.Image) -> list[str]:
+        """Segment a page into lines and read each one."""
+        return [self.transcribe(line) for line in segment_page_into_lines(page)]
+
+
+__all__ = ["FeraounProForCTC", "prepare_line_image", "segment_page_into_lines"]

tokenizer.json

@@ -0,0 +1,291 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Split",
+    "pattern": {
+      "Regex": "[\\s\\S]"
+    },
+    "behavior": "Isolated",
+    "invert": false
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "SpecialToken": {
+          "id": "<s>",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "SpecialToken": {
+          "id": "</s>",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {
+      "</s>": {
+        "id": "</s>",
+        "ids": [
+          2
+        ],
+        "tokens": [
+          "</s>"
+        ]
+      },
+      "<s>": {
+        "id": "<s>",
+        "ids": [
+          1
+        ],
+        "tokens": [
+          "<s>"
+        ]
+      }
+    }
+  },
+  "decoder": {
+    "type": "Fuse"
+  },
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "<pad>": 0,
+      "<s>": 1,
+      "</s>": 2,
+      "<unk>": 3,
+      "a": 4,
+      "b": 5,
+      "c": 6,
+      "d": 7,
+      "e": 8,
+      "f": 9,
+      "g": 10,
+      "h": 11,
+      "i": 12,
+      "j": 13,
+      "k": 14,
+      "l": 15,
+      "m": 16,
+      "n": 17,
+      "o": 18,
+      "p": 19,
+      "q": 20,
+      "r": 21,
+      "s": 22,
+      "t": 23,
+      "u": 24,
+      "v": 25,
+      "w": 26,
+      "x": 27,
+      "y": 28,
+      "z": 29,
+      "A": 30,
+      "B": 31,
+      "C": 32,
+      "D": 33,
+      "E": 34,
+      "F": 35,
+      "G": 36,
+      "H": 37,
+      "I": 38,
+      "J": 39,
+      "K": 40,
+      "L": 41,
+      "M": 42,
+      "N": 43,
+      "O": 44,
+      "P": 45,
+      "Q": 46,
+      "R": 47,
+      "S": 48,
+      "T": 49,
+      "U": 50,
+      "V": 51,
+      "W": 52,
+      "X": 53,
+      "Y": 54,
+      "Z": 55,
+      "ɣ": 56,
+      "Ɣ": 57,
+      "ɛ": 58,
+      "Ɛ": 59,
+      "č": 60,
+      "Č": 61,
+      "ǧ": 62,
+      "Ǧ": 63,
+      "ţ": 64,
+      "Ţ": 65,
+      "ḍ": 66,
+      "Ḍ": 67,
+      "ḥ": 68,
+      "Ḥ": 69,
+      "ṛ": 70,
+      "Ṛ": 71,
+      "ṣ": 72,
+      "Ṣ": 73,
+      "ṭ": 74,
+      "Ṭ": 75,
+      "ẓ": 76,
+      "Ẓ": 77,
+      "é": 78,
+      "è": 79,
+      "ê": 80,
+      "ë": 81,
+      "à": 82,
+      "â": 83,
+      "î": 84,
+      "ï": 85,
+      "ô": 86,
+      "ù": 87,
+      "û": 88,
+      "ç": 89,
+      "É": 90,
+      "È": 91,
+      "Ê": 92,
+      "Ë": 93,
+      "À": 94,
+      "Â": 95,
+      "Î": 96,
+      "Ï": 97,
+      "Ô": 98,
+      "Ù": 99,
+      "Û": 100,
+      "Ç": 101,
+      "0": 102,
+      "1": 103,
+      "2": 104,
+      "3": 105,
+      "4": 106,
+      "5": 107,
+      "6": 108,
+      "7": 109,
+      "8": 110,
+      "9": 111,
+      " ": 112,
+      ".": 113,
+      ",": 114,
+      "?": 115,
+      "!": 116,
+      "«": 117,
+      "»": 118,
+      "-": 119,
+      "—": 120,
+      ":": 121,
+      ";": 122,
+      "'": 123,
+      "\"": 124,
+      "/": 125,
+      "(": 126,
+      ")": 127,
+      "[": 128,
+      "]": 129,
+      "{": 130,
+      "}": 131,
+      "%": 132,
+      "*": 133,
+      "+": 134,
+      "=": 135,
+      "_": 136,
+      "~": 137,
+      "–": 138,
+      "…": 139,
+      "\n": 140,
+      "\t": 141,
+      "ⴰ": 142,
+      "ⴱ": 143,
+      "ⴳ": 144,
+      "ⴷ": 145,
+      "ⴹ": 146,
+      "ⴻ": 147,
+      "ⴼ": 148,
+      "ⴽ": 149,
+      "ⵀ": 150,
+      "ⵃ": 151,
+      "ⵄ": 152,
+      "ⵅ": 153,
+      "ⵇ": 154,
+      "ⵉ": 155,
+      "ⵊ": 156,
+      "ⵍ": 157,
+      "ⵎ": 158,
+      "ⵏ": 159,
+      "ⵓ": 160,
+      "ⵔ": 161,
+      "ⵕ": 162,
+      "ⵙ": 163,
+      "ⵚ": 164,
+      "ⵛ": 165,
+      "ⵜ": 166,
+      "ⵟ": 167,
+      "ⵡ": 168,
+      "ⵢ": 169,
+      "ⵣ": 170,
+      "ⵥ": 171,
+      "ⵖ": 172
+    },
+    "unk_token": "<unk>"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "model_max_length": 1024,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}