--- license: apache-2.0 language: - de library_name: transformers pipeline_tag: text-generation datasets: - histde/dta-documents tags: - historical - historical-german - german - deutsches-textarchiv - fraktur - ocr - character-level - ngram - xlstm - custom_code --- # DTA Character 6-gram A character-level 6-gram language model of historical German (1500-1900), trained on the full [Deutsches Textarchiv](https://huggingface.co/datasets/histde/dta-documents) and packaged as a Hugging Face model. Its purpose is **OCR quality scoring**: the model assigns a bits-per-character (bpc) score to a page of text, where a low score means "looks like clean historical German" and a high score indicates OCR damage (character confusions, garbled words, noise). It was built to score the OCR quality of large digitized newspaper corpora, page by page, on CPU and at corpus scale. The model is a compact pure-numpy reimplementation of a KenLM-style character n-gram with **stupid backoff** smoothing ([Brants et al. 2007](https://aclanthology.org/D07-1090/)), wrapped in the `transformers` causal-LM interface via `trust_remote_code`. It has **no trainable parameters**: the "weights" are sorted n-gram count tables (13.8M distinct 6-grams from 1.37B characters of training text), and scoring is a handful of vectorized binary searches. The bundled character tokenizer carries the full text normalization inside, so scored text and training corpus can never diverge in preprocessing. ## Model description * **Architecture**: character n-gram (order 6) with stupid backoff (alpha 0.4). The score of a character is the relative frequency of the longest matching n-gram ending at that character, multiplied by 0.4 for every order backed off. Because the character vocabulary has 256 entries, an n-gram fits into a single `uint64` key. Each order is stored as one sorted key array plus one count array, registered as model buffers in `model.safetensors` (~314 MB). All computation is vectorized numpy on CPU. * **Tokenizer**: a character-level fast tokenizer (one token per character, 256-entry vocabulary with `` id 0 and `` id 1) whose normalizer pipeline performs the shared preprocessing: dehyphenation, Unicode NFC, lowercasing, long s `ſ` -> `s`, r rotunda `ꝛ` -> `r`, combining-e umlauts (`uͤ` -> `ü`), quote and dash unification, digits -> `0`, whitespace collapsing. * **Interface**: standard `AutoModelForCausalLM` contract. With `labels`, the forward pass returns the mean negative log-likelihood over the target characters (fast scoring path). Without `labels`, it returns full log-score "logits" over the vocabulary, so `generate()` works as well. * **Scores are for ranking, not calibrated probabilities.** Stupid backoff is not a normalized distribution: bpc values order pages by quality but are not true perplexities. For citable modified-Kneser-Ney perplexities, use a KenLM model instead. ## Usage The model ships its own modeling code, so pass `trust_remote_code=True` to both loaders: ```python import math import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "histde/dta-char-ngram" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval() page = "Die Zeitung berichtet über die neueſten Ereigniſſe in der Hauptſtadt." enc = tokenizer(page, return_tensors="pt", add_special_tokens=False) with torch.no_grad(): loss = model(**enc, labels=enc["input_ids"]).loss print(f"bits per character: {loss.item() / math.log(2):.3f}") # lower = cleaner ``` Batched scoring works with right-padding and an `attention_mask`. Padded positions are excluded from the loss. The scoring contract is identical to the DTA character xLSTM checkpoints, so the two scorers are interchangeable by model id. ## Training data The `text` column of [histde/dta-documents](https://huggingface.co/datasets/histde/dta-documents): the historical, layout-faithful transcriptions of all 5,480 DTA works (1472-1987), about 1.38B characters. After normalization the training stream contains **1.37B characters** with a 256-entry character vocabulary covering all but 0.004% of them (`` rate). Documents are separated by ``. ## Training procedure Training is a single counting pass over the encoded corpus: n-grams of order 1-6 are packed into `uint64` keys, counted per shard with `np.unique` and merged. It takes a few minutes on CPU and is fully deterministic. Distinct n-grams per order: | order | distinct n-grams | |:------|-----------------:| | 1 | 256 | | 2 | 15,123 | | 3 | 186,698 | | 4 | 1,125,157 | | 5 | 4,525,763 | | 6 | 13,788,192 | ## Evaluation **Noise ladder.** Clean DTA pages (~2,000 characters) were corrupted at controlled character-error rates with Fraktur-OCR-style noise (confusions such as `s`->`f`, `u`<->`n`, `e`->`c`, plus deletions and insertions), and every version was scored. The score must track the injected damage: | condition | bpc (mean) | |:----------|-----------:| | clean | 1.95 | | 2% CER | 2.39 | | 5% CER | 3.02 | | 10% CER | 3.97 | | 20% CER | 5.63 | | 50% CER | 8.94 | The score increased strictly with every damage step for **100% of individual pages**, and adjacent damage levels are separated with an AUC of 0.97-1.00 - even a clean page and a 2%-CER page are ranked correctly 97% of the time. **Agreement with KenLM.** Validated against a modified-Kneser-Ney character 6-gram trained with KenLM on the same corpus: Spearman correlation **0.987** across 539 clean and corrupted pages (0.955 on clean pages only), with near-identical mean bpc on clean text. **Known blind spot.** Word-shuffled pages (every word intact, order destroyed) are barely penalized (+0.27 bpc vs +0.88 for the whole-page xLSTM scorer): a 6-gram window never sees enough context to notice broken word order. For detecting structural OCR damage (column merges, scrambled reading order), pair this model with a long-context scorer. ## Considerations for using the model * **Scores live in the normalized character space.** The tokenizer lowercases, folds historical glyphs and maps digits to `0` before scoring. Scores therefore ignore capitalization and number errors by design. * **Domain**: historical German print (Early New High German to circa 1900). Scores on modern German, other languages, or non-print text will be systematically higher without indicating OCR damage. * **Ranking, not thresholds.** bpc values are comparable within one model version. Absolute thresholds do not transfer to other scorers (KenLM, xLSTM) or retrained versions. * Generation works technically (`generate()`), but the model is a scorer: sampled text is stupid-backoff German and only useful for sanity checks. ## Licensing The training corpus is the Deutsches Textarchiv, whose works carry per-document licenses (CC BY-SA variants, CC BY-NC, CC0 and others - see the [dataset card](https://huggingface.co/datasets/histde/dta-documents)). This model stores only aggregate character n-gram counts derived from that corpus. The modeling code and the count tables are released under Apache License 2.0. ## Citation Please cite the DTA when using this model: ```bibtex @misc{dta2026, author = {{Berlin-Brandenburgische Akademie der Wissenschaften}}, title = {Deutsches Textarchiv. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache}, year = {2026}, address = {Berlin}, howpublished = {Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften}, url = {https://www.deutschestextarchiv.de/} } ``` ## Acknowledgements A big thank you to the whole team of the Deutsches Textarchiv at the Berlin-Brandenburg Academy of Sciences and Humanities! Also many thanks to all the partner projects, libraries and archives that contributed texts to the DTA extension corpora. ## AI disclosure This model card was drafted with Claude Fable 5 (claude-fable-5) based on the training and evaluation results produced in the [dta-experiments](https://github.com/histde/dta-experiments) repository, and reviewed by the model author. The training and modeling scripts carry their own AI disclosure blocks, following the rules in the [ai-disclosure](https://github.com/stefan-it/ai-disclosure) repository.