histde/dta-char-ngram

DTA Character 6-gram

A character-level 6-gram language model of historical German (1500-1900), trained on the full Deutsches Textarchiv and packaged as a Hugging Face model. Its purpose is OCR quality scoring: the model assigns a bits-per-character (bpc) score to a page of text, where a low score means "looks like clean historical German" and a high score indicates OCR damage (character confusions, garbled words, noise). It was built to score the OCR quality of large digitized newspaper corpora, page by page, on CPU and at corpus scale.

The model is a compact pure-numpy reimplementation of a KenLM-style character n-gram with stupid backoff smoothing (Brants et al. 2007), wrapped in the transformers causal-LM interface via trust_remote_code. It has no trainable parameters: the "weights" are sorted n-gram count tables (13.8M distinct 6-grams from 1.37B characters of training text), and scoring is a handful of vectorized binary searches. The bundled character tokenizer carries the full text normalization inside, so scored text and training corpus can never diverge in preprocessing.

Model description

• Architecture: character n-gram (order 6) with stupid backoff (alpha 0.4). The score of a character is the relative frequency of the longest matching n-gram ending at that character, multiplied by 0.4 for every order backed off. Because the character vocabulary has 256 entries, an n-gram fits into a single uint64 key. Each order is stored as one sorted key array plus one count array, registered as model buffers in model.safetensors (~314 MB). All computation is vectorized numpy on CPU.
• Tokenizer: a character-level fast tokenizer (one token per character, 256-entry vocabulary with <eos> id 0 and <unk> id 1) whose normalizer pipeline performs the shared preprocessing: dehyphenation, Unicode NFC, lowercasing, long s ſ -> s, r rotunda ꝛ -> r, combining-e umlauts (uͤ -> ü), quote and dash unification, digits -> 0, whitespace collapsing.
• Interface: standard AutoModelForCausalLM contract. With labels, the forward pass returns the mean negative log-likelihood over the target characters (fast scoring path). Without labels, it returns full log-score "logits" over the vocabulary, so generate() works as well.
• Scores are for ranking, not calibrated probabilities. Stupid backoff is not a normalized distribution: bpc values order pages by quality but are not true perplexities. For citable modified-Kneser-Ney perplexities, use a KenLM model instead.

Usage

The model ships its own modeling code, so pass trust_remote_code=True to both loaders:

import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "histde/dta-char-ngram"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

page = "Die Zeitung berichtet über die neueſten Ereigniſſe in der Hauptſtadt."
enc = tokenizer(page, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    loss = model(**enc, labels=enc["input_ids"]).loss
print(f"bits per character: {loss.item() / math.log(2):.3f}")  # lower = cleaner

Batched scoring works with right-padding and an attention_mask. Padded positions are excluded from the loss. The scoring contract is identical to the DTA character xLSTM checkpoints, so the two scorers are interchangeable by model id.

Training data

The text column of histde/dta-documents: the historical, layout-faithful transcriptions of all 5,480 DTA works (1472-1987), about 1.38B characters. After normalization the training stream contains 1.37B characters with a 256-entry character vocabulary covering all but 0.004% of them (<unk> rate). Documents are separated by <eos>.

Training procedure

Training is a single counting pass over the encoded corpus: n-grams of order 1-6 are packed into uint64 keys, counted per shard with np.unique and merged. It takes a few minutes on CPU and is fully deterministic. Distinct n-grams per order:

order	distinct n-grams
1	256
2	15,123
3	186,698
4	1,125,157
5	4,525,763
6	13,788,192

Evaluation

Noise ladder. Clean DTA pages (~2,000 characters) were corrupted at controlled character-error rates with Fraktur-OCR-style noise (confusions such as s->f, u<->n, e->c, plus deletions and insertions), and every version was scored. The score must track the injected damage:

condition	bpc (mean)
clean	1.95
2% CER	2.39
5% CER	3.02
10% CER	3.97
20% CER	5.63
50% CER	8.94

The score increased strictly with every damage step for 100% of individual pages, and adjacent damage levels are separated with an AUC of 0.97-1.00 - even a clean page and a 2%-CER page are ranked correctly 97% of the time.

Agreement with KenLM. Validated against a modified-Kneser-Ney character 6-gram trained with KenLM on the same corpus: Spearman correlation 0.987 across 539 clean and corrupted pages (0.955 on clean pages only), with near-identical mean bpc on clean text.

Known blind spot. Word-shuffled pages (every word intact, order destroyed) are barely penalized (+0.27 bpc vs +0.88 for the whole-page xLSTM scorer): a 6-gram window never sees enough context to notice broken word order. For detecting structural OCR damage (column merges, scrambled reading order), pair this model with a long-context scorer.

Considerations for using the model

• Scores live in the normalized character space. The tokenizer lowercases, folds historical glyphs and maps digits to 0 before scoring. Scores therefore ignore capitalization and number errors by design.
• Domain: historical German print (Early New High German to circa 1900). Scores on modern German, other languages, or non-print text will be systematically higher without indicating OCR damage.
• Ranking, not thresholds. bpc values are comparable within one model version. Absolute thresholds do not transfer to other scorers (KenLM, xLSTM) or retrained versions.
• Generation works technically (generate()), but the model is a scorer: sampled text is stupid-backoff German and only useful for sanity checks.

Licensing

The training corpus is the Deutsches Textarchiv, whose works carry per-document licenses (CC BY-SA variants, CC BY-NC, CC0 and others - see the dataset card). This model stores only aggregate character n-gram counts derived from that corpus. The modeling code and the count tables are released under Apache License 2.0.

Citation

Please cite the DTA when using this model:

@misc{dta2026,
  author       = {{Berlin-Brandenburgische Akademie der Wissenschaften}},
  title        = {Deutsches Textarchiv. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache},
  year         = {2026},
  address      = {Berlin},
  howpublished = {Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften},
  url          = {https://www.deutschestextarchiv.de/}
}

Acknowledgements

A big thank you to the whole team of the Deutsches Textarchiv at the Berlin-Brandenburg Academy of Sciences and Humanities! Also many thanks to all the partner projects, libraries and archives that contributed texts to the DTA extension corpora.

AI disclosure

This model card was drafted with Claude Fable 5 (claude-fable-5) based on the training and evaluation results produced in the dta-experiments repository, and reviewed by the model author. The training and modeling scripts carry their own AI disclosure blocks, following the rules in the ai-disclosure repository.