README.md

---
license: apache-2.0
language:
  - de
library_name: transformers
pipeline_tag: text-generation
datasets:
  - histde/dta-documents
tags:
  - historical
  - historical-german
  - german
  - deutsches-textarchiv
  - fraktur
  - ocr
  - character-level
  - ngram
  - xlstm
  - custom_code
---

# DTA Character 6-gram

A character-level 6-gram language model of historical German (1500-1900), trained on the full
[Deutsches Textarchiv](https://huggingface.co/datasets/histde/dta-documents) and packaged as a Hugging Face
model. Its purpose is **OCR quality scoring**: the model assigns a bits-per-character (bpc) score to a page of
text, where a low score means "looks like clean historical German" and a high score indicates OCR damage
(character confusions, garbled words, noise). It was built to score the OCR quality of large digitized
newspaper corpora, page by page, on CPU and at corpus scale.

The model is a compact pure-numpy reimplementation of a KenLM-style character n-gram with
**stupid backoff** smoothing ([Brants et al. 2007](https://aclanthology.org/D07-1090/)),
wrapped in the `transformers` causal-LM interface via `trust_remote_code`.
It has **no trainable parameters**: the "weights" are sorted n-gram count tables
(13.8M distinct 6-grams from 1.37B characters of training text), and scoring is a handful of vectorized
binary searches. The bundled character tokenizer carries the full text normalization inside, so scored text
and training corpus can never diverge in preprocessing.

## Model description

* **Architecture**: character n-gram (order 6) with stupid backoff (alpha 0.4). The score of a character is
  the relative frequency of the longest matching n-gram ending at that character, multiplied by 0.4 for every
  order backed off. Because the character vocabulary has 256 entries, an n-gram fits into a single `uint64`
  key. Each order is stored as one sorted key array plus one count array, registered as model buffers in
  `model.safetensors` (~314 MB). All computation is vectorized numpy on CPU.
* **Tokenizer**: a character-level fast tokenizer (one token per character, 256-entry vocabulary with `<eos>`
  id 0 and `<unk>` id 1) whose normalizer pipeline performs the shared preprocessing: dehyphenation, Unicode
  NFC, lowercasing, long s `ſ` -> `s`, r rotunda `ꝛ` -> `r`, combining-e umlauts (`uͤ` -> `ü`), quote and dash
  unification, digits -> `0`, whitespace collapsing.
* **Interface**: standard `AutoModelForCausalLM` contract. With `labels`, the forward pass returns the mean
  negative log-likelihood over the target characters (fast scoring path). Without `labels`, it returns full
  log-score "logits" over the vocabulary, so `generate()` works as well.
* **Scores are for ranking, not calibrated probabilities.** Stupid backoff is not a normalized distribution:
  bpc values order pages by quality but are not true perplexities. For citable modified-Kneser-Ney
  perplexities, use a KenLM model instead.

## Usage

The model ships its own modeling code, so pass `trust_remote_code=True` to both loaders:

```python
import math
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "histde/dta-char-ngram"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True).eval()

page = "Die Zeitung berichtet über die neueſten Ereigniſſe in der Hauptſtadt."
enc = tokenizer(page, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    loss = model(**enc, labels=enc["input_ids"]).loss
print(f"bits per character: {loss.item() / math.log(2):.3f}")  # lower = cleaner
```

Batched scoring works with right-padding and an `attention_mask`. Padded positions are excluded from the
loss. The scoring contract is identical to the DTA character xLSTM checkpoints, so the two scorers are
interchangeable by model id.

## Training data

The `text` column of [histde/dta-documents](https://huggingface.co/datasets/histde/dta-documents): the
historical, layout-faithful transcriptions of all 5,480 DTA works (1472-1987), about 1.38B characters.
After normalization the training stream contains **1.37B characters** with a 256-entry character vocabulary
covering all but 0.004% of them (`<unk>` rate). Documents are separated by `<eos>`.

## Training procedure

Training is a single counting pass over the encoded corpus: n-grams of order 1-6 are packed into `uint64`
keys, counted per shard with `np.unique` and merged. It takes a few minutes on CPU and is fully
deterministic. Distinct n-grams per order:

| order | distinct n-grams |
|:------|-----------------:|
| 1     |              256 |
| 2     |           15,123 |
| 3     |          186,698 |
| 4     |        1,125,157 |
| 5     |        4,525,763 |
| 6     |       13,788,192 |

## Evaluation

**Noise ladder.** Clean DTA pages (~2,000 characters) were corrupted at controlled character-error rates
with Fraktur-OCR-style noise (confusions such as `s`->`f`, `u`<->`n`, `e`->`c`, plus deletions and
insertions), and every version was scored. The score must track the injected damage:

| condition | bpc (mean) |
|:----------|-----------:|
| clean     |       1.95 |
| 2% CER    |       2.39 |
| 5% CER    |       3.02 |
| 10% CER   |       3.97 |
| 20% CER   |       5.63 |
| 50% CER   |       8.94 |

The score increased strictly with every damage step for **100% of individual pages**, and adjacent damage
levels are separated with an AUC of 0.97-1.00 - even a clean page and a 2%-CER page are ranked correctly
97% of the time.

**Agreement with KenLM.** Validated against a modified-Kneser-Ney character 6-gram trained with KenLM on the
same corpus: Spearman correlation **0.987** across 539 clean and corrupted pages (0.955 on clean pages
only), with near-identical mean bpc on clean text.

**Known blind spot.** Word-shuffled pages (every word intact, order destroyed) are barely penalized
(+0.27 bpc vs +0.88 for the whole-page xLSTM scorer): a 6-gram window never sees enough context to notice
broken word order. For detecting structural OCR damage (column merges, scrambled reading order), pair this
model with a long-context scorer.

## Considerations for using the model

* **Scores live in the normalized character space.** The tokenizer lowercases, folds historical glyphs and
  maps digits to `0` before scoring. Scores therefore ignore capitalization and number errors by design.
* **Domain**: historical German print (Early New High German to circa 1900). Scores on modern German, other
  languages, or non-print text will be systematically higher without indicating OCR damage.
* **Ranking, not thresholds.** bpc values are comparable within one model version. Absolute thresholds do
  not transfer to other scorers (KenLM, xLSTM) or retrained versions.
* Generation works technically (`generate()`), but the model is a scorer: sampled text is stupid-backoff
  German and only useful for sanity checks.

## Licensing

The training corpus is the Deutsches Textarchiv, whose works carry per-document licenses (CC BY-SA variants,
CC BY-NC, CC0 and others - see the [dataset card](https://huggingface.co/datasets/histde/dta-documents)).
This model stores only aggregate character n-gram counts derived from that corpus. The modeling code and the
count tables are released under Apache License 2.0.

## Citation

Please cite the DTA when using this model:

```bibtex
@misc{dta2026,
  author       = {{Berlin-Brandenburgische Akademie der Wissenschaften}},
  title        = {Deutsches Textarchiv. Grundlage für ein Referenzkorpus der neuhochdeutschen Sprache},
  year         = {2026},
  address      = {Berlin},
  howpublished = {Herausgegeben von der Berlin-Brandenburgischen Akademie der Wissenschaften},
  url          = {https://www.deutschestextarchiv.de/}
}
```

## Acknowledgements

A big thank you to the whole team of the Deutsches Textarchiv at the Berlin-Brandenburg Academy of Sciences
and Humanities! Also many thanks to all the partner projects, libraries and archives that contributed texts to the DTA
extension corpora.

## AI disclosure

This model card was drafted with Claude Fable 5 (claude-fable-5) based on the training and evaluation results
produced in the [dta-experiments](https://github.com/histde/dta-experiments) repository, and reviewed by the
model author. The training and modeling scripts carry their own AI disclosure blocks, following the rules in
the [ai-disclosure](https://github.com/stefan-it/ai-disclosure) repository.