Sefaria/BEREL_3.0-hebrew-error-detector

BEREL 3.0 Hebrew error detector

A token classifier that marks the words of a Rabbinic Hebrew passage that are probably errors in the digital text: OCR damage, typos, words run together or split apart, and stray characters. It is fine-tuned from dicta-il/BEREL_3.0 by Sefaria.

Purpose

Sefaria holds thousands of Hebrew texts, many of them OCR'd from printed editions, and their quality varies. The detector has two uses:

1. Point editors at likely errors in a text, so proofreading starts where it is needed.
2. Estimate the quality of a version (a text edition) from the share of its words the detector flags, to decide which versions need correcting before further processing such as machine translation. This use is still being validated; see Limitations.

It is meant for post-biblical works: Talmud, midrash, commentaries, codes, responsa and later Rabbinic literature. It is not suitable for Tanakh, Targum or manuscript-orthography texts (see Limitations).

How it was trained

Labels from real editing history. Sefaria keeps every edit made to the Hebrew of its texts. All ~1.05M edits were rebuilt into before/after pairs and classified by rules. Edits that fix an error became training data, and the word that was fixed is labelled as an error in the text before the edit:

Fix kind	What it is	Hand-checked precision (400-edit sample)
typo	changed words within edit distance 2	0.88
spacing	missing or extra space, same letters	0.94
ocr	a letter error together with a split or join, or an abbreviation mark read as a letter	0.76
junk	stray non-Hebrew characters removed	2 of 2

Excluded: word substitutions (mostly variant readings), added or dropped words, vowel-only edits, citations, formatting, edits by two bulk re-segmentation accounts, and conventions such as joining words with a maqaf. Inside a multi-word edit only the words that actually changed are labelled. This gives 29,159 windows of up to 200 words (1.4% of words labelled as errors), split 25,821 / 1,172 / 2,166 into train / validation / test by (version, book), so no book appears in two splits.

Clean pairs. Each real window is paired with its corrected version, every word labelled ok, so the model does not learn that every passage contains an error.

Synthetic errors. 60,000 windows of train-split text with injected errors, drawn from the confusion statistics of the real corrections: single-letter substitutions (mostly visual OCR confusions such as ר→ד, ס→ם, ו→י, כ→ב), deletions, insertions, joins and splits; 15% have no error. Mixed in at half the weight of real windows.

Training. AutoModelForTokenClassification with two labels (ok, error), the first word piece of each word carrying its label. Cross-entropy with the error class weighted 10×, learning rate 3e-5, batch 64, 3 epochs of 120,000 chunks, chunks of up to 256 word pieces packed on word boundaries, bf16 on one A100. The weights with the best validation PR-AUC were kept (step 5,500 of 5,625). A sweep over the synthetic weight (0, 0.5, 1.0) and the error weight (3, 10) chose this setting on validation; training 5 epochs did not improve it.

Results

Word-level, error class. The threshold (0.872) was chosen on validation and applied to test. Validation and test include each window's corrected copy.

	Validation PR-AUC	Test PR-AUC	Test F1	Test precision	Test recall
This model	0.422	0.323	0.392	0.395	0.390
Flag words not seen in the corrected training text	0.132	0.139	0.093	0.051	0.490
Flag words BEREL 3.0 finds unlikely (masked-LM)	0.050	0.040	0.112	0.071	0.272

Test recall by fix kind: spacing 0.64, typo 0.33, ocr 0.34. 42% of test errors come from one heavily proofread edition with its own conventions (Guggenheimer's Jerusalem Talmud), which explains most of the drop from validation to test.

Precision is understated. Words nobody has corrected yet are labelled ok, so a real error the model finds in an unedited part of a window counts as a false positive. A hand-check of flagged words is in progress.

How to use

The model expects the same normalization as training: remove HTML tags and niqqud/cantillation, split maqaf-joined words, keep punctuation and quote marks (geresh and gershayim help with abbreviations). Score the first piece of each word.

import re, torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "Sefaria/BEREL_3.0-hebrew-error-detector"
tok = AutoTokenizer.from_pretrained(repo)          # the fast tokenizer; BertTokenizer gives bad results with BEREL 3.0
model = AutoModelForTokenClassification.from_pretrained(repo).eval()
THRESHOLD = 0.872                                  # chosen on validation; see error_detection_config.json

TAG = re.compile(r"&lt;.*?&gt;|<[^>]*>|&nbsp;|&[a-z]+;")
NIQQUD = re.compile("[֑-ׇֽֿׁׂׅׄ]")
def normalize(word):
    return TAG.sub("", NIQQUD.sub("", word)).replace("־", " ").strip()

def flag(text):
    words = [w for w in (normalize(t) for t in text.split()) if w]
    enc = tok(words, is_split_into_words=True, truncation=True, max_length=256, return_tensors="pt")
    with torch.no_grad():
        probs = model(**enc).logits.softmax(-1)[0, :, 1]
    out, seen = [], set()
    for pos, w in enumerate(enc.word_ids()):
        if w is not None and w not in seen:          # first piece of each word
            seen.add(w)
            out.append((words[w], round(probs[pos].item(), 3), probs[pos].item() >= THRESHOLD))
    return out

for word, p, is_error in flag("ואמר רבי יוחנן שהאדם צריך ללמודתורה בכל יום"):
    print(word, p, "ERROR" if is_error else "")

Passages longer than about 200 words should be split on word boundaries into chunks of at most 256 word pieces.

Limitations

• Rare valid forms look like errors. Almost all training errors come from Rabbinic prose, so the model flags unusual but correct forms. On Tanakh (Masoretic defective spellings), Targum (Aramaic forms) and manuscript-orthography editions it flags 15 to 24 words per 1,000, nearly all false. Do not use it on those texts.
• Edition conventions (abbreviations truncated with an apostrophe, joined של, defective spelling) can be flagged.
• Not a corrector. It says which words are suspicious, not what they should be.
• Version quality scores are not validated. Across 294 versions the flagged rate does not yet agree with how much each version has been corrected on Sefaria, and there is no ground truth for version quality. Treat per-version rates as a pointer for review, compared only between versions of the same genre.
• Vowel and cantillation errors are invisible by design (niqqud is removed).

Training data

The training data was built from Sefaria's edit history and is not public (it quotes texts with mixed licenses). The training and evaluation code is in Sefaria/ML-Workflows (app/error_detection).

Citation

Please cite BEREL when using this model:

Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Eli Handel, Moshe Koppel, "Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language", 2022, arXiv:2208.01875.