README.md

metadata

license: mit
language: he
library_name: shoshan
tags:
  - hebrew
  - lemmatization
  - retrieval
  - token-classification
pipeline_tag: token-classification

Shoshan — Hebrew lemmatizer (weights)

A context-aware Hebrew lemmatizer that does not hallucinate. It retrieves the lemma from a fixed bank, and when the top retrieval is morphologically implausible for the surface form, transduces it with a learned, form-relative edit script. Every output is a real bank entry or a bounded edit of the input word, so the model cannot emit a free-form string.

Trained only on the openly redistributable Knesset + Wikipedia portions of the IAHLT Hebrew UD treebank, plus public Hebrew lexicons.

• Code: https://github.com/ivrit/shoshan
• Demo: https://huggingface.co/spaces/noamor/shoshan-demo

Contents

folder	what
model/	the fine-tuned encoder (DictaBERT backbone) + POS head + edit-script head and inventory
bank/	the pre-encoded lemma bank (lemmas.csv + lemmas.npy, ~117.6k lemmas)

The bank is normalized to a single undotted, quote-folded form per lemma (so vowel-only variants can't produce arbitrary retrieval ties) and filtered to valid Hebrew lemmas. Inference encodes each sentence once and pools every token from it, so document lemmatization scales with the number of sentences, not tokens.

Usage

pip install git+https://github.com/ivrit/shoshan.git

from shoshan import Lemmatizer

lz = Lemmatizer.from_pretrained()        # pulls these weights, then caches
lz.lemma("המחברות", "המורה חילקה את המחברות לתלמידים בכיתה.")   # -> מחברת

Results (out-of-domain, held-out registers)

• Lemma accuracy 92.4% out-of-domain (94.3% in-domain).
• B³ consistency leads DictaBERT-lex on both precision and recall (0.965 / 0.953 vs 0.906 / 0.932).
• 0.0% low-overlap errors on unseen words, vs 12.3% for DictaBERT-lex (which predicts each lemma as a single token from its vocabulary).

DictaBERT-lex was trained on more data than is used here, including the domains held out for evaluation, so the comparison is conservative.

License and credit

Code: MIT. The encoder is fine-tuned from DictaBERT (dicta-il/dictabert) and is subject to that model's license. The lemma bank is derived from a public Hebrew lemma lexicon and the MILA morphological lexicon; see the code repository's docs/DATA_STATEMENT.md for provenance and terms. We thank Avner Algom and the IAHLT for the treebank data.