bank: niqqud-normalized + junk-removed (117,640 lemmas); v0.2.0 card

README.md

@@ -29,7 +29,12 @@ IAHLT Hebrew UD treebank, plus public Hebrew lexicons.
 | folder | what |
 |---|---|
 | `model/` | the fine-tuned encoder (DictaBERT backbone) + POS head + edit-script head and inventory |
-| `bank/`  | the pre-encoded lemma bank (`lemmas.csv` + `lemmas.npy`, ~118k lemmas) |
+| `bank/`  | the pre-encoded lemma bank (`lemmas.csv` + `lemmas.npy`, ~117.6k lemmas) |
+
+The bank is normalized to a single undotted, quote-folded form per lemma (so vowel-only
+variants can't produce arbitrary retrieval ties) and filtered to valid Hebrew lemmas.
+Inference encodes each sentence once and pools every token from it, so document lemmatization
+scales with the number of sentences, not tokens.
 
 ## Usage
 
@@ -41,7 +46,7 @@ pip install git+https://github.com/ivrit/shoshan.git
 from shoshan import Lemmatizer
 
 lz = Lemmatizer.from_pretrained()        # pulls these weights, then caches
-lz.lemma("המטענים", "הוא פרק את המטענים מהמשאית.")   # -> מטען
+lz.lemma("המחברות", "המורה חילקה את המחברות לתלמידים בכיתה.")   # -> מחברת
 ```
 
 ## Results (out-of-domain, held-out registers)