fa_dep_news_md

Persian dependency pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, trainable_lemmatizer, parser. No NER, see fa_core_news_md.

Install

pip install https://huggingface.co/Phazel/fa_dep_news_md/resolve/main/fa_dep_news_md-3.8.0-py3-none-any.whl

import spacy
nlp = spacy.load("fa_dep_news_md")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.")
print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
print([(e.text, e.label_) for e in doc.ents])

Accuracy

Scored with spacy benchmark accuracy on the held-out PerDT test split.

Metric	Score
Tokenization accuracy	99.96
XPOS tag accuracy	96.25
UPOS tag accuracy	96.64
Morphological features	96.64
Lemma accuracy	97.96
Unlabelled attachment (UAS)	90.52
Labelled attachment (LAS)	86.34
Sentence segmentation F	99.28

Throughput

Median of repeated nlp.pipe passes over the 146-document PerDT test split (23,825 tokens), timing the pipe only. Warmup pass discarded.

Device	Batch	Words/s
gpu:0 (NVIDIA GeForce 940MX, 2048 MiB)	32	18,512
cpu (Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz)	32	9,528

Sources

Source	Author	Licence
UD_Persian-PerDT (PerUDT v1.0)	Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian	CC BY-SA 4.0
spaCy lang/fa language data (stop words originally from HAZM)	Explosion and spaCy contributors	MIT
fa_floret static vectors (50k rows x 300d, floret mode, 400k Persian documents)	Kiyarash Fazeli	CC BY-SA 4.0

Notes

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. Multiword tokens (pronominal clitics, enclitic copulas) were merged with spacy convert --merge-subtokens, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md. This is the md tier: identical architecture to the sm pipeline plus static floret vectors (50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2) trained on 400,000 Persian documents. floret hashes subwords into a fixed table, so there are no out-of-vocabulary tokens and token.has_vector is always True. That matters for Persian, where inconsistent ZWNJ (U+200C) usage splits one word across several surface forms (mi-ravad written joined, with ZWNJ, or with a space) that a classic word-vector table would miss.