| --- |
| language: |
| - fa |
| license: cc-by-sa-4.0 |
| library_name: spacy |
| pipeline_tag: token-classification |
| tags: |
| - spacy |
| - token-classification |
| - persian |
| - farsi |
| --- |
| |
| # fa_core_news_md |
| |
| Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, PCT. |
|
|
| ## Install |
|
|
| ```bash |
| pip install https://huggingface.co/Phazel/fa_core_news_md/resolve/main/fa_core_news_md-3.8.0-py3-none-any.whl |
| ``` |
|
|
| ```python |
| import spacy |
| nlp = spacy.load("fa_core_news_md") |
| doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش میدهد.") |
| print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc]) |
| print([(e.text, e.label_) for e in doc.ents]) |
| ``` |
|
|
| ## Accuracy |
|
|
| Scored with `spacy benchmark accuracy` on the held-out PerDT test split. |
|
|
| | Metric | Score | |
| | --- | ---: | |
| | Tokenization accuracy | 99.96 | |
| | XPOS tag accuracy | 96.25 | |
| | UPOS tag accuracy | 96.64 | |
| | Morphological features | 96.64 | |
| | Lemma accuracy | 97.96 | |
| | Unlabelled attachment (UAS) | 90.52 | |
| | Labelled attachment (LAS) | 86.34 | |
| | Sentence segmentation F | 99.28 | |
| | NER precision | 76.56 | |
| | NER recall | 72.95 | |
| | NER F-score | 74.71 | |
|
|
| ## Throughput |
|
|
| Median of repeated `nlp.pipe` passes over the 146-document PerDT test |
| split (23,825 tokens), timing the pipe only. Warmup pass discarded. |
|
|
| | Device | Batch | Words/s | |
| | --- | ---: | ---: | |
| | gpu:0 (NVIDIA GeForce 940MX, 2048 MiB) | 32 | 7,349 | |
| | cpu (Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz) | 32 | 5,856 | |
|
|
| ## Sources |
|
|
| | Source | Author | Licence | |
| | --- | --- | --- | |
| | [UD_Persian-PerDT (PerUDT v1.0)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian | CC BY-SA 4.0 | |
| | [UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020) | CC BY-SA 4.0 | |
| | [spaCy lang/fa language data (stop words originally from HAZM)](https://github.com/explosion/spaCy/tree/master/spacy/lang/fa) | Explosion and spaCy contributors | MIT | |
| | [fa_floret static vectors (50k rows x 300d, floret mode, 400k Persian documents)](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 | |
|
|
| ## Notes |
|
|
| Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in `performance.ents_per_type` should be read before relying on them. Multiword tokens (pronominal clitics, enclitic copulas) were merged with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md. This is the `md` tier: identical architecture to the `sm` pipeline plus static floret vectors (50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2) trained on 400,000 Persian documents. floret hashes subwords into a fixed table, so there are no out-of-vocabulary tokens and `token.has_vector` is always True. That matters for Persian, where inconsistent ZWNJ (U+200C) usage splits one word across several surface forms (mi-ravad written joined, with ZWNJ, or with a space) that a classic word-vector table would miss. |
|
|
