---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: token-classification
tags:
- spacy
- token-classification
- persian
- farsi
---

# fa_core_news_md

Persian pipeline optimized for CPU. Components: tok2vec, tagger, morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, PCT.

## Install

```bash
pip install https://huggingface.co/Phazel/fa_core_news_md/resolve/main/fa_core_news_md-3.8.0-py3-none-any.whl
```

```python
import spacy
nlp = spacy.load("fa_core_news_md")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش می‌دهد.")
print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
print([(e.text, e.label_) for e in doc.ents])
```

## Accuracy

Scored with `spacy benchmark accuracy` on the held-out PerDT test split.

| Metric | Score |
| --- | ---: |
| Tokenization accuracy | 99.96 |
| XPOS tag accuracy | 96.25 |
| UPOS tag accuracy | 96.64 |
| Morphological features | 96.64 |
| Lemma accuracy | 97.96 |
| Unlabelled attachment (UAS) | 90.52 |
| Labelled attachment (LAS) | 86.34 |
| Sentence segmentation F | 99.28 |
| NER precision | 76.56 |
| NER recall | 72.95 |
| NER F-score | 74.71 |

## Throughput

Median of repeated `nlp.pipe` passes over the 146-document PerDT test
split (23,825 tokens), timing the pipe only. Warmup pass discarded.

| Device | Batch | Words/s |
| --- | ---: | ---: |
| gpu:0 (NVIDIA GeForce 940MX, 2048 MiB) | 32 | 7,349 |
| cpu (Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz) | 32 | 5,856 |

## Sources

| Source | Author | Licence |
| --- | --- | --- |
| [UD_Persian-PerDT (PerUDT v1.0)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian | CC BY-SA 4.0 |
| [UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020) | CC BY-SA 4.0 |
| [spaCy lang/fa language data (stop words originally from HAZM)](https://github.com/explosion/spaCy/tree/master/spacy/lang/fa) | Explosion and spaCy contributors | MIT |
| [fa_floret static vectors (50k rows x 300d, floret mode, 400k Persian documents)](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 |

## Notes

Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in `performance.ents_per_type` should be read before relying on them. Multiword tokens (pronominal clitics, enclitic copulas) were merged with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md. This is the `md` tier: identical architecture to the `sm` pipeline plus static floret vectors (50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2) trained on 400,000 Persian documents. floret hashes subwords into a fixed table, so there are no out-of-vocabulary tokens and `token.has_vector` is always True. That matters for Persian, where inconsistent ZWNJ (U+200C) usage splits one word across several surface forms (mi-ravad written joined, with ZWNJ, or with a space) that a classic word-vector table would miss.