File size: 4,642 Bytes
28c45f9 fccedcd c15bda0 fccedcd 28c45f9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | ---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: token-classification
tags:
- spacy
- token-classification
- persian
- farsi
---
# fa_core_news_trf
Persian pipeline built on a fine-tuned HooshvareLab/bert-base-parsbert-uncased transformer. Components: transformer, tagger, morphologizer, trainable_lemmatizer, parser, ner. Entity labels: PER, LOC, ORG, DAT, MON, TIM, PCT. GPU recommended.
## Install
```bash
pip install https://huggingface.co/Phazel/fa_core_news_trf/resolve/main/fa_core_news_trf-1.0.0-py3-none-any.whl
```
```python
import spacy
nlp = spacy.load("fa_core_news_trf")
doc = nlp("شرکت ایران خودرو اعلام کرد که تولید خود را افزایش میدهد.")
print([(t.text, t.pos_, t.lemma_, t.dep_) for t in doc])
print([(e.text, e.label_) for e in doc.ents])
```
## Accuracy
Scored with `spacy benchmark accuracy` on the held-out PerDT test split.
| Metric | Score |
| --- | ---: |
| Tokenization accuracy | 99.96 |
| XPOS tag accuracy | 97.62 |
| UPOS tag accuracy | 97.63 |
| Morphological features | 97.82 |
| Lemma accuracy | 97.31 |
| Unlabelled attachment (UAS) | 93.87 |
| Labelled attachment (LAS) | 90.79 |
| Sentence segmentation F | 97.35 |
| NER precision | 84.06 |
| NER recall | 81.76 |
| NER F-score | 82.89 |
## Throughput
Median of repeated `nlp.pipe` passes over the 146-document PerDT test
split (23,825 tokens), timing the pipe only. Warmup pass discarded.
| Device | Batch | Words/s |
| --- | ---: | ---: |
| gpu:0 (Tesla T4, 15360 MiB) | 32 | 8,320 |
| gpu:0 (NVIDIA GeForce 940MX, 2048 MiB) | 32 | 1,106 |
| cpu (Intel(R) Xeon(R) CPU @ 2.00GHz) | 32 | 336 |
| cpu (Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz) | 32 | 187 |
## Sources
| Source | Author | Licence |
| --- | --- | --- |
| [UD_Persian-PerDT (PerUDT v1.0)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | Mohammad Sadegh Rasooli, Pegah Safari, Amirsaeid Moloodi, Alireza Nourian | CC BY-SA 4.0 |
| [UD_Persian-PerDT NER layer (not-to-release/Dadegan with NER tag/)](https://github.com/UniversalDependencies/UD_Persian-PerDT) | PerDT authors, tagged with Beheshti-NER (Taher, Hoseini, Shamsfard 2020) | CC BY-SA 4.0 |
| [spaCy lang/fa language data (stop words originally from HAZM)](https://github.com/explosion/spaCy/tree/master/spacy/lang/fa) | Explosion and spaCy contributors | MIT |
| [HooshvareLab/bert-base-parsbert-uncased](https://huggingface.co/HooshvareLab/bert-base-parsbert-uncased) | Hooshvare Team | no licence stated on the model card |
## Notes
Trained on UD_Persian-PerDT, licensed CC BY-SA 4.0; this pipeline is therefore distributed under CC BY-SA 4.0 with attribution to the treebank authors. The ner component is trained on the NER layer shipped in UD_Persian-PerDT's not-to-release/ directory, so it shares the treebank's genre, tokenization and licence. Those labels are SILVER: the treebank README states they were produced by the BERT-based Beheshti-NER tagger (Taher et al., 2020) with manual corrections to extend recall. They were transferred onto this pipeline's tokenization by difflib alignment at a 99.86% transfer rate (scripts/transfer_perdt_ner.py); spans that could not be aligned exactly were dropped rather than guessed. Labels PER, LOC, ORG and DAT have 1,300 or more training examples each; MON (205), TIM (135) and PCT (121) are thin and their scores in `performance.ents_per_type` should be read before relying on them. Multiword tokens (pronominal clitics, enclitic copulas) were merged with `spacy convert --merge-subtokens`, so a small number of XPOS tags are composite (e.g. N_IANM_PR_JOPER) and ~1.5% of lemmas contain a space. doc.noun_chunks under-fires on this pipeline: spacy/lang/fa/syntax_iterators.py matches ClearNLP labels that do not exist in Universal Dependencies, see docs/upstream/fa-noun-chunks.md. This is the `trf` tier: no static vectors. Contextual embeddings come from a fine-tuned HooshvareLab/bert-base-parsbert-uncased (no licence stated on the model card) via spacy-transformers, shared by every component through a TransformerListener, so one encoder forward pass serves the tagger, morphologizer, lemmatizer, parser and ner. Unlike the sm/md/lg tiers the ner is trained jointly rather than sourced, because a shared encoder cannot be fine-tuned twice and then merged. GPU is strongly recommended for both training and inference. REDISTRIBUTION WARNING: HooshvareLab/bert-base-parsbert-uncased states no licence, so this wheel embeds weights whose terms are unknown and must not be republished. Retrain against HooshvareLab/roberta-fa-zwnj-base (Apache-2.0) for a publishable artifact.
|