File size: 2,625 Bytes
8d78e4c ebf5a7f 8d78e4c ebf5a7f 8d78e4c 60d18d6 8d78e4c 60d18d6 8d78e4c 60d18d6 8d78e4c ebf5a7f 8d78e4c 60d18d6 8d78e4c 60d18d6 8d78e4c ebf5a7f 8d78e4c ebf5a7f 8d78e4c ebf5a7f 8d78e4c ebf5a7f 8d78e4c ebf5a7f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: feature-extraction
tags:
- floret
- fasttext
- word-vectors
- spacy
- feature-extraction
- persian
- farsi
---
# fa-floret-wiki-vectors
Floret word vectors for Persian, trained on Persian Wikipedia with
[floret-torch](https://github.com/Fazel94/floret-torch), a GPU (PyTorch)
port of [explosion/floret](https://github.com/explosion/floret).
Trained on the full `fawiki` dump (WikiExtractor, spaCy `blank("fa")`
tokenization): 8,428,449 sentences, 190,781,621 tokens.
```
--model cbow --mode floret --dim 300 --minn 5 --maxn 5 \
--hashCount 2 --bucket 200000 --neg 10 --epoch 5 --lr 0.05 \
--minCount 20 --batch 8192
```
This is the table the `fa_dep_news_lg` / `fa_core_news_lg` / `fa_ent_news_lg`
pipelines are trained against.
## Files
- **fa.floret** — hashed bucket table. Reconstructs a vector for any word,
including ones never seen in training, via subword n-gram hashing.
```
python -m spacy init vectors fa fa.floret ./fa_pipeline --mode floret
```
- **fa.vec** — plain word2vec text format, fixed vocabulary above
`minCount 20`. No OOV lookup.
```python
from gensim.models import KeyedVectors
kv = KeyedVectors.load_word2vec_format("fa.vec")
```
- **fa_floret_wiki_200k-0.1.0-py3-none-any.whl** — installable vectors-only
spaCy pipeline (`pipeline: []`) with `fa.floret` embedded as a 200,000-row
x 300d table.
```bash
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
```
```python
import spacy
nlp = spacy.load("fa_floret_wiki_200k")
print(nlp.vocab.vectors.shape) # (200000, 300)
```
To train against it without installing anything, point spaCy's
`--paths.vectors` at an unpacked copy
(`scripts/unpack_vectors.py` in
[spacy-persian](https://github.com/Fazel94/spacy-persian)).
## Related tables
| Package | Rows | Corpus | Used by |
| --- | ---: | --- | --- |
| [`fa_floret_400k`](https://huggingface.co/Phazel/fa_floret_400k) | 50k | 400k Persian documents | `md` tier |
| [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50k | full Persian Wikipedia dump | — |
| `fa_floret_wiki_200k` (this repo) | 200k | full Persian Wikipedia dump, 5 epochs | `lg` tier |
## Licence
**CC BY-SA 4.0.** The vectors are a derivative of Persian Wikipedia text
(CC BY-SA 4.0), so the share-alike condition carries over; this is also the
licence recorded for this table in the `lg` pipelines' `meta.json`.
The `floret-torch` training tooling is MIT and imposes no condition on the
output.
|