README.md

---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: feature-extraction
tags:
- floret
- fasttext
- word-vectors
- spacy
- feature-extraction
- persian
- farsi
---

# fa-floret-wiki-vectors

Floret word vectors for Persian, trained on Persian Wikipedia with
[floret-torch](https://github.com/Fazel94/floret-torch), a GPU (PyTorch)
port of [explosion/floret](https://github.com/explosion/floret).

Trained on the full `fawiki` dump (WikiExtractor, spaCy `blank("fa")`
tokenization): 8,428,449 sentences, 190,781,621 tokens.

```
--model cbow --mode floret --dim 300 --minn 5 --maxn 5 \
--hashCount 2 --bucket 200000 --neg 10 --epoch 5 --lr 0.05 \
--minCount 20 --batch 8192
```

This is the table the `fa_dep_news_lg` / `fa_core_news_lg` / `fa_ent_news_lg`
pipelines are trained against.

## Files

- **fa.floret** — hashed bucket table. Reconstructs a vector for any word,
  including ones never seen in training, via subword n-gram hashing.
  ```
  python -m spacy init vectors fa fa.floret ./fa_pipeline --mode floret
  ```
- **fa.vec** — plain word2vec text format, fixed vocabulary above
  `minCount 20`. No OOV lookup.
  ```python
  from gensim.models import KeyedVectors
  kv = KeyedVectors.load_word2vec_format("fa.vec")
  ```
- **fa_floret_wiki_200k-0.1.0-py3-none-any.whl** — installable vectors-only
  spaCy pipeline (`pipeline: []`) with `fa.floret` embedded as a 200,000-row
  x 300d table.
  ```bash
  pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
  ```
  ```python
  import spacy
  nlp = spacy.load("fa_floret_wiki_200k")
  print(nlp.vocab.vectors.shape)   # (200000, 300)
  ```
  To train against it without installing anything, point spaCy's
  `--paths.vectors` at an unpacked copy
  (`scripts/unpack_vectors.py` in
  [spacy-persian](https://github.com/Fazel94/spacy-persian)).

## Related tables

| Package | Rows | Corpus | Used by |
| --- | ---: | --- | --- |
| [`fa_floret_400k`](https://huggingface.co/Phazel/fa_floret_400k) | 50k | 400k Persian documents | `md` tier |
| [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50k | full Persian Wikipedia dump | — |
| `fa_floret_wiki_200k` (this repo) | 200k | full Persian Wikipedia dump, 5 epochs | `lg` tier |

## Licence

**CC BY-SA 4.0.** The vectors are a derivative of Persian Wikipedia text
(CC BY-SA 4.0), so the share-alike condition carries over; this is also the
licence recorded for this table in the `lg` pipelines' `meta.json`.
The `floret-torch` training tooling is MIT and imposes no condition on the
output.