File size: 2,890 Bytes
d372082 a8869be d372082 2d1bf6d a8869be d372082 a8869be | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
language:
- fa
license: cc-by-sa-4.0
library_name: spacy
pipeline_tag: feature-extraction
tags:
- spacy
- feature-extraction
- floret
- word-embeddings
- persian
- farsi
---
# fa_floret_full_wiki
Persian floret static vector table: 50,000 rows x 300 dimensions, `minn=maxn=5`,
`hash_count=2`, trained on the full Persian Wikipedia dump. Vectors only, no pipeline
components.
Same table size as `fa_floret_400k` over a larger corpus. No shipped pipeline uses it: the
`lg` tier moved to the 200,000-row `fa_floret_wiki_200k` instead, so this table is published
for comparison — it isolates corpus size from table size, since it shares its 50k rows with
`fa_floret_400k` and its corpus with `fa_floret_wiki_200k`.
Because floret hashes subwords rather than storing whole-word keys, there are no OOV tokens:
every string gets a vector, which is what makes it usable on Persian text where inconsistent
ZWNJ placement otherwise explodes the vocabulary.
## Install
```bash
pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl
```
## Use
```python
import spacy
nlp = spacy.load("fa_floret_full_wiki") # vectors only: nlp.pipe_names == []
print(nlp.vocab.vectors.shape) # (50000, 300)
print(nlp("کتابهای").vector.shape) # (300,)
```
To train a pipeline against this table, pass it to spaCy's `--paths.vectors` and set
`components.tok2vec.model.embed.include_static_vectors = true`:
```bash
python -m spacy train config.cfg --paths.vectors fa_floret_full_wiki
```
## Vector table
| Property | Value |
| --- | --- |
| Rows | 50,000 |
| Dimensions | 300 |
| Mode | floret (subword, Bloom-hashed) |
| `minn` / `maxn` | 5 / 5 |
| `hash_count` | 2 |
| Training corpus | full Persian Wikipedia dump |
## Related
| Package | Rows | Corpus | Used by |
| --- | ---: | --- | --- |
| [`fa_floret_400k`](https://huggingface.co/Phazel/fa_floret_400k) | 50k | 400k Persian documents | `md` tier |
| [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50k | full Persian Wikipedia dump | — |
| [`fa_floret_wiki_200k`](https://huggingface.co/Phazel/fa-floret-wiki-vectors) | 200k | full Persian Wikipedia dump, 5 epochs | `lg` tier |
Trained pipelines that consume these tables, plus the measured accuracy deltas each table
buys, are in [`spacy-persian`](https://github.com/Fazel94/spacy-persian) (`docs/MODELS.md`
§6-7).
## Sources and licence
| Source | Author | Licence |
| --- | --- | --- |
| [`fa_floret` static vectors, 50k rows x 300d, full Persian Wikipedia dump](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 |
| [Persian Wikipedia dump (fawiki)](https://dumps.wikimedia.org/fawiki/) | Wikipedia contributors | CC BY-SA 4.0 |
The table is a derivative of Persian Wikipedia text, so it is released under **CC BY-SA 4.0**
to match.
|