| --- |
| language: |
| - fa |
| license: cc-by-sa-4.0 |
| library_name: spacy |
| pipeline_tag: feature-extraction |
| tags: |
| - spacy |
| - feature-extraction |
| - floret |
| - word-embeddings |
| - persian |
| - farsi |
| --- |
| |
| # fa_floret_full_wiki |
| |
| Persian floret static vector table: 50,000 rows x 300 dimensions, `minn=maxn=5`, |
| `hash_count=2`, trained on the full Persian Wikipedia dump. Vectors only, no pipeline |
| components. |
|
|
| Same table size as `fa_floret_400k` over a larger corpus. No shipped pipeline uses it: the |
| `lg` tier moved to the 200,000-row `fa_floret_wiki_200k` instead, so this table is published |
| for comparison — it isolates corpus size from table size, since it shares its 50k rows with |
| `fa_floret_400k` and its corpus with `fa_floret_wiki_200k`. |
|
|
| Because floret hashes subwords rather than storing whole-word keys, there are no OOV tokens: |
| every string gets a vector, which is what makes it usable on Persian text where inconsistent |
| ZWNJ placement otherwise explodes the vocabulary. |
|
|
| ## Install |
|
|
| ```bash |
| pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl |
| ``` |
|
|
| ## Use |
|
|
| ```python |
| import spacy |
| |
| nlp = spacy.load("fa_floret_full_wiki") # vectors only: nlp.pipe_names == [] |
| print(nlp.vocab.vectors.shape) # (50000, 300) |
| print(nlp("کتابهای").vector.shape) # (300,) |
| ``` |
|
|
| To train a pipeline against this table, pass it to spaCy's `--paths.vectors` and set |
| `components.tok2vec.model.embed.include_static_vectors = true`: |
|
|
| ```bash |
| python -m spacy train config.cfg --paths.vectors fa_floret_full_wiki |
| ``` |
|
|
| ## Vector table |
|
|
| | Property | Value | |
| | --- | --- | |
| | Rows | 50,000 | |
| | Dimensions | 300 | |
| | Mode | floret (subword, Bloom-hashed) | |
| | `minn` / `maxn` | 5 / 5 | |
| | `hash_count` | 2 | |
| | Training corpus | full Persian Wikipedia dump | |
|
|
| ## Related |
|
|
| | Package | Rows | Corpus | Used by | |
| | --- | ---: | --- | --- | |
| | [`fa_floret_400k`](https://huggingface.co/Phazel/fa_floret_400k) | 50k | 400k Persian documents | `md` tier | |
| | [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50k | full Persian Wikipedia dump | — | |
| | [`fa_floret_wiki_200k`](https://huggingface.co/Phazel/fa-floret-wiki-vectors) | 200k | full Persian Wikipedia dump, 5 epochs | `lg` tier | |
|
|
| Trained pipelines that consume these tables, plus the measured accuracy deltas each table |
| buys, are in [`spacy-persian`](https://github.com/Fazel94/spacy-persian) (`docs/MODELS.md` |
| §6-7). |
|
|
| ## Sources and licence |
|
|
| | Source | Author | Licence | |
| | --- | --- | --- | |
| | [`fa_floret` static vectors, 50k rows x 300d, full Persian Wikipedia dump](https://github.com/Fazel94/spacy-persian) | Kiyarash Fazeli | CC BY-SA 4.0 | |
| | [Persian Wikipedia dump (fawiki)](https://dumps.wikimedia.org/fawiki/) | Wikipedia contributors | CC BY-SA 4.0 | |
|
|
| The table is a derivative of Persian Wikipedia text, so it is released under **CC BY-SA 4.0** |
| to match. |
|
|
