fa_floret_full_wiki

Persian floret static vector table: 50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2, trained on the full Persian Wikipedia dump. Vectors only, no pipeline components.

Same table size as fa_floret_400k over a larger corpus. No shipped pipeline uses it: the lg tier moved to the 200,000-row fa_floret_wiki_200k instead, so this table is published for comparison — it isolates corpus size from table size, since it shares its 50k rows with fa_floret_400k and its corpus with fa_floret_wiki_200k.

Because floret hashes subwords rather than storing whole-word keys, there are no OOV tokens: every string gets a vector, which is what makes it usable on Persian text where inconsistent ZWNJ placement otherwise explodes the vocabulary.

Install

pip install https://huggingface.co/Phazel/fa_floret_full_wiki/resolve/main/fa_floret_full_wiki-0.1.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_floret_full_wiki")   # vectors only: nlp.pipe_names == []
print(nlp.vocab.vectors.shape)            # (50000, 300)
print(nlp("کتاب‌های").vector.shape)        # (300,)

To train a pipeline against this table, pass it to spaCy's --paths.vectors and set components.tok2vec.model.embed.include_static_vectors = true:

python -m spacy train config.cfg --paths.vectors fa_floret_full_wiki

Vector table

Property	Value
Rows	50,000
Dimensions	300
Mode	floret (subword, Bloom-hashed)
minn / maxn	5 / 5
hash_count	2
Training corpus	full Persian Wikipedia dump

Related

Package	Rows	Corpus	Used by
fa_floret_400k	50k	400k Persian documents	md tier
fa_floret_full_wiki	50k	full Persian Wikipedia dump	—
fa_floret_wiki_200k	200k	full Persian Wikipedia dump, 5 epochs	lg tier

Trained pipelines that consume these tables, plus the measured accuracy deltas each table buys, are in spacy-persian (docs/MODELS.md §6-7).

Sources and licence

Source	Author	Licence
fa_floret static vectors, 50k rows x 300d, full Persian Wikipedia dump	Kiyarash Fazeli	CC BY-SA 4.0
Persian Wikipedia dump (fawiki)	Wikipedia contributors	CC BY-SA 4.0

The table is a derivative of Persian Wikipedia text, so it is released under CC BY-SA 4.0 to match.