fa_floret_400k

Persian floret static vector table: 50,000 rows x 300 dimensions, minn=maxn=5, hash_count=2, trained on 400,000 Persian documents. Vectors only, no pipeline components.

This is the table used by the fa_dep_news_md / fa_core_news_md / fa_ent_news_md tier. Because floret hashes subwords rather than storing whole-word keys, there are no OOV tokens: every string gets a vector, which is what makes it usable on Persian text where inconsistent ZWNJ placement otherwise explodes the vocabulary.

Install

pip install https://huggingface.co/Phazel/fa_floret_400k/resolve/main/fa_floret_400k-0.1.0-py3-none-any.whl

Use

import spacy

nlp = spacy.load("fa_floret_400k")   # vectors only: nlp.pipe_names == []
print(nlp.vocab.vectors.shape)       # (50000, 300)
print(nlp("کتاب‌های").vector.shape)   # (300,)

To train a pipeline against this table, pass it to spaCy's --paths.vectors and set components.tok2vec.model.embed.include_static_vectors = true:

python -m spacy train config.cfg --paths.vectors fa_floret_400k

Vector table

Property	Value
Rows	50,000
Dimensions	300
Mode	floret (subword, Bloom-hashed)
minn / maxn	5 / 5
hash_count	2
Training corpus	400,000 Persian documents

Related

Package	Rows	Corpus	Used by
fa_floret_400k	50k	400k Persian documents	md tier
fa_floret_full_wiki	50k	full Persian Wikipedia dump	—
fa_floret_wiki_200k	200k	full Persian Wikipedia dump, 5 epochs	lg tier

Trained pipelines that consume these tables, plus the measured accuracy deltas each table buys, are in spacy-persian (docs/MODELS.md §6-7).

Sources and licence

Source	Author	Licence
fa_floret static vectors, 50k rows x 300d, 400k Persian documents	Kiyarash Fazeli	CC BY-SA 4.0

Released under CC BY-SA 4.0, the same licence the md pipelines that embed this table carry.