Add/refresh README.md
Browse files
README.md
CHANGED
|
@@ -1,14 +1,17 @@
|
|
| 1 |
---
|
| 2 |
-
language:
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
| 4 |
tags:
|
| 5 |
- floret
|
| 6 |
- fasttext
|
| 7 |
- word-vectors
|
| 8 |
- spacy
|
|
|
|
| 9 |
- persian
|
| 10 |
- farsi
|
| 11 |
-
library_name: spacy
|
| 12 |
---
|
| 13 |
|
| 14 |
# fa-floret-wiki-vectors
|
|
@@ -26,6 +29,9 @@ tokenization): 8,428,449 sentences, 190,781,621 tokens.
|
|
| 26 |
--minCount 20 --batch 8192
|
| 27 |
```
|
| 28 |
|
|
|
|
|
|
|
|
|
|
| 29 |
## Files
|
| 30 |
|
| 31 |
- **fa.floret** — hashed bucket table. Reconstructs a vector for any word,
|
|
@@ -39,14 +45,34 @@ tokenization): 8,428,449 sentences, 190,781,621 tokens.
|
|
| 39 |
from gensim.models import KeyedVectors
|
| 40 |
kv = KeyedVectors.load_word2vec_format("fa.vec")
|
| 41 |
```
|
| 42 |
-
- **
|
| 43 |
-
fa.floret embedded
|
|
|
|
| 44 |
```bash
|
| 45 |
-
pip install
|
| 46 |
```
|
| 47 |
```python
|
| 48 |
-
import
|
| 49 |
-
nlp =
|
|
|
|
| 50 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- fa
|
| 4 |
+
license: cc-by-sa-4.0
|
| 5 |
+
library_name: spacy
|
| 6 |
+
pipeline_tag: feature-extraction
|
| 7 |
tags:
|
| 8 |
- floret
|
| 9 |
- fasttext
|
| 10 |
- word-vectors
|
| 11 |
- spacy
|
| 12 |
+
- feature-extraction
|
| 13 |
- persian
|
| 14 |
- farsi
|
|
|
|
| 15 |
---
|
| 16 |
|
| 17 |
# fa-floret-wiki-vectors
|
|
|
|
| 29 |
--minCount 20 --batch 8192
|
| 30 |
```
|
| 31 |
|
| 32 |
+
This is the table the `fa_dep_news_lg` / `fa_core_news_lg` / `fa_ent_news_lg`
|
| 33 |
+
pipelines are trained against.
|
| 34 |
+
|
| 35 |
## Files
|
| 36 |
|
| 37 |
- **fa.floret** — hashed bucket table. Reconstructs a vector for any word,
|
|
|
|
| 45 |
from gensim.models import KeyedVectors
|
| 46 |
kv = KeyedVectors.load_word2vec_format("fa.vec")
|
| 47 |
```
|
| 48 |
+
- **fa_floret_wiki_200k-0.1.0-py3-none-any.whl** — installable vectors-only
|
| 49 |
+
spaCy pipeline (`pipeline: []`) with `fa.floret` embedded as a 200,000-row
|
| 50 |
+
x 300d table.
|
| 51 |
```bash
|
| 52 |
+
pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
|
| 53 |
```
|
| 54 |
```python
|
| 55 |
+
import spacy
|
| 56 |
+
nlp = spacy.load("fa_floret_wiki_200k")
|
| 57 |
+
print(nlp.vocab.vectors.shape) # (200000, 300)
|
| 58 |
```
|
| 59 |
+
To train against it without installing anything, point spaCy's
|
| 60 |
+
`--paths.vectors` at an unpacked copy
|
| 61 |
+
(`scripts/unpack_vectors.py` in
|
| 62 |
+
[spacy-persian](https://github.com/Fazel94/spacy-persian)).
|
| 63 |
+
|
| 64 |
+
## Related tables
|
| 65 |
+
|
| 66 |
+
| Package | Rows | Corpus | Used by |
|
| 67 |
+
| --- | ---: | --- | --- |
|
| 68 |
+
| [`fa_floret_400k`](https://huggingface.co/Phazel/fa_floret_400k) | 50k | 400k Persian documents | `md` tier |
|
| 69 |
+
| [`fa_floret_full_wiki`](https://huggingface.co/Phazel/fa_floret_full_wiki) | 50k | full Persian Wikipedia dump | — |
|
| 70 |
+
| `fa_floret_wiki_200k` (this repo) | 200k | full Persian Wikipedia dump, 5 epochs | `lg` tier |
|
| 71 |
+
|
| 72 |
+
## Licence
|
| 73 |
|
| 74 |
+
**CC BY-SA 4.0.** The vectors are a derivative of Persian Wikipedia text
|
| 75 |
+
(CC BY-SA 4.0), so the share-alike condition carries over; this is also the
|
| 76 |
+
licence recorded for this table in the `lg` pipelines' `meta.json`.
|
| 77 |
+
The `floret-torch` training tooling is MIT and imposes no condition on the
|
| 78 |
+
output.
|