|
|
--- |
|
|
license: cc-by-sa-4.0 |
|
|
language: |
|
|
- bg |
|
|
- cs |
|
|
- da |
|
|
- de |
|
|
- el |
|
|
- en |
|
|
- es |
|
|
- et |
|
|
- fi |
|
|
- fr |
|
|
- ga |
|
|
- hr |
|
|
- hu |
|
|
- it |
|
|
- lt |
|
|
- lv |
|
|
- mt |
|
|
- nl |
|
|
- pl |
|
|
- pt |
|
|
- ro |
|
|
- sk |
|
|
- sl |
|
|
- sv |
|
|
--- |
|
|
# Dactory models |
|
|
|
|
|
## Model description |
|
|
|
|
|
This is a set of fastText-based models to evaluate the quality and domain of text, in the 24 official languages of the European Union. |
|
|
The main usage of these models is to preprocess data from the Common Crawl project, to obtain a training set for large language models. |
|
|
These models can be used as part of the dactory pipeline, released by Kyutai to process Common Crawl. |
|
|
|
|
|
There is one model per language, and each model is a multilabel classifier with the eight following labels: |
|
|
random webpages (`rand`), Wikipedia articles (`wiki`), textbooks (`books`), scientific articles from pes2o (`science`), |
|
|
Stack Exchange websites related to STEM (`stem`), Humanities (`hum`), pop culture (`pop`) and life advices (`life`). |
|
|
The models were trained to distinguish lines sampled uniformly from these different sources. |
|
|
To get training data for the languages other than English, we translated the English training set with MADLAD, except for the `rand` and `wiki` labels, for which data is readily available in all languages. |
|
|
|
|
|
* **Model name**: Dactory models |
|
|
* **Languages**: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Irish, Croatian, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovenian, Swedish |
|
|
* **Developed by**: Kyutai |
|
|
* **Model type**: Classification |
|
|
* **License**: CC-BY-SA 4.0 |
|
|
* **Version**: 1.0 |
|
|
* **Released**: April 2025 |
|
|
|
|
|
## Use cases |
|
|
|
|
|
These models can we used to evaluate the quality of text, by estimating how similar it is to text from high quality sources. |
|
|
In particular, one can take the score corresponding to the `rand` label as an estimate of the text quality. |
|
|
They can also be used to organize a collection of documents, by similarity to the different data sources used to train the model. |
|
|
For example, a large language model trained mostly on documents labeled as `books` will perform well on multi-choice Q&A benchmarks such as MMLU, while a LLM trained mostly on documents labeled as `wiki` will perform well on general knowledge Q&A benchmark such as TriviaQA. |
|
|
|
|
|
## How to use |
|
|
|
|
|
You can download the files locally by using the [huggingface-hub Python package](https://huggingface.co/docs/hub/en/models-downloading). |
|
|
|
|
|
For example: |
|
|
|
|
|
```python |
|
|
import fasttext |
|
|
from huggingface_hub import hf_hub_download |
|
|
|
|
|
local_path = hf_hub_download(repo_id="kyutai/dactory-models", filename="filter_en.bin") |
|
|
model = fasttext.load_model(local_path) |
|
|
print(model.predict("A computer scientist is a scientist who specializes in the academic study of computer science.")) |
|
|
``` |
