README.md

---
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
tags:
- language-identification
- text-classification
- multilingual
- byte-level
- onnx
language:
- multilingual
datasets:
- baobabtech/millid-dataset
metrics:
- f1
- accuracy
model-index:
- name: millid-tiny-v1.5
  results:
  - task:
      type: text-classification
      name: Language identification
    dataset:
      name: MillID test split (the full build)
      type: baobabtech/millid-dataset
      split: test
    metrics:
    - type: f1
      name: Macro F1
      value: 0.9554
    - type: accuracy
      name: Top-1 accuracy
      value: 0.9587
  - task:
      type: text-classification
      name: Language identification
    dataset:
      name: UDHR-LID, the languages MillID knows
      type: baobabtech/millid-dataset
      config: udhr
      split: test
    metrics:
    - type: f1
      name: Macro F1
      value: 0.8764
    - type: accuracy
      name: Top-1 accuracy
      value: 0.8492
---

<p align="center">
  <a href="https://github.com/baobab-tech/millid" target="_blank">GitHub</a> |
  <a href="https://huggingface.co/baobabtech/millid" target="_blank">Leaderboard</a> |
  <a href="https://huggingface.co/datasets/baobabtech/millid-dataset" target="_blank">Dataset</a> |
  <a href="https://github.com/baobab-tech/millid/blob/main/training/ARCHITECTURE.md" target="_blank">Architecture</a>
  <br>
  <b>License</b>: <a href="https://www.apache.org/licenses/LICENSE-2.0" target="_blank">Apache 2.0</a> | <b>Authors</b>: <a href="https://huggingface.co/baobabtech" target="_blank">Baobab Tech</a>
</p>

# MillID tiny v1.5

**MillID** (MultiLingual Language IDentifier) is an open language identification model built by Baobab Tech. It names the language of a passage of text, from a table cell to a paragraph, among 1,641 language-script labels: 1,550 languages written in 56 scripts, and 37 labels for text in a script whose language it cannot name (`und_<Script>`). MillID tiny v1.5 has 2.9M parameters and an ONNX file of 11 MB.

Small enough to run on a CPU, a phone or in a web page (491 passages a second on four Arm cores, 48 a second on one laptop thread), it identifies language where the text is, with no server and no GPU.

MillID offers:

* **Language and script:** labels are ISO 639-3 plus ISO 15924 script, so Serbian in Cyrillic (`srp_Cyrl`) and in Latin (`srp_Latn`) are told apart. Many of the languages are low-resource, with a few hundred passages each to learn from.
* **No tokenizer:** it reads UTF-8 bytes, each with its Unicode script and the character n-grams ending there, so any script is input it can read and nothing is an unknown token.
* **Short text:** a quarter of its training passages were cut to 20 to 80 characters, the length of a heading or a table cell, where language identification is hardest.
* **Whole documents:** `identify_document` reads text of any length in windows and returns the document's languages and the segments of each.
* **No server, no PyTorch:** one ONNX file and a small encoder, in Python (NumPy) or JavaScript (Node and browsers); the two are tested to give the same answers.
* **Three sizes, one label set:** tiny v1.5, medium v1 and large v1.5 share their labels, so changing size changes no code.

### Model overview

| **Parameters** | Total | 2.9M |
|---|---|---|
| | Embeddings | 935K |
| | Convolution stem | 38K |
| | Transformer blocks | 1.3M |
| | Language head | 625K |
| **Architecture** | Blocks | 3 |
| | Model dimension | 192 |
| | Attention heads | 3 |
| | Bytes per token | 1 |
| | N-gram buckets | 4,096 |
| | Normalisation | RMSNorm, pre-norm |
| | Positions | RoPE |
| | Feed-forward | SwiGLU |
| | Pooling | Masked mean |
| **Input and output** | Input | UTF-8 bytes, NFKC |
| | Passage length | Up to 255 bytes |
| | Labels | 1,641 |
| | Runtime | ONNX (11 MB) |

## Benchmark results

MillID was evaluated on two test sets: `test`, passages from documents never seen in training, split from the training data by document; and UDHR-LID (`udhr`), the Universal Declaration of Human Rights labelled by its translators. It is compared with [GlotLID](https://huggingface.co/cis-lmu/glotlid) and Meta's [fastText language identifier](https://huggingface.co/facebook/fasttext-language-identification) (NLLB, 218 labels), each restricted to MillID's labels, its best answer among them: the like-for-like comparison.

### Overall results

| Set | Languages | Metric | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `test` | 1,621 | Macro F1 | 0.9554 | 0.9755 | 0.0659 |
| | | Top-1 | 0.9587 | 0.9783 | 0.2515 |
| `test-common` | 213 | Macro F1 | 0.9513 | 0.9846 | 0.9173 |
| | | Top-1 | 0.9396 | 0.9749 | 0.9352 |
| `udhr` | 358 | Macro F1 | 0.8764 | 0.8873 | 0.3499 |
| | | Top-1 | 0.8492 | 0.8614 | 0.4629 |
| `udhr-common` | 170 | Macro F1 | 0.9513 | 0.9659 | 0.9329 |
| | | Top-1 | 0.9123 | 0.9338 | 0.9195 |
| `udhr-all` | 429 | Macro F1 | 0.7149 | 0.7245 | 0.2797 |
| | | Top-1 | 0.7274 | 0.7378 | 0.3964 |

* `test`: every language, 1,621 of them. `test-common`: the languages all three systems know, the head-to-head on equal terms, since fastText cannot answer most of MillID's.
* `udhr`: the Declaration in the languages MillID knows. `udhr-common`: in the languages all three know. `udhr-all`: in every language it has, MillID's unknown ones counted wrong.
* GlotLID's labels came from training on data that includes UDHR, and it labelled the FinePDFs and FineWeb-2 passages in `test`, so its scores are optimistic on both.

### The three MillID models

Macro F1 on `test` and `udhr`; this model in bold. All three, language by language, are on the [leaderboard](https://huggingface.co/baobabtech/millid).

| Model | Parameters | ONNX | Test F1 | UDHR F1 | Runs on |
|---|---|---|---|---|---|
| **MillID tiny v1.5** | 2.9M | 11 MB | 0.9554 | 0.8764 | CPU, web page |
| [MillID medium v1](https://huggingface.co/baobabtech/millid-medium-v1) | 115M | 455 MB | 0.9576 | 0.8780 | GPU |
| [MillID large v1.5](https://huggingface.co/baobabtech/millid-large-v1.5) | 193M | 764 MB | 0.9646 | 0.8812 | GPU |

### By passage length

Top-1 accuracy on passages of 20 to 80 characters (short), up to 300 (sentence) and up to 1,000 (paragraph):

| Set | System | Short | Sentence | Paragraph |
|---|---|---|---|---|
| `validation` | MillID | 0.9312 | 0.9788 | 0.9850 |
| `test` | MillID | 0.9333 | 0.9797 | 0.9833 |
| `test` | GlotLID | 0.9674 | 0.9858 | 0.9912 |
| `test` | fastText | 0.2764 | 0.2506 | 0.1959 |
| `test-common` | MillID | 0.9125 | 0.9685 | 0.9725 |
| `test-common` | GlotLID | 0.9609 | 0.9885 | 0.9949 |
| `test-common` | fastText | 0.9125 | 0.9581 | 0.9655 |

### MillID on each set

| Set | Top-1 | Top-3 | Top-5 | Macro P | Macro R | Macro F1 | Median F1 |
|---|---|---|---|---|---|---|---|
| `validation` | 0.9578 | 0.9889 | 0.9933 | 0.9539 | 0.9591 | 0.9551 | 0.9822 |
| `test` | 0.9587 | 0.9890 | 0.9932 | 0.9550 | 0.9588 | 0.9554 | 0.9825 |
| `test-common` | 0.9396 | 0.9857 | 0.9909 | 0.9709 | 0.9353 | 0.9513 | 0.9726 |
| `udhr` | 0.8492 | 0.9117 | 0.9257 | 0.9148 | 0.8736 | 0.8764 | 0.9907 |
| `udhr-common` | 0.9123 | 0.9606 | 0.9655 | 0.9794 | 0.9393 | 0.9513 | 1.0000 |
| `udhr-all` | 0.7274 | 0.7809 | 0.7928 | 0.7340 | 0.7290 | 0.7149 | 0.9649 |

#### Per language: test

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **103** languages, worse on **869** and within 0.01 on 649; against fastText, MillID is better on **1616** languages, worse on **0** and within 0.01 on 5. Every language, with precision, recall and the unrestricted scores, is in `reports/test/leaderboard.csv`.

The 20 languages with the most passages:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `fra_Latn` | French | 1652 | 0.929 | 0.928 | 0.808 |
| `eng_Latn` | English | 1444 | 0.917 | 0.859 | 0.648 |
| `spa_Latn` | Spanish | 1136 | 0.930 | 0.951 | 0.574 |
| `tam_Taml` | Tamil | 754 | 0.997 | 0.993 | 0.965 |
| `arb_Arab` | Arabic (Modern Standard) | 745 | 0.814 | 0.785 | 0.512 |
| `por_Latn` | Portuguese | 657 | 0.947 | 0.949 | 0.755 |
| `kan_Knda` | Kannada | 601 | 0.969 | 0.993 | 0.922 |
| `zul_Latn` | Zulu | 594 | 0.846 | 0.896 | 0.539 |
| `swh_Latn` | Swahili | 592 | 0.848 | 0.916 | 0.346 |
| `mar_Deva` | Marathi | 589 | 0.965 | 0.990 | 0.926 |
| `tso_Latn` | Tsonga | 581 | 0.965 | 0.956 | 0.425 |
| `hun_Latn` | Hungarian | 576 | 0.987 | 0.999 | 0.232 |
| `afr_Latn` | Afrikaans | 569 | 0.967 | 0.985 | 0.860 |
| `tel_Telu` | Telugu | 555 | 0.991 | 0.994 | 0.951 |
| `rus_Cyrl` | Russian | 528 | 0.932 | 0.949 | 0.462 |
| `nbl_Latn` | South Ndebele | 528 | 0.903 | 0.949 | 0.000 |
| `ory_Orya` | Odia (individual language) | 511 | 0.988 | 0.994 | 0.976 |
| `sot_Latn` | Southern Sotho | 491 | 0.966 | 0.978 | 0.660 |
| `nso_Latn` | Northern Sotho | 479 | 0.956 | 0.969 | 0.914 |
| `kua_Latn` | Kuanyama | 476 | 0.962 | 0.986 | 0.000 |

Where MillID gains most over GlotLID:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `lbj_Tibt` | Ladakhi | 66 | 0.943 | 0.000 | 0.000 |
| `ain_Latn` | Ainu | 1 | 0.667 | 0.000 | 0.000 |
| `ajp_Arab` | South Levantine Arabic | 28 | 0.946 | 0.581 | 0.000 |
| `und_Laoo` | und | 20 | 0.923 | 0.621 | 0.000 |
| `und_Thaa` | und | 29 | 0.881 | 0.651 | 0.000 |
| `sgh_Cyrl` | Shughni | 9 | 0.857 | 0.667 | 0.000 |
| `und_Geor` | und | 32 | 0.877 | 0.694 | 0.000 |
| `tuk_Arab` | Turkmen | 135 | 0.905 | 0.761 | 0.000 |
| `und_Syrc` | und | 43 | 0.966 | 0.822 | 0.000 |
| `gmh_Latn` | Middle High German | 177 | 0.922 | 0.786 | 0.000 |
| `orv_Cyrl` | Old Russian | 239 | 0.931 | 0.800 | 0.000 |
| `und_Kana` | und | 136 | 0.982 | 0.862 | 0.000 |
| `und_Yiii` | und | 38 | 1.000 | 0.884 | 0.000 |
| `neb_Latn` | Toura (Côte d'Ivoire) | 11 | 0.957 | 0.842 | 0.000 |
| `dag_Latn` | Dagbani | 149 | 0.767 | 0.660 | 0.000 |

Where MillID loses most to GlotLID:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `mus_Latn` | Muscogee | 2 | 0.333 | 1.000 | 0.000 |
| `ttj_Latn` | Tooro | 28 | 0.485 | 0.946 | 0.000 |
| `mbd_Latn` | Dibabawon Manobo | 14 | 0.596 | 1.000 | 0.000 |
| `knc_Arab` | Central Kanuri | 68 | 0.462 | 0.825 | 0.007 |
| `biu_Latn` | Biete | 7 | 0.667 | 1.000 | 0.000 |
| `dks_Latn` | Southeastern Dinka | 3 | 0.667 | 1.000 | 0.000 |
| `esi_Latn` | North Alaskan Inupiatun | 9 | 0.455 | 0.783 | 0.000 |
| `hrv_Latn` | Croatian | 168 | 0.622 | 0.944 | 0.178 |
| `plw_Latn` | Brooke's Point Palawano | 8 | 0.727 | 1.000 | 0.000 |
| `bos_Latn` | Bosnian | 198 | 0.649 | 0.902 | 0.181 |
| `xbr_Latn` | Kambera | 8 | 0.750 | 1.000 | 0.000 |
| `nyf_Latn` | Giryama | 10 | 0.645 | 0.870 | 0.000 |
| `ldi_Latn` | Laari | 62 | 0.777 | 0.992 | 0.000 |
| `hak_Hani` | Hakka Chinese | 29 | 0.621 | 0.824 | 0.000 |
| `ccp_Latn` | Chakma | 3 | 0.800 | 1.000 | 0.000 |

#### Per language: udhr

F1 language by language, GlotLID and fastText restricted to MillID's labels: against GlotLID, MillID is better on **52** languages, worse on **105** and within 0.01 on 201; against fastText, MillID is better on **303** languages, worse on **15** and within 0.01 on 40. Every language, with precision, recall and the unrestricted scores, is in `reports/udhr/leaderboard.csv`.

The 20 languages with the most passages:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `roh_Latn` | Romansh | 399 | 0.994 | 0.999 | 0.946 |
| `oci_Latn` | Occitan | 398 | 0.277 | 0.376 | 0.435 |
| `cmn_Hani` | Chinese (Mandarin) | 319 | 0.892 | 0.868 | 0.468 |
| `krl_Latn` | Karelian | 183 | 0.000 | 0.011 | 0.000 |
| `qug_Latn` | Chimborazo Highland Quichua | 182 | 0.011 | 0.000 | 0.000 |
| `acu_Latn` | Achuar-Shiwiar | 180 | 0.828 | 0.854 | 0.000 |
| `hus_Latn` | Huastec | 175 | 0.977 | 0.986 | 0.000 |
| `hau_Latn` | Hausa | 155 | 0.990 | 1.000 | 0.876 |
| `auc_Latn` | Waorani | 154 | 0.000 | 0.000 | 0.000 |
| `kwi_Latn` | Awa-Cuaiquer | 145 | 0.880 | 0.877 | 0.000 |
| `jpn_Jpan` | Japanese | 140 | 1.000 | 1.000 | 0.944 |
| `snn_Latn` | Siona | 136 | 0.630 | 0.514 | 0.000 |
| `ktu_Latn` | Kituba (Democratic Republic of Congo) | 118 | 0.765 | 0.680 | 0.000 |
| `lat_Latn` | Latin | 118 | 0.996 | 0.992 | 0.000 |
| `hat_Latn` | Haitian Creole | 117 | 0.992 | 0.979 | 0.699 |
| `nya_Latn` | Nyanja | 116 | 0.991 | 1.000 | 0.806 |
| `kmb_Latn` | Kimbundu | 116 | 0.970 | 1.000 | 0.877 |
| `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 |
| `twi_Latn` | Twi | 116 | 0.983 | 0.996 | 0.396 |
| `chr_Cher` | Cherokee | 115 | 0.670 | 0.663 | 0.000 |

Where MillID gains most over GlotLID:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `mic_Latn` | Mi'kmaw | 59 | 0.992 | 0.427 | 0.000 |
| `tet_Latn` | Tetum | 58 | 0.638 | 0.167 | 0.000 |
| `fuf_Latn` | Pular | 58 | 0.906 | 0.483 | 0.000 |
| `sag_Latn` | Sango | 59 | 0.992 | 0.578 | 0.189 |
| `skr_Arab` | Saraiki | 63 | 0.867 | 0.500 | 0.000 |
| `qvh_Latn` | Huamalíes-Dos de Mayo Huánuco Quechua | 58 | 0.468 | 0.154 | 0.000 |
| `zgh_Tfng` | Standard Moroccan Tamazight | 60 | 0.750 | 0.462 | 0.000 |
| `lld_Latn` | Ladin | 59 | 0.909 | 0.731 | 0.000 |
| `tah_Latn` | Tahitian | 57 | 0.973 | 0.812 | 0.966 |
| `yue_Hani` | Cantonese | 49 | 0.734 | 0.580 | 0.111 |
| `ote_Latn` | Mezquital Otomi | 59 | 0.154 | 0.000 | 0.000 |
| `pnb_Arab` | Western Panjabi | 61 | 0.882 | 0.739 | 0.000 |
| `mxv_Latn` | Metlatónoc Mixtec | 58 | 0.129 | 0.000 | 0.000 |
| `lij_Latn` | Ligurian | 58 | 0.983 | 0.859 | 0.487 |
| `bum_Latn` | Bulu | 58 | 0.215 | 0.098 | 0.000 |

Where MillID loses most to GlotLID:

| Label | Language | Passages | MillID | GlotLID | fastText |
|---|---|---|---|---|---|
| `gyr_Latn` | Guarayu | 49 | 0.537 | 0.990 | 0.000 |
| `shp_Latn` | Shipibo-Conibo | 62 | 0.469 | 0.912 | 0.000 |
| `dyu_Latn` | Dyula | 58 | 0.000 | 0.434 | 0.015 |
| `tbz_Latn` | Ditammari | 60 | 0.000 | 0.378 | 0.000 |
| `cbu_Latn` | Candoshi-Shapra | 57 | 0.500 | 0.874 | 0.000 |
| `idu_Latn` | Idoma | 51 | 0.603 | 0.938 | 0.000 |
| `ell_Grek` | Greek | 115 | 0.670 | 1.000 | 1.000 |
| `qvc_Latn` | Cajamarca Quechua | 58 | 0.648 | 0.906 | 0.000 |
| `cpu_Latn` | Pichis Ashéninka | 60 | 0.729 | 0.974 | 0.000 |
| `qvn_Latn` | North Junín Quechua | 57 | 0.219 | 0.460 | 0.000 |
| `hin_Deva` | Hindi | 59 | 0.723 | 0.957 | 0.975 |
| `hrv_Latn` | Croatian | 58 | 0.500 | 0.727 | 0.474 |
| `kbp_Latn` | Kabiyè | 59 | 0.781 | 1.000 | 0.389 |
| `cni_Latn` | Asháninka | 52 | 0.773 | 0.963 | 0.000 |
| `rmn_Latn` | Balkan Romani | 116 | 0.678 | 0.857 | 0.000 |

Precision, recall (sensitivity), F1 and specificity for every language and system, restricted and as it is, are in `reports/<set>/`, and the totals in `reports/summary.json`.

## Quick start

Install the `millid` package, with ONNX Runtime for the CPU (`[gpu]` for CUDA):

```bash
pip install "millid[cpu] @ git+https://github.com/baobab-tech/millid.git"
```

Identify a passage, several, or a whole document:

```python
from millid import MillID

model = MillID.from_pretrained("baobabtech/millid-tiny-v1.5")
model.predict("Lave men ou ak savon anvan ou manje.", k=3)
# [Prediction(label='hat_Latn', language='Haitian Creole', probability=...), ...]
model.predict(["one text", "another"], k=3)  # one list of answers per text
model.identify_document(long_text)           # any length: its languages and segments
```

From the command line:

```bash
millid predict "Lave men ou ak savon anvan ou manje." --model baobabtech/millid-tiny-v1.5 -k 3
```

In JavaScript, in Node or a browser, with [`@baobabtechai/millid`](https://www.npmjs.com/package/@baobabtechai/millid) from npm:

```bash
npm install @baobabtechai/millid onnxruntime-node   # onnxruntime-web in a browser app
```

```js
import { MillID } from "@baobabtechai/millid";   // "@baobabtechai/millid/web" in a browser, with { ort }
const model = await MillID.fromPretrained("baobabtech/millid-tiny-v1.5");
await model.predict("Lave men ou ak savon anvan ou manje.", 3);
```

`millid.py` in this repository is the PyTorch training code, which reads `model.safetensors` to fine-tune or export.

## Best practices

### 1. Pass the text as it is

The encoder applies Unicode NFKC and collapses whitespace itself. Accents, diacritics and the script are much of what it reads, so do not strip, transliterate or romanise text before passing it.

### 2. Read long text as a document

`predict` reads the first 255 bytes of a passage. For anything longer, `identify_document` reads consecutive windows (at most 256, spread evenly over a longer text), weighs their probabilities by length, and reports the document's languages and the runs of each.

### 3. Treat a low top probability as unsure

In our tests on real PDFs, a text layer whose characters were mapped to the wrong Unicode code points got a top probability below 0.5 from every MillID model, and reading the same pages from their images gave the right language. A top answer below 0.5 is worth a second look; but see the legacy fonts under Limitations, where that check does not hold.

### 4. Expect less from very short text

On `test`, top-1 accuracy is 0.9333 for passages of 20 to 80 characters against 0.9833 for paragraphs. For a heading or a table cell, look at the top three answers rather than the first alone.

### 5. One text alone and the same text in a batch can score differently

A batch is lower-cased as one string, so a dotted capital `İ` anywhere in it changes how every text in it is lower-cased (one Greek passage scored 0.976 alone and 0.985 beside Turkish text). The Python and JavaScript runtimes do this identically; send one text at a time where scores must not depend on their neighbours.

### 6. Choose the size for where it runs

tiny v1.5 on a CPU or in a web page; medium v1 or large v1.5 on a GPU, large v1.5 when accuracy matters most. The three share their labels, so the choice changes no code.

## Model data

MillID was trained on passages cut from pages the way a page is read: short lines, sentences and paragraphs.

* **PDFs:** [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs), PDF text from Common Crawl, labelled by GlotLID.
* **Web pages:** [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), web text from Common Crawl, labelled by GlotLID.
* **Storybooks:** [GlotStoryBook](https://huggingface.co/datasets/cis-lmu/GlotStoryBook) (African Storybook, Pratham Books, LIDA, Little Cree Books) and Nal'ibali's stories in South African languages, labelled by their publishers.
* **News in sparse languages:** [GlotSparse](https://huggingface.co/datasets/cis-lmu/GlotSparse), labelled by the sites.
* **Publishers' documents:** [CAWST](https://washresources.cawst.org)'s training material and the [Save the Children Resource Centre](https://resourcecentre.savethechildren.net)'s reports and guidance, each document labelled with the language its publisher catalogued.
* **Not trained on:** the Universal Declaration of Human Rights, kept for testing; passages quoting it were removed from training.

The training data is the full MillID build. The public [baobabtech/millid-dataset](https://huggingface.co/datasets/baobabtech/millid-dataset) is that build without Save the Children's and Nal'ibali's text (1.6% of its passages), which is not ours to redistribute. Its splits are the same, so its `test` split is the one scored above, without those passages.

### Data processing

* **Passages:** Unicode NFKC, which folds the ligatures and presentation forms PDF text layers are full of; URLs, emails and dot leaders removed; a passage must be mostly letters. Exact duplicates were removed across every source, and no PDF gives more than 40 passages, article 20 or web page 10.
* **Labels checked:** a publisher's label is kept unless GlotLID is confident (≥ 0.8) the passage is in another major language, typically a copyright line; a passage from FinePDFs or FineWeb-2 is kept only when GlotLID's reading of the passage agrees with its document's.
* **Text that is no language removed:** passages GlotLID is confident are `zxx` (no linguistic content), mostly PDFs whose fonts map letters to the wrong code points, and documents catalogued in the wrong language.
* **Splits by document:** no test passage shares a document with a training one.

### Training

1,943,957 passages; the best validation macro F1 at step 106,000, after 0.6 hours on 1 GPU. Languages sampled in proportion to n^0.3; 25% of passages cut to a 20 to 80 character snippet; batch 512, AdamW at 0.001, cosine schedule, label smoothing 0.1.

## Architecture

![MillID's architecture: a passage is normalised, encoded per UTF-8 byte into byte, script, byte-in-character and hashed character n-gram ids, embedded and summed, passed through a convolutional stem, downsampling and pre-norm transformer blocks, mean-pooled, and scored against label, language and script embeddings](assets/millid-architecture.png)

*Drawn at the medium preset's sizes (12 blocks, d = 768); this model's are in the table above.*

UTF-8 bytes (after Unicode NFKC), each embedded as byte + Unicode script + byte-in-character + hashed character 2-, 3- and 4-grams (4,096 buckets) -> depthwise convolution stem (kernel 5) -> strided convolution, 1 byte per token -> 3 pre-norm blocks (RMSNorm, attention with RoPE, 3 heads, SwiGLU; d=192) -> masked mean pooling -> language head, each language's vector the sum of its own, its language's and its script's. See `ARCHITECTURE.md`.

## Usage and limitations

### Intended usage

* **Routing documents:** choosing the OCR, translation or language model for a document by its language.
* **Building and filtering corpora:** sorting web or PDF text by language, low-resource languages included.
* **Describing collections:** which languages a set of documents holds, and how much of each.
* **On-device detection:** in a browser or on a phone with tiny v1.5, where text should not leave the device.

### Limitations

* **Close languages:** languages that differ by a few words are its weakest (F1 on `test`: Croatian 0.62, Bosnian 0.65, Malay 0.77, Indonesian 0.81, Norwegian Bokmål 0.81). Their short passages are often genuinely ambiguous.
* **Text that is no language:** there is no "no language" class. Nepali typed in the legacy Preeti font, whose text layer is Latin letters standing for Devanagari, was labelled as another language by MillID models, some with a top probability up to 0.74.
* **One label per passage:** a passage that mixes languages gets one; `identify_document` gives the segments of a longer text.
* **Labels learned from GlotLID:** the FinePDFs and FineWeb-2 passages were labelled by GlotLID, so MillID learned its labels, mistakes included; `udhr`, labelled by translators, is the independent test.
* **Genre:** for many low-resource languages the training text is mostly religious web text or children's stories, and a model can learn the genre along with the language.

### Ethical considerations and risks

* **Errors fall on the smallest languages:** filtering or routing text by language can drop or mishandle the text of speakers of low-resource and closely related languages, where MillID is least accurate. Check the per-language scores of the languages that matter before filtering with it.
* **A language is not an identity:** do not use a language label to infer a person's nationality, ethnicity or origin.
* **Labels can be contested:** whether two varieties are one language or two is political as well as linguistic. MillID's labels follow ISO 639-3 and its training data, not a judgement on that question.

### Benefits

MillID tiny v1.5 is within 0.020 macro F1 of GlotLID, a 1.7 GB fastText model, on the 1,621 languages of `test`, and within 0.011 on `udhr`. At 11 MB, it loads in a web page or ships inside an app. All three sizes are open under Apache-2.0, with the code that trained and evaluated them and the dataset, less the text that is not ours to share.

## Licence

The weights and the code are Apache-2.0. The training data carries its own licence row by row (see the [dataset card](https://huggingface.co/datasets/baobabtech/millid-dataset#licensing)), including text licensed for non-commercial use only (CC BY-NC, CC BY-NC-SA).