README.md

---
license: mit
library_name: redup-topicmodel
tags:
- topic-modeling
- bigartm
- multilingual
- feature-extraction
---

# REDUP multilingual topic model

Multilingual topic model that maps a document to a **125-dimensional topic
distribution**. The repository contains inference weights and per-language BPE
tokenizers for **100 languages**.

The dump is built for **BigARTM 0.9.2**. It is **not** a Transformers /
Diffusers checkpoint: `AutoModel` and `AutoTokenizer` are not supported.

| | |
|---|---|
| Framework | BigARTM 0.9.2 |
| Topics | 125 (`topic_0` … `topic_124`) |
| Languages | 100 |
| Output | topic distribution (sums ≈ 1) |
| License | MIT |

## Related repositories

| Role | Repository |
|------|------------|
| **Training / research pipeline** | [machine-intelligence-laboratory/text_categorization](https://github.com/machine-intelligence-laboratory/text_categorization) |
| **Inference service (recommended)** | [redup-ai/redup.python.topicmodel](https://github.com/redup-ai/redup.python.topicmodel) |

Use **text_categorization** as the historical training reference. For serving
embeddings and explanations in production, prefer **redup.python.topicmodel**.

## Repository contents

| Path | Description |
|------|-------------|
| `artm/parameters.bin` | BigARTM model metadata and regularizers |
| `artm/p_wt.bin` | Topic–word matrix Φ used at inference |
| `tokenizers.json.gz` | BPE packs keyed by language: `{lang: {vocab, merges}}` |
| `config.json` | Machine-readable metadata |
| `LICENSE` | MIT |

Total size is about **251 MB**.

## Languages

Each document must provide modality `lang` with an ISO-like code from the table
below. Tokens are scored under BigARTM class id `@{lang}`.

| | | | | |
|---|---|---|---|---|
| `af` | `am` | `ar` | `av` | `az` |
| `ba` | `be` | `bg` | `bn` | `bs` |
| `ca` | `ce` | `cs` | `cv` | `cy` |
| `da` | `de` | `el` | `en` | `eo` |
| `es` | `et` | `eu` | `fa` | `fi` |
| `fr` | `gd` | `gl` | `gu` | `he` |
| `hi` | `hr` | `hu` | `hy` | `ia` |
| `id` | `inh` | `is` | `it` | `ja` |
| `jv` | `ka` | `kaa` | `kbd` | `kk` |
| `kl` | `km` | `ko` | `krc` | `ky` |
| `la` | `lez` | `lo` | `lt` | `lv` |
| `mg` | `mhr` | `mi` | `mk` | `ml` |
| `mn` | `mo` | `ms` | `my` | `myv` |
| `ne` | `nl` | `no` | `oc` | `os` |
| `pl` | `pt` | `rm` | `rn` | `ro` |
| `ru` | `sah` | `sh` | `si` | `sk` |
| `sl` | `sm` | `so` | `sq` | `sr` |
| `sv` | `sw` | `ta` | `tg` | `th` |
| `tk` | `tr` | `tt` | `udm` | `uk` |
| `ur` | `uz` | `vi` | `yi` | `zh` |

## How to download

```bash
pip install huggingface_hub
```

```python
from huggingface_hub import snapshot_download

root = snapshot_download(repo_id="redup-ai/topicmodel-multilingual")
```

## Usage

Recommended path: the inference helpers from
[redup.python.topicmodel](https://github.com/redup-ai/redup.python.topicmodel).

```bash
pip install huggingface_hub bigartm==0.9.2
# install redup-topicmodel from the service repository / package index you use
```

```python
import asyncio
from types import SimpleNamespace

from redup_topicmodel.topicmodel.worker import TopicModel


def document(document_id: str, tokens: list[str], lang: str):
    return SimpleNamespace(
        document_id=document_id,
        tokens=tokens,
        modalities={"lang": lang},
    )


async def main(root: str):
    model = TopicModel({
        "artifact_root": root,
    })
    pack = SimpleNamespace(documents=[
        document("doc-en", ["hello", "world"], "en"),
        document("doc-ru", ["привет", "мир"], "ru"),
    ])
    result = await model.get_documents_embedding("example", pack)
    for embedding in result["embeddings"]:
        # length-125 topic distribution
        print(len(embedding["values"]), sum(embedding["values"]))


asyncio.run(main(root))
```

### gRPC service

Point the service config at the downloaded directory:

```yaml
TopicModel:
  artifact_root: /path/from/snapshot_download
```

### Tokenizer only

```python
from redup_topicmodel.topicmodel.bpe import Tokenizers

tokenizers = Tokenizers.load(f"{root}/tokenizers.json.gz")
print(tokenizers["en"].encode("hello world"))
```

## Limitations

- Requires BigARTM and the redup inference stack; not compatible with
  `transformers.AutoModel` / `AutoTokenizer`.
- `parameters.bin` is a BigARTM pickle — load only trusted artifacts.
- `tokenizers.json.gz` stores independent BPE packs per language; it is not a
  Hugging Face `tokenizer.json` export.
- Short or out-of-vocabulary inputs may produce degenerate topic mass (for
  example concentrated on `topic_0`).

## License

MIT. See `LICENSE`.