SWT — Unified SentencePiece BPE Tokenizer
Beta / early access. This tokenizer is not in its final version. Request access via the repo's Access tab (approved by the Wonhui team); the vocabulary, normalizer and special-token layout may still change before the release.
Fast (tokenizers) port of the SWT unified SentencePiece model — a byte-fallback
BPE tokenizer trained over the Wonhui SWT multilingual mixture (464 languages,
37 scripts in the evaluation holdout).
This repository ships a self-contained fast tokenizer (tokenizer.json). It does not require the
sentencepiece library at inference time. The original unified.model is also
included for users who want raw SentencePiece parity.
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
ids = tok.encode("Morphologie et syntaxe")
Note: a user can signal the language by declaring a language ISO-693-3 tag
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
tok.encode("<|lang:fra|> Morphologie et syntaxe")
Run it on your own machine
From the Hub (after access is approved)
pip install -r requirements.txt # transformers, tokenizers
python - <<'PY'
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import sentencepiece as spm
# 1) Fast path (tokenizer.json only, no sentencepiece needed)
tok = AutoTokenizer.from_pretrained("wonhui-labs/swt-tokenizer", token=True)
ids = tok.encode("<|lang:fra|> Morphologie et syntaxe")
# 2) Raw SentencePiece path (exact spm parity)
path = hf_hub_download("wonhui-labs/swt-tokenizer", "unified.model", token=True)
sp = spm.SentencePieceProcessor(model_file=path)
assert tok.encode("Bonjour le monde") == sp.encode("Bonjour le monde")
PY
Fully local (downloaded repo folder)
Download the repo (git clone or the web UI), then load straight from the folder —
only tokenizer.json and tokenizer_config.json are required:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./swt-tokenizer")
ids = tok.encode("Morphologie et syntaxe")