README.md

---
language:
- ko
- en
license: apache-2.0
library_name: tokenizers
datasets:
- HuggingFaceFW/fineweb-2
- wikimedia/wikipedia
- eduagarcia/multilingual_tokenizer_benchmark
- HAERAE-HUB/KMMLU
tags:
- tokenizer
- korean
- byte-level-bpe
- lossless
---

# KorByte-128K

KorByte-128K is a Korean-focused, Unicode-aware byte-level BPE tokenizer with
128,000 learned tokens and 256 stable special-token IDs. It performs no Unicode
normalization, so it preserves spaces, line endings, decomposed Hangul, emoji, and
arbitrary UTF-8 text exactly.

It ranks **first among 8 successfully loaded, revision-pinned public systems**
by both fertility and effective bits per byte (EBPB) on the Korean slice of the
[Multilingual Tokenizer Benchmark](https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark).
It also ranks first on a frozen, post-selection
[KMMLU](https://huggingface.co/datasets/HAERAE-HUB/KMMLU) test audit. This is a scoped
intrinsic result, not proof of universal or downstream language-model superiority.

## Quick start

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dawncr0w/KorByte-128K", use_fast=True)
text = "새 기능을 배포하기 전에 테스트 결과를 확인해 주세요."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
```

## Measured result

- Public Korean benchmark fertility / EBPB: **1.4463 / 3.0710** (rank 1)
- KMMLU audit fertility / EBPB: **1.9364 / 3.2942** (rank 1)
- Macro token reduction vs. `kakaocorp/kanana-2-3b-base`: **18.92%**
- Public first-place gate: **PASS**
- Exact round-trip release validation: **passed**
- Core / total vocabulary: **128,000 / 128,256**

See [`reports/comparison.md`](reports/comparison.md) for the pinned public ranking and
unavailable artifacts, and [`reports/research.md`](reports/research.md) for the
accepted and rejected variants. KLUE domain counts and throughput are in
[`reports/benchmark.md`](reports/benchmark.md). Machine-readable evidence is under
[`reports/`](reports/).

## Why OKT and MeCab-ko are not the primary baseline

OKT and MeCab-ko are morphological analyzers. They do not provide the same fixed-vocabulary,
lossless, byte-complete encoding contract required by an LLM tokenizer. Their output counts
and speed are reported as useful context; Kanana-2 is the like-for-like tokenizer baseline.

## Design

- Unicode-aware word-boundary segmentation with six-digit number chunks
- Byte-level alphabet, decoder, and no normalizer for complete coverage
- 700 million-character Korean-heavy public training mixture with a smaller English allocation
- Deterministic source revisions, shuffle seed, filtering, deduplication, and manifests
- 256 contiguous special-token IDs from 128,000 through 128,255

## Intended use and limitations

This artifact is intended for Korean-heavy language-model experiments, token-count analysis,
and as a starting vocabulary for training a new model. Replacing the tokenizer of an existing
model without retraining or vocabulary adaptation will break that model. Compression alone does
not guarantee better accuracy, latency, safety, or training efficiency. The Thunder public
artifact could not be loaded because it uses a custom tokenizer model; the comparison report
records the exact failure instead of silently omitting it.

## Reproduce

```bash
uv sync --all-extras
uv run korbyte prepare --scale 1.0
uv run korbyte train
uv run korbyte benchmark
uv run korbyte compare
uv run korbyte render
uv run korbyte validate
```

The prepared training text is intentionally excluded from this repository. Exact source
revisions, accepted character counts, filtering, and hashes are documented in
[`DATA_SOURCES.md`](DATA_SOURCES.md) and [`provenance/`](provenance/).