README.md

---
language:
  - en
  - fr
  - es
  - ro
license: apache-2.0
library_name: transformers
tags:
  - tokenizer
  - bpe
  - byte-level
  - multilingual
  - cijov
  - cijov-lang
pipeline_tag: text-generation
---

# Cijov-lang Tokenizer

A byte-level BPE tokenizer trained from scratch on a multilingual corpus
covering **English, French, Spanish, and Romanian**, with additional
coverage of Python code and mathematics.

## Overview

| Property | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Vocab size | 151,936 |
| Languages | EN, FR, ES, RO |
| Additional domains | Python code, mathematics |
| Special tokens | 25 (ChatML + tool-call + FIM) |
| Training data | ~840k documents (~1.3 GB raw text) |
| License | Apache 2.0 |

## Training Data Sources

The tokenizer was trained on a balanced multilingual corpus collected from
publicly available datasets:

| Source | Languages | Proportion |
|---|---|---|
| [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | EN, FR, ES, RO | ~68% (web text) |
| [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | EN, FR, ES, RO | ~15% (encyclopedic) |
| [CodeXGlue (Python)](https://huggingface.co/datasets/google/code_x_glue_ct_code_to_text) | Python | ~2.4% (code) |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | EN | ~2.4% (mathematics) |

Each language received equal document counts to ensure balanced merge
learning across all four target languages.

## Architecture

- **Pre-tokenizer**: Byte-level with GPT-2 regex splitting (whitespace-aware)
- **Model**: BPE with merges learned entirely from the training corpus above
- **Decoder**: Byte-level (lossless roundtrip for any UTF-8 input)
- **Post-processor**: Byte-level with untrimmed offsets

## Special Tokens

| Token | ID | Purpose |
|---|---|---|
| `<\|endoftext\|>` | 151643 | End of document / padding |
| `<\|im_start\|>` | 151644 | Chat turn start (ChatML) |
| `<\|im_end\|>` | 151645 | Chat turn end (ChatML) |
| `<tool_call>` | 151657 | Tool/function call start |
| `</tool_call>` | 151658 | Tool/function call end |
| `<\|fim_prefix\|>` | 151659 | Fill-in-the-middle prefix |
| `<\|fim_middle\|>` | 151660 | Fill-in-the-middle middle |
| `<\|fim_suffix\|>` | 151661 | Fill-in-the-middle suffix |
| `<tool_response>` | 151665 | Tool response start |
| `</tool_response>` | 151666 | Tool response end |
| `<\|cijov\|>` | 151667 | Model identity sentinel |

Full list of 25 special tokens available in `special_tokens_map.json`.

## Chat Template

Built-in ChatML template:

```
<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{assistant_message}<|im_end|>
```

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("cijov/cijov-lang-tokenizer")

# Encode text
text = "Once upon a time in a faraway land"
ids = tokenizer.encode(text)
print(f"Tokens: {len(ids)}")

# Chat template
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me a story."},
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(formatted)
```

## Performance

Compression efficiency (characters per token) on held-out samples:

| Language | Cijov-lang | Qwen3 baseline | Improvement |
|---|---|---|---|
| English | 4.17 | 4.17 | same |
| French | 4.27 | 3.53 | +21% |
| Spanish | 4.31 | 3.21 | +34% |
| Romanian | 3.97 | 2.26 | +76% |

Higher is better (more characters encoded per token = more efficient).

## Files

```
├── tokenizer.json            # Full BPE vocab + merges
├── tokenizer_config.json     # HF tokenizer configuration
├── special_tokens_map.json   # Special token definitions
└── chat_template.jinja       # Standalone chat template
```

## Training Procedure

1. **Corpus collection**: Streamed ~200k documents per language from
   public HuggingFace datasets (web, Wikipedia, code, math).
2. **BPE training**: Byte-level BPE with minimum frequency threshold of 2,
   learning merges until reaching 151,936 vocabulary entries.
3. **Special token anchoring**: Reserved token padding ensures special
   tokens land at fixed IDs (151643–151667) regardless of learned vocab.
4. **Validation**: Verified roundtrip integrity, compression ratios, and
   special token ID correctness.

## Intended Use

This tokenizer is designed for:
- Multilingual text generation (EN/FR/ES/RO)
- Code completion (Python)
- Mathematical reasoning
- Chat / instruction-following (ChatML format)
- Fill-in-the-middle code completion (FIM tokens)

## Limitations

- Optimised for Latin-script languages. CJK / Arabic / Cyrillic coverage
  exists (byte-level guarantees no UNK) but compression will be poor.
- Trained on publicly available web data — inherits any biases present
  in the source corpora.

## Citation

```bibtex
@misc{cijov-lang-tokenizer-2026,
  title  = {Cijov-lang Tokenizer: A Multilingual Byte-Level BPE Tokenizer},
  author = {Cijov},
  year   = {2026},
  url    = {https://huggingface.co/cijov/cijov-lang-tokenizer}
}
```

## License

Apache 2.0