---
license: isc
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-2
- nick007x/github-code-2025
language:
- fr
- en
- zh
pipeline_tag: token-classification
tags:
- code
---
# Multistral Tokenizer

Training completed successfully!

## Configuration
- Vocabulary size: 127,989
- Special tokens: 13
- Min frequency: 2
- Training samples: up to 500,000

## Datasets
- nick007x/github-code-2025 (35%)
- HuggingFaceFW/fineweb-2 - Lojban (10%)
- HuggingFaceFW/fineweb-2 - French (15%)
- HuggingFaceFW/fineweb-2 - Chinese (15%)
- HuggingFaceFW/fineweb - English (25%)

## Special Tokens
```<|begin|>, <|return|>, <|pad|>, <|start|>, <|channel|>, <|end|>, <|message|>, <|image|>, <|video|>, <|audio|>, <|call|>, <|constrain|>, <|unknown|>```

## Enforced Vocabulary
```analysis, assistant, commentary, developer, final, json, system, tool, toon, user, yaml```

## Usage

```python
from multistral.multistraltokenizer import MultistralTokenizer

tokenizer = MultistralTokenizer.from_pretrained("models/aizia_tokenizer")
tokens = tokenizer.encode("Your text here")
text = tokenizer.decode(tokens)
```