OpenEuroLLM Tokenizer (256k) with reason tokens
A 262,144-token SentencePiece BPE tokenizer designed for efficient tokenization across all EU official languages and additional European languages. Trained on 173 GB of curated multilingual text from the OpenEuroLLM data catalogue on LUMI HPC. This is a patched version of OpenEuroLLM Tokenizer (256k) where reasoning tokens have been added using Tokenizer patch
See OpenEuroLLM Tokenizer (256k) for result and usage.
Changed Tokens
| Token | Original token | ID | Purpose |
|---|---|---|---|
<|im_start|> |
<start_of_turn> |
3 | Chat turn start |
<|im_end|> |
<end_of_turn> |
4 | Chat turn end |
<tools> |
<unused_0> |
13 | Tools start |
</tools> |
<unused_1> |
14 | Tools end |
<tool_response> |
<unused_2> |
15 | Tool response start |
</tool_response> |
<unused_3> |
16 | Tool response end |
</think> |
<unused_4> |
15 | Think end |
<think> |
<unused_5> |
16 | Think start |
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support