license: apache-2.0
base_model: allenai/Olmo-3-7B-Instruct
tags:
- tokenizer
Olmo-3-Instruct tokenizer (eos_token fixed to <|im_end|>)
Tokenizer-only fork of the Olmo 3 Instruct family (e.g.
allenai/Olmo-3-7B-Instruct) —
should work for any size sharing the same Instruct tokenizer/chat template.
No model weights here — use this repo for the tokenizer/chat template only,
alongside the original repo's weights.
What changed
eos_token is set to <|im_end|> (token id 100265) instead of the
upstream default <|endoftext|> (token id 100257). Everything else
(vocab, chat_template.jinja, special tokens) is untouched.
Why
Olmo 3's chat template only emits the literal eos_token after the final
assistant turn when that turn is the last message in the list:
{%- if not loop.last -%}{{- '<|im_end|>' + '\n' -}}
{%- else -%}{{- eos_token -}}{%- endif -%}
Every other turn boundary closes with <|im_end|> unconditionally.
Tooling that renders a partial trajectory (e.g. to find the templated
prefix before the last assistant turn, without that turn present) never
sees the real eos_token and errors — even though <|im_end|> (id
100265) is already registered as a valid stop token in the model's own
generation_config.json (eos_token_id: [100265, 100257]), and is a real,
official special token (special: true) in the model's own vocabulary.
This fork just makes tokenizer.eos_token point at the one that actually
appears at every turn boundary, sidestepping the conditional.