README.md

---
license: apache-2.0
base_model: allenai/Olmo-3-7B-Instruct
tags:
  - tokenizer
---

# Olmo-3-Instruct tokenizer (`eos_token` fixed to `<|im_end|>`)

Tokenizer-only fork of the Olmo 3 Instruct family (e.g.
[`allenai/Olmo-3-7B-Instruct`](https://huggingface.co/allenai/Olmo-3-7B-Instruct)) —
should work for any size sharing the same Instruct tokenizer/chat template.
No model weights here — use this repo for the tokenizer/chat template only,
alongside the original repo's weights.

## What changed

`eos_token` is set to `<|im_end|>` (token id `100265`) instead of the
upstream default `<|endoftext|>` (token id `100257`). Everything else
(vocab, `chat_template.jinja`, special tokens) is untouched.

## Why

Olmo 3's chat template only emits the literal `eos_token` after the final
assistant turn *when that turn is the last message in the list*:

```jinja
{%- if not loop.last -%}{{- '<|im_end|>' + '\n' -}}
{%- else -%}{{- eos_token -}}{%- endif -%}
```

Every other turn boundary closes with `<|im_end|>` unconditionally.
Tooling that renders a partial trajectory (e.g. to find the templated
prefix before the last assistant turn, without that turn present) never
sees the real `eos_token` and errors — even though `<|im_end|>` (id
`100265`) is already registered as a valid stop token in the model's own
`generation_config.json` (`eos_token_id: [100265, 100257]`), and is a real,
official special token (`special: true`) in the model's own vocabulary.
This fork just makes `tokenizer.eos_token` point at the one that actually
appears at every turn boundary, sidestepping the conditional.