mlboydaisuke commited on
Commit
1745bd1
·
verified ·
1 Parent(s): 59b97ca

Stop chat turns on <end_of_turn>, not <eos>

Browse files

tokenizer_config declared eos_token = "<eos>" (id 1, document end), but the chat template ends a turn with <end_of_turn> (id 106) and upstream generation_config lists eos_token_id = [1, 106]. A runtime that derives its stop token from eos_token alone (swift-transformers does) never saw the turn terminator and generated to the token cap.

Render-safe: the template emits the terminator literally, not via {{ eos_token }}, so rendering is unchanged. Only this one field differs; the rest of the file is byte-identical.

Files changed (1) hide show
  1. hf_model/tokenizer_config.json +1 -1
hf_model/tokenizer_config.json CHANGED
@@ -51335,7 +51335,7 @@
51335
  "bos_token": "<bos>",
51336
  "clean_up_tokenization_spaces": false,
51337
  "eoi_token": "<end_of_image>",
51338
- "eos_token": "<eos>",
51339
  "extra_special_tokens": {
51340
  "boi_token": "<start_of_image>",
51341
  "eoi_token": "<end_of_image>",
 
51335
  "bos_token": "<bos>",
51336
  "clean_up_tokenization_spaces": false,
51337
  "eoi_token": "<end_of_image>",
51338
+ "eos_token": "<end_of_turn>",
51339
  "extra_special_tokens": {
51340
  "boi_token": "<start_of_image>",
51341
  "eoi_token": "<end_of_image>",