Stop chat turns on <end_of_turn>, not <eos>
Browse filestokenizer_config declared eos_token = "<eos>" (id 1, document end), but the chat template ends a turn with <end_of_turn> (id 106) and upstream generation_config lists eos_token_id = [1, 106]. A runtime that derives its stop token from eos_token alone (swift-transformers does) never saw the turn terminator and generated to the token cap.
Render-safe: the template emits the terminator literally, not via {{ eos_token }}, so rendering is unchanged. Only this one field differs; the rest of the file is byte-identical.
hf_model/tokenizer_config.json
CHANGED
|
@@ -51335,7 +51335,7 @@
|
|
| 51335 |
"bos_token": "<bos>",
|
| 51336 |
"clean_up_tokenization_spaces": false,
|
| 51337 |
"eoi_token": "<end_of_image>",
|
| 51338 |
-
"eos_token": "<
|
| 51339 |
"extra_special_tokens": {
|
| 51340 |
"boi_token": "<start_of_image>",
|
| 51341 |
"eoi_token": "<end_of_image>",
|
|
|
|
| 51335 |
"bos_token": "<bos>",
|
| 51336 |
"clean_up_tokenization_spaces": false,
|
| 51337 |
"eoi_token": "<end_of_image>",
|
| 51338 |
+
"eos_token": "<end_of_turn>",
|
| 51339 |
"extra_special_tokens": {
|
| 51340 |
"boi_token": "<start_of_image>",
|
| 51341 |
"eoi_token": "<end_of_image>",
|