kevinlu1248 commited on
Commit
5eef76f
·
verified ·
1 Parent(s): 5860bec

Update EOS token to match 7B equivalent.

Browse files

Maintain consistency with https://huggingface.co/unsloth/Qwen2.5-Coder-7B/blob/main/special_tokens_map.json. I found out that this was set incorrectly when I tried to distill Unsloth's 7B into 3B and it never stopped generating. Crazy bug.

Files changed (1) hide show
  1. special_tokens_map.json +1 -1
special_tokens_map.json CHANGED
@@ -15,7 +15,7 @@
15
  "<|video_pad|>"
16
  ],
17
  "eos_token": {
18
- "content": "<|im_end|>",
19
  "lstrip": false,
20
  "normalized": false,
21
  "rstrip": false,
 
15
  "<|video_pad|>"
16
  ],
17
  "eos_token": {
18
+ "content": "<|endoftext|>",
19
  "lstrip": false,
20
  "normalized": false,
21
  "rstrip": false,