Upload tokenizer

chat_template.jinja

@@ -9,7 +9,7 @@ A. testosterone.
 B. cortisol.
 C. progesterone.
 D. aldosterone.
-Answer: A
+Answer:A
 
 The following are multiple choice questions (with answers) about knowledge and skills in advanced master-level STEM courses.
 Asp235Phe in a molecular report indicates that:
@@ -17,7 +17,7 @@ A. asparagine has been replaced by phenylalanine.
 B. phenylalanine has been replaced by asparagine.
 C. aspartic acid has been replaced by phenylalanine.
 D. phenylalanine has been replaced by aspartic acid.
-Answer: C
+Answer:C
 
 The following are multiple choice questions (with answers) about knowledge and skills in advanced master-level STEM courses.
 The concept of V/f control of inverters driving induction motors results in:
@@ -25,6 +25,7 @@ A. constant torque operation
 B. speed reversal  
 C. reduced magnetic loss  
 D. harmonic elimination  
-Answer: A
+Answer:A
 
-Answer the following question in the same way:<|im_end|>{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|im_start|>system\n' + message['content'] + '<|im_end|>\n' }}{% elif message['role'] == 'user' %}{{ '<|im_start|>user\n' + message['content'] + '<|im_end|>\n' }}{% elif message['role'] == 'assistant' %}{{ '<|im_start|>assistant\n' + message['content'] + '<|im_end|>\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}<think>Okay, just answer with A, B, C, or D.</think>
+Answer the following question in the same way:<|im_end|>
+{% for message in messages %}{% if message['role'] == 'user' %}{{ '<|im_start|>user\n' + message['content'] + '<|im_end|>\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n<think>\nOkay, just answer with A, B, C, or D.\n</think>' }}{% endif %}

special_tokens_map.json

@@ -21,11 +21,5 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": {
-    "content": "<|endoftext|>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
+  "pad_token": "<|im_end|>"
 }

tokenizer_config.json

@@ -232,7 +232,7 @@
   "errors": "replace",
   "extra_special_tokens": {},
   "model_max_length": 131072,
-  "pad_token": "<|endoftext|>",
+  "pad_token": "<|im_end|>",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
   "unk_token": null