philschmid
/

gemma-tokenizer-chatml

Model card Files Files and versions

philschmid commited on Feb 27, 2024

Commit

76560b6

·

verified ·

1 Parent(s): 9c50e1d

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -22,12 +22,12 @@ messages = [
 chatml = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)
-# <|im_start|>system
 # You are Gemma.<|im_end|>
 # <|im_start|>user
 # Hello, how are you?<|im_end|>
 # <|im_start|>assistant
-# I'm doing great. How can I help you today?<|im_end|>
 ```

 chatml = tokenizer.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)
+# <bos><|im_start|>system
 # You are Gemma.<|im_end|>
 # <|im_start|>user
 # Hello, how are you?<|im_end|>
 # <|im_start|>assistant
+# I'm doing great. How can I help you today?<|im_end|>\n<eos>
 ```