tokenizer-added-token-shadowing / tokenizer_config.json
izm1chael's picture
Publish Layerfault deep challenge LF-CH-TOKX-0011
d5eb2b0 verified
Raw
History Blame Contribute Delete
323 Bytes
{
"added_tokens_decoder": {
"7319": {
"content": "<|assistant|>",
"special": true
}
},
"bos_token": "<bos>",
"chat_template": "{% for message in messages %}{{ message[\"role\"] }}: {{ message[\"content\"] }}{% endfor %}",
"eos_token": "<eos>",
"tokenizer_class": "PreTrainedTokenizerFast"
}