tokenizer-bos-eos-conflict / tokenizer_config.json
izm1chael's picture
Publish Layerfault deep challenge LF-CH-TOKX-0002
9a03978 verified
Raw
History Blame Contribute Delete
214 Bytes
{
"bos_token": "<eos>",
"chat_template": "{% for message in messages %}{{ message[\"role\"] }}: {{ message[\"content\"] }}{% endfor %}",
"eos_token": "<eos>",
"tokenizer_class": "PreTrainedTokenizerFast"
}