processor-custom-class-reference / tokenizer_config.json
izm1chael's picture
Publish Layerfault deep challenge LF-CH-TOKX-0012
031e905 verified
Raw
History Blame
214 Bytes
{
"bos_token": "<bos>",
"chat_template": "{% for message in messages %}{{ message[\"role\"] }}: {{ message[\"content\"] }}{% endfor %}",
"eos_token": "<eos>",
"tokenizer_class": "PreTrainedTokenizerFast"
}