Transformers
GDV-tokenizer-4nodev3 / special_tokens_map.json
Yuyeong's picture
Upload tokenizer
19d3657 verified
Raw
History Blame Contribute Delete
235 Bytes
{
"additional_special_tokens": [
"<|begin_of_text|>",
"<|end_of_text|>",
"<|finetune_right_pad_id|>"
],
"bos_token": "<|begin_of_text|>",
"eos_token": "<|end_of_text|>",
"pad_token": "<|finetune_right_pad_id|>"
}