open-sci-ref-v0.02-1.7b-dclm-1T-4096 / tokenizer_config.json
Harsh1729's picture
Upload open-sci-ref-0.02 checkpoint
598ef96 verified
Raw
History Blame Contribute Delete
169 Bytes
{
"unk_token": "<|endoftext|>",
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"add_prefix_space": false,
"tokenizer_class": "GPTNeoXTokenizer"
}