PebbleLM-117M / tokenizer_config.json
nameissakthi's picture
Initial commit: PebbleLM-117M base model
7d4d882
raw
history blame contribute delete
195 Bytes
{
"vocab_size": 16384,
"pad_token": "<|pad|>",
"bos_token": "<|bos|>",
"eos_token": "<|eos|>",
"unk_token": "<|unk|>",
"user_token": "<|user|>",
"assistant_token": "<|assistant|>"
}