emma-5 / tokenizer_config.json
Davide Palma (ElDavo)
reupload
ccc5b7e unverified
Raw
History Blame Contribute Delete
128 Bytes
{
"tokenizer_type": "sentencepiece",
"model_file": "bpe.model",
"pad_id": 0,
"unk_id": 1,
"bos_id": 2,
"eos_id": 3
}