Wiki-Test2 / binaryllm_vocab.json
PhysiQuanty's picture
Duplicate from PhysiQuanty/Patent-Test-Radix-65536-AutoTokenizer_FineTune
efda231
raw
history blame contribute delete
249 Bytes
{
"base_vocab_size": 65536,
"vocab_size": 65538,
"bos_token": "<BOS>",
"bos_token_id": 65536,
"eos_token": "<EOS>",
"eos_token_id": 65537,
"unk_token": "<EOS>",
"unk_token_id": 65537,
"pad_token": "<EOS>",
"pad_token_id": 65537
}