parameter-golf-tokenizers / tokenizer_config.export.json
sproos's picture
Upload folder using huggingface_hub
c5f9e16 verified
{
"tokenizers": [
{
"name": "sp_bpe_8192",
"dataset_suffix": "sp8192",
"vocab_size": 8192,
"tokenizer_train_docs": 50000
}
]
}