Mini-Turkish-Tokenizer / tokenizer_config.json
kaanilker's picture
Upload 3 files
6e169aa verified
{
"vocab_size": 5610,
"pad_token_id": 0,
"bos_token_id": 2,
"eos_token_id": 3,
"unk_token_id": 1,
"tokenizer_class": "PreTrainedTokenizerFast"
}