Duplicated from mertcobanov/turkish-wordpiece-tokenizer

Taklaxbr
/

Turkish-WordPiece-Tokenizer

hf_repo:Taklaxbr/Turkish-WordPiece-Tokenizer

Model card Files Files and versions

Turkish-WordPiece-Tokenizer / tokenizer_config.json

Taklaxbr's picture

Duplicate from mertcobanov/turkish-wordpiece-tokenizer

3809759 about 1 month ago

History Blame Contribute Delete

115 Bytes

{"model_type": "wordpiece", "unk_token": "[UNK]", "bos_token": "[CLS]", "eos_token": "[SEP]", "pad_token": "[PAD]"}