solidity-pygments-tokenizer / tokenizer_config.json
Radda's picture
Upload folder using huggingface_hub
2850ea1 verified
raw
history blame contribute delete
229 Bytes
{"special_tokens": {"pad_token": "[PAD]", "unk_token": "[UNK]", "cls_token": "[CLS]", "sep_token": "[SEP]", "mask_token": "[MASK]"}, "tokenizer_class": "SolidityPygmentsTokenizer", "vocab_size": 16777216, "model_max_length": 512}