unicrs-rec / tokenizer_config.json
ZhankuiHe's picture
Upload tokenizer
dc53bee verified
raw
history blame contribute delete
169 Bytes
{
"tokenizers": {
"item": "EntityTokenizer",
"context": "GPT2TokenizerFast",
"prompt": "RobertaTokenizerFast"
},
"tokenizer_key_for_decoding": "item"
}