PAD tokens is present in `tokenizer.json` but not in `special_tokens_map.json`.
#4
by
sadra-barikbin
- opened
Hi there!
PAD token is present in tokenizer's added tokens in tokenizer.json as a special token with content "<|padding|>", while it's not in special_tokens_map.json and also in tokenizer_config.json which results in tokenizer.pad_token being None.