Tokenizer cannot recognise unused tokens in its vocab
#13
by
ArvinZhuang
- opened
Sometimes we can leverage unused tokens for some tasks.
But this can be fixed by self.tokenizer.add_special_tokens({"additional_special_tokens": ["[unused39]"]})
ArvinZhuang
changed discussion title from
Tokenizer cannot recognise unused tokens.
to Tokenizer cannot recognise unused tokens in its vocab