vesteinn

/

ScandiBERT

like 5

This changes the tokenizer class to PreTrainedTokenizerFast, which allows it to be loaded properly with the TokenizersBackend. This is because it uses a BPE tokenizer and not the unigram tokenizer that XLM-RoBERTa does.

Source: https://github.com/huggingface/transformers/issues/47020#issuecomment-4893582482