Set tokenizer class to `PreTrainedTokenizerFast` 976ad55 verified Xenova HF Staff commited on Jul 19, 2024
Set `clean_up_tokenization_spaces` to false 6a4dc9f verified Xenova HF Staff commited on Jul 19, 2024
Do not add eos token by default (#8) 4f81d78 verified patrickvonplaten Xenova HF Staff commited on Jul 18, 2024
