Update tokenizer_class to PreTrainedTokenizerFast for GGUF compatibility dab7200 verified spitfire4794 commited on 2 days ago
Add standard contractions to pre-tokenizer Regex for GGUF compatibility feceaf6 verified spitfire4794 commited on 2 days ago
Update tokenizer_class to LlamaTokenizerFast for GGUF compatibility 6ccfa3b verified spitfire4794 commited on 2 days ago