Update tokenizer_class to PreTrainedTokenizerFast for GGUF compatibility 5d34c66 verified spitfire4794 commited on 14 days ago
Add standard contractions to pre-tokenizer Regex for GGUF compatibility f15cc49 verified spitfire4794 commited on 14 days ago
Update tokenizer_class to LlamaTokenizerFast for GGUF compatibility db101bf verified spitfire4794 commited on 14 days ago