move AutoTokenizer auto_map to tokenizer_config.json only 79ace9d George Treacy commited on about 19 hours ago
fix tokenizer resolution: use auto_map only, remove tokenizer_class 66ab6c3 George Treacy commited on about 20 hours ago
add MakemoreTokenizer, wire AutoTokenizer, drop tokenizer.json 9e25bfd George Treacy commited on about 20 hours ago