move AutoTokenizer auto_map to tokenizer_config.json only 79ace9d George Treacy commited on about 12 hours ago
fix tokenizer resolution: use auto_map only, remove tokenizer_class 66ab6c3 George Treacy commited on about 12 hours ago
add tokenizer_config.json for AutoTokenizer resolution f673925 George Treacy commited on about 13 hours ago
add all_tied_weights_keys property for transformers compat 3b04946 George Treacy commited on about 13 hours ago
add MakemoreTokenizer, wire AutoTokenizer, drop tokenizer.json 9e25bfd George Treacy commited on about 13 hours ago