Commit History

Update tokenizer_config for fast tokenizer (keep auto_map for backwards compat)
e45ad24
verified

DanieleSalatti commited on

Add tokenizer.json for AutoTokenizer support and download tracking
e266751
verified

DanieleSalatti commited on

Update README with AutoTokenizer usage
0abb4d8
verified

DanieleSalatti commited on

Update tokenizer_config.json with auto_map for AutoTokenizer
fc201ae
verified

DanieleSalatti commited on

Add custom LatinTokenizer for AutoTokenizer support
937f093
verified

DanieleSalatti commited on

Upload README.md with huggingface_hub
68129ff
verified

DanieleSalatti commited on

Upload folder using huggingface_hub
0b17963
verified

DanieleSalatti commited on

initial commit
9055c2b
verified

DanieleSalatti commited on