Replace tokenizer.json with Xenova M2M-100 418M version (correct SP-aligned vocab IDs) for SMaLL-100 2063eec verified burakaydinofficial commited on Mar 30
Fix tokenizer.json: add language tokens to added_tokens for alirezamsh/small100 f09a6ec verified burakaydinofficial commited on Mar 30
Fix tokenizer.json for SMaLL-100 (correct BPE merge order) 5bdb2d4 verified burakaydinofficial commited on Mar 28
Fix tokenizer.json merge order for SMaLL-100 INT8 171300f verified burakaydinofficial commited on Mar 28
Fix tokenizer.json merge order for SMaLL-100 INT8 cc45707 verified burakaydinofficial commited on Mar 28