fix: load stage3 tokenizer from stage1a subfolder (stage3/tokenizer.json is incompatible) 492e26a itsLu commited on Mar 22
fix: use use_fast=False for stage3 tokenizer to bypass tokenizer.json bb3a4d7 itsLu commited on Mar 22
fix: pin tokenizers>=0.19 to fix stage3 tokenizer.json compatibility 125f123 itsLu commited on Mar 22
fix: use AutoTokenizer for stage3 to fix tokenizer.json version mismatch ce51983 itsLu commited on Mar 22