Hugging Face's logo

RA-ALTA
/
tokenizer-zh

Model card Files
xet
Community
tokenizer-zh / tokenizer.json

Commit History

Final Rescue: Full vocab restoration with Unigram scores and Metaspace fix
24cc4f3
verified

suchirsalhan commited on

Final Fix: Correct Metaspace mapping and Unigram scores
62d1849
verified

suchirsalhan commited on

Fix: Final Metaspace decoding using exact vocab marker
0b1bb48
verified

suchirsalhan commited on

Fix: Robust SPM-to-HF conversion with merges and byte-fallback
286d6b7
verified

suchirsalhan commited on

Fix: Final Metaspace decoding using exact vocab marker
4027cb6
verified

suchirsalhan commited on

Fix: Reverted to native SentencePiece handling (removed ByteLevel mismatch)
b26d639
verified

suchirsalhan commited on

Fix: Applied ByteLevel pre-tokenization and decoding for proper spacing/hex handling
5548968
verified

suchirsalhan commited on

Fix: Universal Metaspace decoding without keyword conflicts
f9c19ca
verified

suchirsalhan commited on

Fix: Manual vocab extraction from spm.model pieces
deacafe
verified

suchirsalhan commited on

Upload folder using huggingface_hub
9dec524
verified

suchirsalhan commited on

Upload folder using huggingface_hub
8aedd17
verified

suchirsalhan commited on

Upload folder using huggingface_hub
db3af0d
verified

suchirsalhan commited on

Upload folder using huggingface_hub
f979d10
verified

suchirsalhan commited on

Upload folder using huggingface_hub
bae84db
verified

suchirsalhan commited on

Upload folder using huggingface_hub
a5afb11
verified

suchirsalhan commited on