Hugging Face's logo

RA-ALTA
/
tokenizer-en

Model card Files
xet
Community
tokenizer-en / tokenizer.json

Commit History

Final Rescue: Full vocab restoration with Unigram scores and Metaspace fix
8b94045
verified

suchirsalhan commited on

Final Fix: Correct Metaspace mapping and Unigram scores
5217fda
verified

suchirsalhan commited on

Fix: Final Metaspace decoding using exact vocab marker
ab058af
verified

suchirsalhan commited on

Fix: Robust SPM-to-HF conversion with merges and byte-fallback
13b4b4b
verified

suchirsalhan commited on

Fix: Final Metaspace decoding using exact vocab marker
3cac7ea
verified

suchirsalhan commited on

Fix: Reverted to native SentencePiece handling (removed ByteLevel mismatch)
5bece31
verified

suchirsalhan commited on

Fix: Applied ByteLevel pre-tokenization and decoding for proper spacing/hex handling
b0d7b42
verified

suchirsalhan commited on

Fix: Universal Metaspace decoding without keyword conflicts
2f8342a
verified

suchirsalhan commited on

Fix: Manual vocab extraction from spm.model pieces
23dd760
verified

suchirsalhan commited on

Upload folder using huggingface_hub
2f21705
verified

suchirsalhan commited on

Upload folder using huggingface_hub
1a52514
verified

suchirsalhan commited on

Upload folder using huggingface_hub
6887103
verified

suchirsalhan commited on

Upload folder using huggingface_hub
8835ef9
verified

suchirsalhan commited on

Upload folder using huggingface_hub
c530f27
verified

suchirsalhan commited on

Upload folder using huggingface_hub
d09c06e
verified

suchirsalhan commited on