Final Rescue: Full vocab restoration with Unigram scores and Metaspace fix

Final Fix: Correct Metaspace mapping and Unigram scores

Fix: Final Metaspace decoding using exact vocab marker

Fix: Robust SPM-to-HF conversion with merges and byte-fallback

Fix: Final Metaspace decoding using exact vocab marker

Fix: Used Llama-style identity wrapping to preserve SPM IDs and fix spacing

Fix: Final Metaspace decoding using exact vocab marker

Fix: Reverted to native SentencePiece handling (removed ByteLevel mismatch)

Fix: Applied ByteLevel pre-tokenization and decoding for proper spacing/hex handling

Fix: Clean base tokenizer using universal metaspace decoding

Fix: Universal Metaspace decoding without keyword conflicts

Fix: Manual vocab extraction from spm.model pieces

Fix: Environment issues resolved, full Fast Tokenizer uploaded.

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Upload folder using huggingface_hub

Update tokenizer_config.json

Update tokenizer_config.json

Update tokenizer_config.json

Upload folder using huggingface_hub

Upload folder using huggingface_hub

initial commit