Remove unused tokenizer training code (train_bpe, load_sentencepiece, load_trained_hf) 33ba3d1 Vjeong Claude Opus 4.6 commited on 15 days ago
Use LLaMA 2 pretrained tokenizer and remove tokenizer_mode option a5ca4e4 Vjeong Claude Opus 4.6 commited on 19 days ago
Fix BPE tokenizer ByteLevel decoder and update evaluation notebook 8626149 Vjeong Claude Sonnet 4.6 commited on 21 days ago
docs: translate all Korean comments and docstrings to English 858e8b2 Vjeong Claude Sonnet 4.6 commited on Feb 27