j32-tokenizer-v2 / tokenizer.model

Commit History

24 days ago

  1. Add j32-tokenizer-v2 (SentencePiece Unigram, vocab 32k)

    Verified signature
    e755e4f