Commit History

  1. Upload README.md with huggingface_hub

    Verified signature
    36d0de6
  2. Upload tokenizer_8192.json with huggingface_hub

    Verified signature
    abe9ca9
  3. Upload tokenizer_50257.json with huggingface_hub

    Verified signature
    423f97f
  4. Upload tokenizer_4096.json with huggingface_hub

    Verified signature
    711b691
  5. Upload tokenizer_3072.json with huggingface_hub

    Verified signature
    7b728db
  6. Upload tokenizer_2560.json with huggingface_hub

    Verified signature
    bfd227b
  7. Upload tokenizer_2048.json with huggingface_hub

    Verified signature
    501950c
  8. Upload tokenizer_16384.json with huggingface_hub

    Verified signature
    667d1d0
  9. Upload tokenizer_1536.json with huggingface_hub

    Verified signature
    a8b7664
  10. Upload tokenizer_1024.json with huggingface_hub

    Verified signature
    1a4a14a

  1. Update README.md

    Verified signature
    32f7e91
  2. Update README.md

    Verified signature
    419c311
  3. Add full suite of BPE tokenizers (1k to 50k)⁠

    Verified signature
    da47eb4
  4. initial commit

    Verified signature
    0caa6e8