Commit History

  1. Revert back to NFD tokenizer

    Verified signature
    3341d84
  2. Remove normalizer

    Verified signature
    767e594
  3. Switch from NFD -> NFC

    Verified signature
    0dcea10

  1. Update README.md

    Verified signature
    6d5f592
  2. Upload tokenizer

    Verified signature
    57dcf89

  1. initial commit

    Verified signature
    74fae32