Hugging Face's logo

hf-internal-testing
/
llama3-tokenizer-tiktoken

test
tiktoken
tokenizer
Model card Files
xet
 Community
llama3-tokenizer-tiktoken

Commit History

  1. Delete tokenizer.model

    itazap HF Staff
    Verified signature
    a1d81e8

  2. Add tiktoken tokenizer files from Meta-Llama-3-8B-Instruct/original

    itazap HF Staff
    Verified signature
    c8bff8b

  3. Add tiktoken tokenizer files from Meta-Llama-3-8B-Instruct/original

    itazap HF Staff
    Verified signature
    7862ff8

  4. initial commit

    itazap HF Staff
    Verified signature
    5bc9fcf