| tags: | |
| - test | |
| - tiktoken | |
| - tokenizer | |
| # Llama 3 TikToken Tokenizer (Test Model) | |
| This is a test model for `hf-internal-testing` containing only the tiktoken tokenizer files from `meta-llama/Meta-Llama-3-8B-Instruct`. | |
| This model contains the tiktoken format tokenizer.model file for testing purposes. | |
| ## Files | |
| - `tiktoken.model` - TikToken format tokenizer (renamed from tokenizer.model) | |
| - `tokenizer_config.json` - Tokenizer configuration | |
| - `special_tokens_map.json` - Special tokens mapping | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("hf-internal-testing/llama3-tokenizer-tiktoken") | |
| ``` | |
| ## Source | |
| This is a subset of files from [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), specifically from the `original/` subfolder. | |