| license: llama2 | |
| This is a copy of the llama2 tokenizer for use as a fallback tokenizer for KoboldAI, optimized with defaults for text completion. | |
| We aim to keep this copy functional / identical to the upstream llama2 tokenizer with minor differences in its defaults. | |
| In case of differences a more functional copy is chosen. |