tokenizers/pr_2139/en/llms.txt

# Tokenizers

## Docs

- [Components](https://huggingface.co/docs/tokenizers/pr_2139/components.md)
- [Tokenizers](https://huggingface.co/docs/tokenizers/pr_2139/index.md)
- [Training from memory](https://huggingface.co/docs/tokenizers/pr_2139/training_from_memory.md)
- [The tokenization pipeline](https://huggingface.co/docs/tokenizers/pr_2139/pipeline.md)
- [Quicktour](https://huggingface.co/docs/tokenizers/pr_2139/quicktour.md)
- [Installation](https://huggingface.co/docs/tokenizers/pr_2139/installation.md)
- [Visualizer](https://huggingface.co/docs/tokenizers/pr_2139/api/visualizer.md)
- [Trainers](https://huggingface.co/docs/tokenizers/pr_2139/api/trainers.md)
- [Encode Inputs](https://huggingface.co/docs/tokenizers/pr_2139/api/encode-inputs.md)
- [Input Sequences](https://huggingface.co/docs/tokenizers/pr_2139/api/input-sequences.md)
- [Encoding](https://huggingface.co/docs/tokenizers/pr_2139/api/encoding.md)
- [Models](https://huggingface.co/docs/tokenizers/pr_2139/api/models.md)
- [Pre-tokenizers](https://huggingface.co/docs/tokenizers/pr_2139/api/pre-tokenizers.md)
- [Normalizers](https://huggingface.co/docs/tokenizers/pr_2139/api/normalizers.md)
- [Post-processors](https://huggingface.co/docs/tokenizers/pr_2139/api/post-processors.md)
- [Decoders](https://huggingface.co/docs/tokenizers/pr_2139/api/decoders.md)
- [Added Tokens](https://huggingface.co/docs/tokenizers/pr_2139/api/added-tokens.md)
- [Tokenizer](https://huggingface.co/docs/tokenizers/pr_2139/api/tokenizer.md)