| # Tokenizer | |
| <tokenizerslangcontent> | |
| <python> | |
| ## Tokenizer | |
| [[autodoc]] tokenizers.Tokenizer | |
| - all | |
| - decoder | |
| - model | |
| - normalizer | |
| - padding | |
| - post_processor | |
| - pre_tokenizer | |
| - truncation | |
| </python> | |
| <rust> | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
| </rust> | |
| <node> | |
| The node API has not been documented yet. | |
| </node> | |
| </tokenizerslangcontent> |