| # Added Tokens | |
| <tokenizerslangcontent> | |
| <python> | |
| ## AddedToken | |
| [[autodoc]] tokenizers.AddedToken | |
| - content | |
| - lstrip | |
| - normalized | |
| - rstrip | |
| - single_word | |
| </python> | |
| <rust> | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
| </rust> | |
| <node> | |
| The node API has not been documented yet. | |
| </node> | |
| </tokenizerslangcontent> |