Buckets:
| # Visualizer | |
| ## Annotation[[tokenizers.tools.Annotation]] | |
| ## EncodingVisualizer[[tokenizers.tools.EncodingVisualizer]] | |
| - **tokenizer** ([Tokenizer](/docs/tokenizers/pr_2139/en/api/tokenizer#tokenizers.Tokenizer)) -- | |
| A tokenizer instance | |
| - **default_to_notebook** (`bool`) -- | |
| Whether to render html output in a notebook by default | |
| - **annotation_converter** (`Callable`, *optional*) -- | |
| An optional (lambda) function that takes an annotation in any format and returns | |
| an Annotation object | |
| Build an EncodingVisualizer | |
| - **text** (`str`) -- | |
| The text to tokenize | |
| - **annotations** (`List[Annotation]`, *optional*) -- | |
| An optional list of annotations of the text. The can either be an annotation class | |
| or anything else if you instantiated the visualizer with a converter function | |
| - **default_to_notebook** (`bool`, *optional*, defaults to *False*) -- | |
| If True, will render the html in a notebook. Otherwise returns an html string.The HTML string if default_to_notebook is False, otherwise (default) returns None and | |
| renders the HTML in the notebook | |
| Build a visualization of the given text | |
| The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. | |
| The node API has not been documented yet. | |
Xet Storage Details
- Size:
- 1.27 kB
- Xet hash:
- c9dde5a443574b6e523f37e3088bd108c687a1820837c1e53890b8a00d79690c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.