tokenizers/pr_2119/en/api/visualizer.md

Visualizer

Annotation[[tokenizers.tools.Annotation]]

EncodingVisualizer[[tokenizers.tools.EncodingVisualizer]]

• tokenizer (Tokenizer) -- A tokenizer instance

• default_to_notebook (bool) -- Whether to render html output in a notebook by default

• annotation_converter (Callable, optional) -- An optional (lambda) function that takes an annotation in any format and returns an Annotation object

Build an EncodingVisualizer

• text (str) -- The text to tokenize

• annotations (List[Annotation], optional) -- An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function

• default_to_notebook (bool, optional, defaults to False) -- If True, will render the html in a notebook. Otherwise returns an html string.The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook

Build a visualization of the given text

The Rust API Reference is available directly on the Docs.rs website.

The node API has not been documented yet.