tokenizers/pr_2502/en/api/visualizer.md

Visualizer

Annotation[[tokenizers.tools.Annotation]]

tokenizers.tools.Annotation[[tokenizers.tools.Annotation]]

tokenizers.tools.Annotation(start: int, end: int, label: str)

Source

EncodingVisualizer[[tokenizers.tools.EncodingVisualizer]]

tokenizers.tools.EncodingVisualizer[[tokenizers.tools.EncodingVisualizer]]

tokenizers.tools.EncodingVisualizer(tokenizer: Tokenizer, default_to_notebook: bool = True, annotation_converter: typing.Optional[typing.Callable[[typing.Any], tokenizers.tools.visualizer.Annotation]] = None)

Source

Parameters:

tokenizer (Tokenizer) : A tokenizer instance

default_to_notebook (bool) : Whether to render html output in a notebook by default

annotation_converter (Callable, optional) : An optional (lambda) function that takes an annotation in any format and returns an Annotation object

Build an EncodingVisualizer

call[[tokenizers.tools.EncodingVisualizer.call]]

__call__(text: str, annotations: typing.Optional[typing.List[typing.Any]] = None, default_to_notebook: typing.Optional[bool] = None)

Source

Parameters:

text (str) : The text to tokenize

annotations (List[Annotation], optional) : An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function

default_to_notebook (bool, optional, defaults to False) : If True, will render the html in a notebook. Otherwise returns an html string.

Returns:

The HTML string if default_to_notebook is False, otherwise (default) returns None and renders the HTML in the notebook

Build a visualization of the given text

The Rust API Reference is available directly on the Docs.rs website.

The node API has not been documented yet.