Tokenizers documentation
Visualizer
Visualizer
Python
Rust
Node
Annotation
EncodingVisualizer
class tokenizers.tools.EncodingVisualizer
< source >( tokenizer: Tokenizer default_to_notebook: bool = True annotation_converter: Optional = None )
Parameters
- tokenizer (Tokenizer) — A tokenizer instance
- default_to_notebook (
bool) — Whether to render html output in a notebook by default - annotation_converter (
Callable, optional) — An optional (lambda) function that takes an annotation in any format and returns an Annotation object
Build an EncodingVisualizer
__call__
< source >( text: str annotations: List = [] default_to_notebook: Optional = None )
Parameters
- text (
str) — The text to tokenize - annotations (
List[Annotation], optional) — An optional list of annotations of the text. The can either be an annotation class or anything else if you instantiated the visualizer with a converter function - default_to_notebook (
bool, optional, defaults to False) — If True, will render the html in a notebook. Otherwise returns an html string.
Build a visualization of the given text