michaelfeil commited on
Commit
eda65ed
·
verified ·
1 Parent(s): d1a3e8f

Add root fast tokenizer serialization

Browse files

Add tokenizer.json at the repository root for consumers that load the fast tokenizer serialization directly, including Infinity v3.

Converted from the original vocabulary and tokenizer configuration using AutoTokenizer with use_fast=True. The resulting JSON is identical to the existing onnx/tokenizer.json at this revision. No weights or tokenizer configuration are changed.

Validation: 156 comparisons of token IDs, offsets, token type IDs and attention masks covering casing, Unicode, sentence pairs, padding and left/right truncation. Infinity 3.0.0 GPU NER inference and all aggregation modes passed using this tokenizer.

Files changed (1) hide show
  1. tokenizer.json +0 -0
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff