Commit History
-
fix: whitespace-split before subword encoding (full-string tokenization)
The HF tokenizer's full-string path char-class-split the entire string without dropping inter-word spaces (unlike tensor2tensor tokenizer.encode and gen_berts.py's LatinTokenizer, which split on whitespace first). Every space was escaped to `\32;_`, injecting 3 spurious subtokens between words — input the model never trained on. This corrupted masked-word predictions and broke subword->word alignment for the documented `tokenizer("sentence")` usage, while the word-by-word case-study path stayed correct (which is why validation passed). Reported in two independent bug reports. Fix (slow + root copy + fast): split on whitespace first, then char-class encode each word. Native full-string output now equals the original per-word tokenization (verified: slow == fast IDs). Decode renders word boundaries as spaces (BERT-style; "tres." -> "tres ."). Also fixes a latent fast-tokenizer bug: it ignored do_lower_case (no Rust normalizer), escaping capitals to codepoints (`Gallia` -> `\71;allia`) …b8d6c6c
-
fix: add do_lower_case=True to tokenizer (v1.1.1)
The original Latin BERT vocab was trained on lowercased text and all original case studies called .lower() before tokenizing. The HF wrapper was missing this step, causing uppercase characters to be escaped to ASCII codepoints (e.g. C → \67;), inflating token counts ~4x and producing embeddings the model never saw during pretraining. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c7b1be1
-
Fix tokenizer ID offset: reserve IDs 0-4 for BERT special tokens
The original Latin BERT model expects [PAD]=0, [UNK]=1, [CLS]=2, [SEP]=3, [MASK]=4 with SubwordTextEncoder IDs shifted by +5. This commit adds those five special tokens, shifts all subtoken IDs accordingly, and implements build_inputs_with_special_tokens so encode(add_special_tokens=True) wraps with [CLS]/[SEP]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>ce59834
-
Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020)
PreTrainedTokenizer wrapping the original tensor2tensor SubwordTextEncoder. Produces identical token IDs to the original (verified against POS case study: Perseus 95.2%, ITTB 99.2%, PROIEL 98.2%). No TensorFlow dependency. 13/13 tests passing: encoding, roundtrip decode, save/load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>68d8806