Commit History
-
Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020)
PreTrainedTokenizer wrapping the original tensor2tensor SubwordTextEncoder. Produces identical token IDs to the original (verified against POS case study: Perseus 95.2%, ITTB 99.2%, PROIEL 98.2%). No TensorFlow dependency. 13/13 tests passing: encoding, roundtrip decode, save/load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>68d8806
