fix: add do_lower_case=True to tokenizer (v1.1.1) c7b1be1 diyclassics Claude Opus 4.6 (1M context) commited on 11 days ago
Fix tokenizer ID offset: reserve IDs 0-4 for BERT special tokens ce59834 diyclassics Claude Opus 4.6 commited on 18 days ago
Initial: HF-compatible Latin BERT tokenizer (Bamman & Burns 2020) 68d8806 diyclassics Claude Opus 4.6 commited on 18 days ago