Hugging Face's logo

rtferraz
/
domainTokenizer

Model card Files
xet
 Community
domainTokenizer / src /domain_tokenizer
72.6 kB
  • 1 contributor
 History: 22 commits
rtferraz's picture
rtferraz
CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences
a9c4a62 verified