rtferraz
/

domainTokenizer

Model card Files Files and versions

domainTokenizer / src

72.6 kB

Ctrl+K

Ctrl+K

1 contributor

History: 22 commits

rtferraz's picture

CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences

a9c4a62 verified 1 day ago

domain_tokenizer
CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences 1 day ago