rtferraz
/

domainTokenizer

Model card Files Files and versions

domainTokenizer / src /domain_tokenizer /tokenizers

23.2 kB

Ctrl+K

Ctrl+K

1 contributor

History: 4 commits

rtferraz's picture

CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences

a9c4a62 verified 1 day ago

__init__.py

269 Bytes
Add tokenizers package init 8 days ago
domain_tokenizer.py

10.4 kB
CRITICAL FIX: Switch from ByteLevel to Whitespace pre-tokenizer — fixes 42% UNK rate on domain token sequences 1 day ago
field_tokenizers.py

12.5 kB
Add field_tokenizers.py — Sign, MagnitudeBucket, Calendar, Categorical, DiscreteNumerical tokenizers 8 days ago