ggunio
/

intelligent-tokenizer-v6

intelligent_tokenizer

neural-tokenizer

pattern-learning

vocabulary-free

Eval Results (legacy)

Model card Files Files and versions

intelligent-tokenizer-v6 / src /core

8.66 kB

Ctrl+K

Ctrl+K

1 contributor

History: 1 commit

ggunio's picture

Upload src/core/byte_tokenizer_v6.py with huggingface_hub

0a81e70 verified 8 months ago

byte_tokenizer_v6.py

8.66 kB
Upload src/core/byte_tokenizer_v6.py with huggingface_hub 8 months ago