12.8 MB

Ctrl+K

1 contributor

Use the Qwen3.5 base pre-tokenizer and decoder (regex [\p{L}\p{M}]+); tokenizer_class TokenizersBackend so transformers loads tokenizer.json verbatim; supersedes 177cdc85 (T1's Qwen2 pre-tokenizer)

dc692bd verified 2 days ago

.gitattributes

1.57 kB
Chiboard 1.1 tokenizer v3: T1 + <|chiboard_screen_ref|>=248081, <|chiboard_screen|>=248082 2 days ago
README.md

3.82 kB
Use the Qwen3.5 base pre-tokenizer and decoder (regex [\p{L}\p{M}]+); tokenizer_class TokenizersBackend so transformers loads tokenizer.json verbatim; supersedes 177cdc85 (T1's Qwen2 pre-tokenizer) 2 days ago
chat_template.jinja

7.76 kB
Chiboard 1.1 tokenizer v3: T1 + <|chiboard_screen_ref|>=248081, <|chiboard_screen|>=248082 2 days ago
tokenizer.json

12.8 MB
xet

Use the Qwen3.5 base pre-tokenizer and decoder (regex [\p{L}\p{M}]+); tokenizer_class TokenizersBackend so transformers loads tokenizer.json verbatim; supersedes 177cdc85 (T1's Qwen2 pre-tokenizer) 2 days ago
tokenizer_config.json

1.28 kB
Use the Qwen3.5 base pre-tokenizer and decoder (regex [\p{L}\p{M}]+); tokenizer_class TokenizersBackend so transformers loads tokenizer.json verbatim; supersedes 177cdc85 (T1's Qwen2 pre-tokenizer) 2 days ago