toksuite/mistralai-tekken
Text Generation • 2B • Updated
• 17
Edit Models filters
English
Chinese
French
Spanish
German
Japanese
Korean
Portuguese
Italian
Russian
Hindi
Arabic
Thai
Turkish
multilingual
Vietnamese
Indonesian
Polish
Dutch
Romanian
Swedish
Ukrainian
Persian
Czech
Finnish
Bengali
Nepali
Danish
Greek
Hebrew
Malay
Tamil
Hungarian
Urdu
Bulgarian
Catalan
Telugu
Norwegian
French
Swahili
Marathi
Serbian
Slovak
Gujarati
Slovenian
Estonian
Burmese
Croatian
Malayalam
Tagalog
Lithuanian
Galician
Latvian
Khmer
Kannada
Basque
Icelandic
Panjabi
Amharic
Lao
Afrikaans
Kazakh
Mongolian
Georgian
Hausa
Assamese
Armenian
Welsh
Macedonian
Sinhala
Belarusian
Azerbaijani
Uzbek
Yoruba
Javanese
English
Irish
Sundanese
Albanian
Bosnian
Latin
Maltese
Somali
Sanskrit
Sindhi
Oriya
code
Spanish
Thai
Russian
Active filters: bpe
toksuite/Qwen-Qwen3-8B
Text Generation • 2B • Updated
• 68
nursimakgul/turkish-tokenizer
Updated
toksuite/tiktoken-gpt-4o
Text Generation • 2B • Updated
• 73
nursimakgul/turkish-fen-tokenizer-v1
Updated
toksuite/CohereLabs-aya-expanse-8b
Text Generation • 2B • Updated
• 7
Rogarcia18/wikitext2-bpe-tokenizer
Updated
PABITRA07/hindi-english-BPE-nmt
Updated
manoskary/miditok-REMI
Updated
Msok99/lfm2-khmer-merged-18k
Updated
SherifAnar/russian-bpe-16k
Updated
mjbommar/glaurung-binary-tokenizer-001
Feature Extraction • Updated
mjbommar/glaurung-binary-tokenizer-002
Feature Extraction • Updated
• 1
pandurangpatil/sample-marathi-bpe-tokenizer
Updated
• 1
OpenMachineAI/UnitronX-Tokenizer-32k-v1
Updated
ansul90/hindi-bpe-tokenizer
Updated
vnaren13/telugu-bpe-tokenizer
Updated
agileabhi/hindi-bpe-tokenizer
Token Classification • Updated
dar5115/NLP_BPE
Updated
mjbommar/binary-tokenizer-001-4k
Feature Extraction • Updated
mjbommar/binary-tokenizer-001-8k
Feature Extraction • Updated
mjbommar/binary-tokenizer-001-16k
Feature Extraction • Updated
mjbommar/binary-tokenizer-001-64k
Feature Extraction • Updated
mjbommar/binary-tokenizer-001-32k
Feature Extraction • Updated
shwethd/kannada-tokenizer-50k
Updated
Soldier-Boy/twitch-tokenizer
Updated
alea-institute/kl3m-multi-word-001-4k
Fill-Mask • Updated
alea-institute/kl3m-multi-word-001-8k
Fill-Mask • Updated
alea-institute/kl3m-multi-word-001-16k
Fill-Mask • Updated
alea-institute/kl3m-multi-word-001-32k
Fill-Mask • Updated