supertokenizer-industry-bigscience-bloom / participating_tokenizers.json
gsaltintas's picture
Upload participating_tokenizers.json with huggingface_hub
c0eb608 verified
raw
history blame contribute delete
358 Bytes
["google/byt5-small", "tiktoken/gpt-4o", "google-bert/bert-base-multilingual-cased", "facebook/xglm-564M", "google/gemma-2-2b", "microsoft/Phi-3-mini-4k-instruct", "tokenmonster/englishcode-32000-consistent-v1", "bigscience/bloom", "gpt2", "mistralai/tekken", "meta-llama/Llama-3.2-1B", "Qwen/Qwen3-8B", "CohereLabs/aya-expanse-8b", "common-pile/comma-v0.1"]