bpe_script_SEAS_16000 / script_1 /participating_tokenizers.json
gsaltintas's picture
Upload script_1/participating_tokenizers.json with huggingface_hub
61a4274 verified
raw
history blame contribute delete
389 Bytes
["flexitok/bpe_script_Arab_16000", "flexitok/bpe_script_CmJp_16000", "flexitok/bpe_ltr_ell_Grek_8000_v2", "flexitok/bpe_ltr_fw_edu_32000_v2", "flexitok/bpe_ltr_hun_Latn_8000_v2", "flexitok/bpe_ltr_rus_Cyrl_16000_v2", "flexitok/bpe_ltr_tur_Latn_8000_v2", "flexitok/bpe_script_Germ_32000", "flexitok/bpe_script_Roma_32000", "flexitok/bpe_script_SEAS_16000", "flexitok/bpe_script_Slav_16000"]