Hugging Face's logo

flexitok
/
maddition_digit_2000

Digo
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
 Community
gsaltintas commited on
Commit
c75b118
·
verified ·
1 Parent(s): 5532c98

Upload maddition/participating_tokenizers.json with huggingface_hub

Browse files
Files changed (1) hide show
  1. maddition/participating_tokenizers.json +1 -0
maddition/participating_tokenizers.json ADDED
@@ -0,0 +1 @@
 
 
1
+ ["flexitok/maddition_jpn_Jpan_2000", "flexitok/maddition_cmn_Hani_2000", "flexitok/maddition_ita_Latn_2000", "flexitok/maddition_por_Latn_2000", "flexitok/maddition_swe_Latn_2000", "flexitok/maddition_deu_Latn_2000", "flexitok/maddition_ind_Latn_2000", "flexitok/maddition_fas_Arab_2000", "flexitok/maddition_vie_Latn_2000", "flexitok/maddition_nld_Latn_2000", "flexitok/maddition_fra_Latn_2000", "flexitok/maddition_spa_Latn_2000", "flexitok/maddition_dan_Latn_2000", "flexitok/maddition_eng_Latn_2000", "flexitok/maddition_arb_Arab_2000", "flexitok/maddition_ces_Latn_2000", "flexitok/maddition_hun_Latn_2000", "flexitok/maddition_tur_Latn_2000", "flexitok/maddition_pol_Latn_2000", "flexitok/maddition_ell_Grek_2000", "flexitok/maddition_rus_Cyrl_2000", "flexitok/maddition_digit_2000"]