Hugging Face's logo

flexitok
/
bpe_cmn_Hani_8000

Mandarin Chinese
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
bpe_cmn_Hani_8000

Commit History

Upload fineweb2_hq/flexitok--bpe_cmn_Hani_8000_overlap.json with huggingface_hub
dc85a41
verified

gsaltintas commited on

Upload fineweb2_hq/participating_tokenizers.json with huggingface_hub
6ca9c16
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_cmn_Hani_8000.yaml with huggingface_hub
7fcfbca
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_cmn_Hani_8000_info.json with huggingface_hub
265780f
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_cmn_Hani_8000_vocab.json with huggingface_hub
a3c149a
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_cmn_Hani_8000_super_mapping.json with huggingface_hub
37d4139
verified

gsaltintas commited on

Upload folder using huggingface_hub
49ed2ef
verified

gsaltintas commited on

initial commit
a350242
verified

gsaltintas commited on