Hugging Face's logo

flexitok
/
bpe_jpn_Jpan_8000

Japanese
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community
bpe_jpn_Jpan_8000

Commit History

Upload fineweb2_hq/flexitok--bpe_jpn_Jpan_8000_overlap.json with huggingface_hub
c8982f1
verified

gsaltintas commited on

Upload fineweb2_hq/participating_tokenizers.json with huggingface_hub
1934bfb
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_jpn_Jpan_8000.yaml with huggingface_hub
b2a35be
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_jpn_Jpan_8000_info.json with huggingface_hub
7262f9e
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_jpn_Jpan_8000_vocab.json with huggingface_hub
f168280
verified

gsaltintas commited on

Upload fineweb2_hq/flexitok--bpe_jpn_Jpan_8000_super_mapping.json with huggingface_hub
b40db6b
verified

gsaltintas commited on

Upload folder using huggingface_hub
42f231e
verified

gsaltintas commited on

initial commit
1111988
verified

gsaltintas commited on