Upload folder using huggingface_hub

README.md

@@ -1,7 +1,8 @@
 ---
 license: mit
 language:
-- cmn  # ISO 639-3 code or "und" if not identifiable
+- cmn #
+- jpn #['cmn_Hani', 'jpn_Jpan']  # ISO 639-3 code or "und" if not identifiable
 tags:
 - tokenizer
 - bpe
@@ -9,24 +10,24 @@ tags:
 - fineweb2
 ---
 
-# Byte-Level BPE Tokenizer: cmn_Hani (16K)
+# Byte-Level BPE Tokenizer: ['cmn_Hani', 'jpn_Jpan'] (16K)
 
-A **Byte-Level BPE** tokenizer trained on **cmn_Hani** data from Fineweb-2-HQ.
+A **Byte-Level BPE** tokenizer trained on **['cmn_Hani', 'jpn_Jpan']** data from Fineweb-2-HQ.
 
 ## Training Details
 
 | Parameter | Value |
 |-----------|-------|
 | Algorithm | Byte-Level BPE |
-| Language | `cmn_Hani` |
+| Language | `['cmn_Hani', 'jpn_Jpan']` |
 | Target Vocab Size | 16,000 |
-| Final Vocab Size | 19,127 |
-| Pre-tokenizer | custom:cmn_Hani |
+| Final Vocab Size | 17,900 |
+| Pre-tokenizer | custom:jpn_Jpan |
 | Number handling | ltr_3digit |
 | Contraction handling | True |
 | Normalizer | NFC |
 | Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` |
-| Training Shards | 2 |
+| Training Shards | 4 |
 
 ## Usage
 

merges.txt

@@ -5926,9 +5926,9 @@
 ['æĮĩ', 'åįĹ']
 ['个', 'ä½ĵ']
 ['èĥ', 'ļ']
+['Â', '°']
 ['å°¼', 'æĸ¯']
 ['å¿ĺ', 'è®°']
-['Â', '°']
 ['èĬ±', 'è´¹']
 ['æİ¥', 'ç§į']
 ['ç»Ī', 'æŃ¢']
@@ -10661,8 +10661,8 @@
 ['r', 'ic']
 ['åı²', 'å¯Ĩ']
 ['åIJ¯', 'åıij']
-['å·²', 'çŁ¥']
 ['Â', '®']
+['å·²', 'çŁ¥']
 ['å¤Ħ', 'åľ¨']
 ['çļĦ', 'æĿ¡ä»¶']
 ['äºĮ', '级']