Hugging Face's logo

tspersian
/
mana_tokenizer

Persian
English
Arabic
Model card Files
xet
Community
tspersian commited on
Commit
04d6e15
·
1 Parent(s): 50eafb3

regex

Browse files
Files changed (2) hide show
  1. README.md +1 -1
  2. mana.model +2 -2
README.md CHANGED
@@ -45,7 +45,7 @@ tokenizer.batch_encode(["یک متن طولانی"])
45
  - **Benchmark DateTime:** 2024-11-06 16:12:50
46
  - **Mana Batch Encode Time:** 0.10711932182312012 seconds
47
  - **Mana Batch Encode Memory Usage:** 13.203125 KB
48
- - **Total characters in benchmark:** 131000
49
 
50
  ## Special Tokens
51
 
 
45
  - **Benchmark DateTime:** 2024-11-06 16:12:50
46
  - **Mana Batch Encode Time:** 0.10711932182312012 seconds
47
  - **Mana Batch Encode Memory Usage:** 13.203125 KB
48
+ - **Total characters in benchmark:** 131,000
49
 
50
  ## Special Tokens
51
 
mana.model CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e666a42308d210e029f8e9aa8c0056950ecd785e61514230fda35ec2962aa490
3
- size 2915213
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46f22baf03e043b23c7bda2a2a0453e350de271a7d53840d036e418fd094a019
3
+ size 2915210