Hugging Face's logo

shirzady1934
/
parsig_tokenizer

Model card Files
xet
 Community
shirzady1934 commited on
Commit
4ff3818
·
1 Parent(s): da643fb

Upload tokenizer

Browse files
Files changed (4) hide show
  1. added_tokens.json +1 -1
  2. merges.txt +0 -0
  3. tokenizer.json +0 -0
  4. vocab.json +0 -0
added_tokens.json CHANGED
@@ -1,3 +1,3 @@
1
  {
2
- "<|endoftext|>": 7852
3
  }
 
1
  {
2
+ "<|endoftext|>": 6881
3
  }
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
vocab.json CHANGED
The diff for this file is too large to render. See raw diff