Hugging Face's logo

Joycean0301
/
test-tokenizer

Transformers
Model card Files
xet
 Community
Joycean0301 commited on
Commit
2f127a6
·
verified ·
1 Parent(s): ff189f8

Update added_tokens.json

Browse files
Files changed (1) hide show
  1. added_tokens.json +2 -2
added_tokens.json CHANGED
@@ -38,6 +38,6 @@
38
  " ": 50258,
39
  " ": 50257,
40
  "<|padding|>": 50295,
41
- "<|beginoftext|>": 50296,
42
- "<|endoftext|>": 50297
43
  }
 
38
  " ": 50258,
39
  " ": 50257,
40
  "<|padding|>": 50295,
41
+ "<|beginoftext|>": 50300,
42
+ "<|endoftext|>": 50400
43
  }