Hugging Face's logo

nroggendorff
/
dictionary

Transformers
Model card Files
xet
Community
nroggendorff commited on
Commit
be80182
·
verified ·
1 Parent(s): 2db1964

Upload tokenizer

Browse files
Files changed (3) hide show
  1. added_tokens.json +1 -1
  2. tokenizer_config.json +1 -1
  3. vocab.json +0 -0
added_tokens.json CHANGED
@@ -1,3 +1,3 @@
1
  {
2
- "<|endoftext|>": 123916
3
  }
 
1
  {
2
+ "<|endoftext|>": 75524
3
  }
tokenizer_config.json CHANGED
@@ -2,7 +2,7 @@
2
  "add_bos_token": false,
3
  "add_prefix_space": false,
4
  "added_tokens_decoder": {
5
- "123916": {
6
  "content": "<|endoftext|>",
7
  "lstrip": false,
8
  "normalized": true,
 
2
  "add_bos_token": false,
3
  "add_prefix_space": false,
4
  "added_tokens_decoder": {
5
+ "75524": {
6
  "content": "<|endoftext|>",
7
  "lstrip": false,
8
  "normalized": true,
vocab.json CHANGED
The diff for this file is too large to render. See raw diff