Hugging Face's logo

AuriLab
/
gpt-bi

Text Generation
Transformers
Safetensors
Basque
gpt2
conversational
text-generation-inference
Model card Files
xet
Community
ZeusCabanas commited on
Commit
bf19f82
·
verified ·
1 Parent(s): 68d11c3

Upload tokenizer

Browse files
Files changed (5) hide show
  1. added_tokens.json +2 -2
  2. merges.txt +0 -0
  3. tokenizer.json +0 -0
  4. tokenizer_config.json +3 -3
  5. vocab.json +0 -0
added_tokens.json CHANGED
@@ -1,4 +1,4 @@
1
  {
2
- "<|im_end|>": 50258,
3
- "<|im_start|>": 50257
4
  }
 
1
  {
2
+ "<|im_end|>": 32001,
3
+ "<|im_start|>": 32000
4
  }
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "add_prefix_space": false,
3
  "added_tokens_decoder": {
4
- "50256": {
5
  "content": "<|endoftext|>",
6
  "lstrip": false,
7
  "normalized": false,
@@ -9,7 +9,7 @@
9
  "single_word": false,
10
  "special": true
11
  },
12
- "50257": {
13
  "content": "<|im_start|>",
14
  "lstrip": false,
15
  "normalized": false,
@@ -17,7 +17,7 @@
17
  "single_word": false,
18
  "special": true
19
  },
20
- "50258": {
21
  "content": "<|im_end|>",
22
  "lstrip": false,
23
  "normalized": false,
 
1
  {
2
  "add_prefix_space": false,
3
  "added_tokens_decoder": {
4
+ "0": {
5
  "content": "<|endoftext|>",
6
  "lstrip": false,
7
  "normalized": false,
 
9
  "single_word": false,
10
  "special": true
11
  },
12
+ "32000": {
13
  "content": "<|im_start|>",
14
  "lstrip": false,
15
  "normalized": false,
 
17
  "single_word": false,
18
  "special": true
19
  },
20
+ "32001": {
21
  "content": "<|im_end|>",
22
  "lstrip": false,
23
  "normalized": false,
vocab.json CHANGED
The diff for this file is too large to render. See raw diff