Mir-2002 commited on
Commit
aa3ead3
·
verified ·
1 Parent(s): 4dd2328

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -0
README.md CHANGED
@@ -115,6 +115,18 @@ tokenizer.save_pretrained(model_path)
115
  model.save_pretrained(model_path)
116
  ```
117
 
 
 
 
 
 
 
 
 
 
 
 
 
118
  # Hyperparameters
119
 
120
  MAX_SOURCE_LENGTH = 256 <br>
 
115
  model.save_pretrained(model_path)
116
  ```
117
 
118
+ I then verified the token was added using this script:
119
+
120
+ ```python
121
+ print("Token ID for <tdec>:", tokenizer.convert_tokens_to_ids("<tdec>"))
122
+ print("Tokenized form of '<tdec>':", tokenizer.tokenize("<tdec>"))
123
+
124
+ # Token ID for <tdec>: 32103
125
+ # Tokenized form of '<tdec>': ['<tdec>']
126
+ ```
127
+
128
+ These scripts were run before fine tuning the model.
129
+
130
  # Hyperparameters
131
 
132
  MAX_SOURCE_LENGTH = 256 <br>