Update README.md
Browse files
README.md
CHANGED
|
@@ -115,6 +115,18 @@ tokenizer.save_pretrained(model_path)
|
|
| 115 |
model.save_pretrained(model_path)
|
| 116 |
```
|
| 117 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
# Hyperparameters
|
| 119 |
|
| 120 |
MAX_SOURCE_LENGTH = 256 <br>
|
|
|
|
| 115 |
model.save_pretrained(model_path)
|
| 116 |
```
|
| 117 |
|
| 118 |
+
I then verified the token was added using this script:
|
| 119 |
+
|
| 120 |
+
```python
|
| 121 |
+
print("Token ID for <tdec>:", tokenizer.convert_tokens_to_ids("<tdec>"))
|
| 122 |
+
print("Tokenized form of '<tdec>':", tokenizer.tokenize("<tdec>"))
|
| 123 |
+
|
| 124 |
+
# Token ID for <tdec>: 32103
|
| 125 |
+
# Tokenized form of '<tdec>': ['<tdec>']
|
| 126 |
+
```
|
| 127 |
+
|
| 128 |
+
These scripts were run before fine tuning the model.
|
| 129 |
+
|
| 130 |
# Hyperparameters
|
| 131 |
|
| 132 |
MAX_SOURCE_LENGTH = 256 <br>
|