Update README.md
Browse files
README.md
CHANGED
|
@@ -19,7 +19,7 @@ output = model.generate(**tokenizer(text, return_tensors="pt", add_special_token
|
|
| 19 |
tokenizer.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
|
| 20 |
# Output: ' Das ist meine Katze.'
|
| 21 |
```
|
| 22 |
-
(remember the `trust_remote_code=True` because of custom modeling
|
| 23 |
## Training:
|
| 24 |
| Parameter | Value |
|
| 25 |
|----------------------|-------------------------------------------------------------------------------------------------|
|
|
@@ -31,4 +31,8 @@ tokenizer.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spac
|
|
| 31 |
| Effective Batch Size | 128 (16 * 8) |
|
| 32 |
| Training Script | [train.py](https://github.com/ubaada/scratch-transformer/blob/main/train.py) |
|
| 33 |
| Optimiser | Adam (learning rate = 0.0001) |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
|
|
|
| 19 |
tokenizer.decode(output[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
|
| 20 |
# Output: ' Das ist meine Katze.'
|
| 21 |
```
|
| 22 |
+
(remember the `trust_remote_code=True` because of custom modeling file)
|
| 23 |
## Training:
|
| 24 |
| Parameter | Value |
|
| 25 |
|----------------------|-------------------------------------------------------------------------------------------------|
|
|
|
|
| 31 |
| Effective Batch Size | 128 (16 * 8) |
|
| 32 |
| Training Script | [train.py](https://github.com/ubaada/scratch-transformer/blob/main/train.py) |
|
| 33 |
| Optimiser | Adam (learning rate = 0.0001) |
|
| 34 |
+
| Loss Type | Cross Entropy |
|
| 35 |
+
| Final Test Loss | 1.9 |
|
| 36 |
+
| GPU. | RTX 4070 (12GB) |
|
| 37 |
+
|
| 38 |
|