Add completed Task 2 checkpoints and evaluation metrics
Browse filesFive optimizer reruns, best/final model weights, full evaluation records, current training source, and loading instructions.
- README.md +53 -5
- checkpoints_part2/adamw/oj3opt7s/best.pt +3 -0
- checkpoints_part2/adamw/oj3opt7s/last.pt +3 -0
- checkpoints_part2/adamw/oj3opt7s/metrics.json +129 -0
- checkpoints_part2/cautious/p2yw4r11/best.pt +3 -0
- checkpoints_part2/cautious/p2yw4r11/last.pt +3 -0
- checkpoints_part2/cautious/p2yw4r11/metrics.json +129 -0
- checkpoints_part2/lion/ydyarkxk/best.pt +3 -0
- checkpoints_part2/lion/ydyarkxk/last.pt +3 -0
- checkpoints_part2/lion/ydyarkxk/metrics.json +129 -0
- checkpoints_part2/scion/89yiklpr/best.pt +3 -0
- checkpoints_part2/scion/89yiklpr/last.pt +3 -0
- checkpoints_part2/scion/89yiklpr/metrics.json +129 -0
- checkpoints_part2/sophia/3j32zwf3/best.pt +3 -0
- checkpoints_part2/sophia/3j32zwf3/last.pt +3 -0
- checkpoints_part2/sophia/3j32zwf3/metrics.json +130 -0
- src/part2/train.py +57 -3
README.md
CHANGED
|
@@ -1,8 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# ANLP Assignment 2
|
| 2 |
|
| 3 |
## Links
|
| 4 |
|
| 5 |
-
- **Weights & Biases:**
|
| 6 |
- **Checkpoints (Hugging Face):** https://huggingface.co/Chechaaa09/anlp-a2-models
|
| 7 |
|
| 8 |
## Setup
|
|
@@ -30,9 +39,48 @@ uv run python -m src.part3.evaluate --num_samples 1000 --max_new_tokens 40
|
|
| 30 |
|
| 31 |
Task 1 checkpoints are written to `checkpoints_part1/best_variant_<n>.pt`,
|
| 32 |
and evaluation results to `assets/part1_results/`. Task 2 logs metrics to
|
| 33 |
-
Weights & Biases
|
|
|
|
|
|
|
| 34 |
Task 3 results are written to `task3_results.json`.
|
| 35 |
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
tags:
|
| 5 |
+
- pytorch
|
| 6 |
+
- mixture-of-experts
|
| 7 |
+
- optimizer-comparison
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
# ANLP Assignment 2
|
| 11 |
|
| 12 |
## Links
|
| 13 |
|
| 14 |
+
- **Weights & Biases:** [Task 1](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part1), [Task 2](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2), [Task 3](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part3).
|
| 15 |
- **Checkpoints (Hugging Face):** https://huggingface.co/Chechaaa09/anlp-a2-models
|
| 16 |
|
| 17 |
## Setup
|
|
|
|
| 39 |
|
| 40 |
Task 1 checkpoints are written to `checkpoints_part1/best_variant_<n>.pt`,
|
| 41 |
and evaluation results to `assets/part1_results/`. Task 2 logs metrics to
|
| 42 |
+
Weights & Biases and saves `best.pt`, `last.pt`, and `metrics.json` under
|
| 43 |
+
`checkpoints_part2/<optimizer>/<run_id>/`. The weights are PyTorch state
|
| 44 |
+
dictionaries; `metrics.json` includes the model configuration for loading them.
|
| 45 |
Task 3 results are written to `task3_results.json`.
|
| 46 |
|
| 47 |
+
Task 3 uses the original pretrained [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) checkpoint.
|
| 48 |
+
|
| 49 |
+
## Loading a Task 2 checkpoint
|
| 50 |
+
|
| 51 |
+
These are state dictionaries for the custom PyTorch architecture in this repository.
|
| 52 |
+
Clone or download the source files, install the dependencies, and run from that checkout:
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
import json
|
| 56 |
+
import torch
|
| 57 |
+
from huggingface_hub import hf_hub_download
|
| 58 |
+
from src.part2.model import DecoderOnlyTransformer
|
| 59 |
+
|
| 60 |
+
repo_id = "Chechaaa09/anlp-a2-models"
|
| 61 |
+
folder = "checkpoints_part2/adamw/oj3opt7s"
|
| 62 |
+
with open(hf_hub_download(repo_id, f"{folder}/metrics.json")) as handle:
|
| 63 |
+
metadata = json.load(handle)
|
| 64 |
+
model = DecoderOnlyTransformer(**metadata["model_config"])
|
| 65 |
+
weights = torch.load(
|
| 66 |
+
hf_hub_download(repo_id, f"{folder}/best.pt"),
|
| 67 |
+
map_location="cpu", weights_only=True,
|
| 68 |
+
)
|
| 69 |
+
model.load_state_dict(weights)
|
| 70 |
+
model.eval()
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
Use the GPT-2 tokenizer for Task 2. Each run includes `best.pt` (lowest
|
| 74 |
+
validation loss), `last.pt` (final weights), and `metrics.json` (configuration,
|
| 75 |
+
all ten evaluations, and its W&B link). Task 1 uses the saved custom tokenizer
|
| 76 |
+
in `tokenizer_part1/`.
|
| 77 |
+
|
| 78 |
+
| Optimizer | Checkpoint folder | Validation loss | Validation perplexity | Test sentence BLEU |
|
| 79 |
+
| --- | --- | ---: | ---: | ---: |
|
| 80 |
+
| adamw | `checkpoints_part2/adamw/oj3opt7s/` | 5.2531 | 191.15 | 2.9645 |
|
| 81 |
+
| cautious | `checkpoints_part2/cautious/p2yw4r11/` | 5.4965 | 243.84 | 2.5585 |
|
| 82 |
+
| lion | `checkpoints_part2/lion/ydyarkxk/` | 4.9409 | 139.90 | 2.5707 |
|
| 83 |
+
| scion | `checkpoints_part2/scion/89yiklpr/` | 5.5330 | 252.91 | 2.7522 |
|
| 84 |
+
| sophia | `checkpoints_part2/sophia/3j32zwf3/` | 5.3892 | 219.02 | 2.5534 |
|
| 85 |
+
|
| 86 |
+
Task 2 results are from the completed checkpoint-saving runs on 4 October 2026. BLEU is on the 0–100 scale.
|
checkpoints_part2/adamw/oj3opt7s/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:177d34e5a7d5820988c2e7b3d45b01e310c26d14b4a1a8fd816c7698ca90f20c
|
| 3 |
+
size 69249142
|
checkpoints_part2/adamw/oj3opt7s/last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2986940829e50d7b4a01d9b38f0cfb4078d9d4b438fab50035ed897451f2f437
|
| 3 |
+
size 69249142
|
checkpoints_part2/adamw/oj3opt7s/metrics.json
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"optimizer": "adamw",
|
| 4 |
+
"lr": 0.001,
|
| 5 |
+
"weight_decay": 0.01,
|
| 6 |
+
"batch_size": 8,
|
| 7 |
+
"seq_len": 256,
|
| 8 |
+
"d_model": 256,
|
| 9 |
+
"n_heads": 8,
|
| 10 |
+
"d_ff": 1024,
|
| 11 |
+
"n_layers": 4,
|
| 12 |
+
"dropout": 0.1,
|
| 13 |
+
"epochs": 1,
|
| 14 |
+
"save_dir": "checkpoints_part2"
|
| 15 |
+
},
|
| 16 |
+
"model_config": {
|
| 17 |
+
"vocab_size": 50257,
|
| 18 |
+
"d_model": 256,
|
| 19 |
+
"n_heads": 8,
|
| 20 |
+
"d_ff": 1024,
|
| 21 |
+
"n_layers": 4,
|
| 22 |
+
"max_seq_len": 256,
|
| 23 |
+
"dropout": 0.1
|
| 24 |
+
},
|
| 25 |
+
"tokenizer": "gpt2",
|
| 26 |
+
"wandb_run_id": "oj3opt7s",
|
| 27 |
+
"wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/oj3opt7s",
|
| 28 |
+
"evaluations": [
|
| 29 |
+
{
|
| 30 |
+
"step": 1804,
|
| 31 |
+
"tokens": 3694592,
|
| 32 |
+
"fraction": 0.10003119043493329,
|
| 33 |
+
"val_loss": 6.6341400855553605,
|
| 34 |
+
"val_perplexity": 760.6247123420002,
|
| 35 |
+
"val_bleu": 1.6350012582058588
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"step": 3607,
|
| 39 |
+
"tokens": 7387136,
|
| 40 |
+
"fraction": 0.20000693120776294,
|
| 41 |
+
"val_loss": 6.198281208198227,
|
| 42 |
+
"val_perplexity": 491.9028355150239,
|
| 43 |
+
"val_bleu": 2.268582026127263
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"step": 5411,
|
| 47 |
+
"tokens": 11081728,
|
| 48 |
+
"fraction": 0.3000381216426962,
|
| 49 |
+
"val_loss": 5.955291291672788,
|
| 50 |
+
"val_perplexity": 385.7892715145811,
|
| 51 |
+
"val_bleu": 2.5229026037852225
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"step": 7214,
|
| 55 |
+
"tokens": 14774272,
|
| 56 |
+
"fraction": 0.4000138624155259,
|
| 57 |
+
"val_loss": 5.7803394494656315,
|
| 58 |
+
"val_perplexity": 323.8691089562624,
|
| 59 |
+
"val_bleu": 2.0795491380807083
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"step": 9018,
|
| 63 |
+
"tokens": 18468864,
|
| 64 |
+
"fraction": 0.5000450528504592,
|
| 65 |
+
"val_loss": 5.637433009471246,
|
| 66 |
+
"val_perplexity": 280.7411328906744,
|
| 67 |
+
"val_bleu": 2.5986586550967212
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"step": 10821,
|
| 71 |
+
"tokens": 22161408,
|
| 72 |
+
"fraction": 0.6000207936232889,
|
| 73 |
+
"val_loss": 5.5303351569794374,
|
| 74 |
+
"val_perplexity": 252.22843297838526,
|
| 75 |
+
"val_bleu": 2.516097418373473
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"step": 12625,
|
| 79 |
+
"tokens": 25856000,
|
| 80 |
+
"fraction": 0.7000519840582221,
|
| 81 |
+
"val_loss": 5.437061781892758,
|
| 82 |
+
"val_perplexity": 229.76608780916973,
|
| 83 |
+
"val_bleu": 2.36290606336987
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"step": 14428,
|
| 87 |
+
"tokens": 29548544,
|
| 88 |
+
"fraction": 0.8000277248310518,
|
| 89 |
+
"val_loss": 5.366506714782791,
|
| 90 |
+
"val_perplexity": 214.11359988989213,
|
| 91 |
+
"val_bleu": 2.6501718493221613
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"step": 16231,
|
| 95 |
+
"tokens": 33241088,
|
| 96 |
+
"fraction": 0.9000034656038814,
|
| 97 |
+
"val_loss": 5.3192690577573645,
|
| 98 |
+
"val_perplexity": 204.234543765364,
|
| 99 |
+
"val_bleu": 2.5978208001168825
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"step": 18035,
|
| 103 |
+
"tokens": 36935680,
|
| 104 |
+
"fraction": 1.0000346560388147,
|
| 105 |
+
"val_loss": 5.253066647790387,
|
| 106 |
+
"val_perplexity": 191.15156507531572,
|
| 107 |
+
"val_bleu": 2.9645072122027476
|
| 108 |
+
}
|
| 109 |
+
],
|
| 110 |
+
"best": {
|
| 111 |
+
"step": 18035,
|
| 112 |
+
"tokens": 36935680,
|
| 113 |
+
"fraction": 1.0000346560388147,
|
| 114 |
+
"val_loss": 5.253066647790387,
|
| 115 |
+
"val_perplexity": 191.15156507531572,
|
| 116 |
+
"val_bleu": 2.9645072122027476
|
| 117 |
+
},
|
| 118 |
+
"last": {
|
| 119 |
+
"step": 18035,
|
| 120 |
+
"tokens": 36935680,
|
| 121 |
+
"fraction": 1.0000346560388147,
|
| 122 |
+
"val_loss": 5.253066647790387,
|
| 123 |
+
"val_perplexity": 191.15156507531572,
|
| 124 |
+
"val_bleu": 2.9645072122027476
|
| 125 |
+
},
|
| 126 |
+
"last_checkpoint_step": 18035,
|
| 127 |
+
"last_checkpoint_tokens": 36935680,
|
| 128 |
+
"complete": true
|
| 129 |
+
}
|
checkpoints_part2/cautious/p2yw4r11/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:09ef92900ad49774d121f8093bbbbab9deedb0f381f4e2e1cc5d84ad995201b7
|
| 3 |
+
size 69249142
|
checkpoints_part2/cautious/p2yw4r11/last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:818165692b91a95607893f71214bdd70d130cbc503a04d4fb2663735a5d5ebe9
|
| 3 |
+
size 69249142
|
checkpoints_part2/cautious/p2yw4r11/metrics.json
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"optimizer": "cautious",
|
| 4 |
+
"lr": 0.001,
|
| 5 |
+
"weight_decay": 0.01,
|
| 6 |
+
"batch_size": 8,
|
| 7 |
+
"seq_len": 256,
|
| 8 |
+
"d_model": 256,
|
| 9 |
+
"n_heads": 8,
|
| 10 |
+
"d_ff": 1024,
|
| 11 |
+
"n_layers": 4,
|
| 12 |
+
"dropout": 0.1,
|
| 13 |
+
"epochs": 1,
|
| 14 |
+
"save_dir": "checkpoints_part2"
|
| 15 |
+
},
|
| 16 |
+
"model_config": {
|
| 17 |
+
"vocab_size": 50257,
|
| 18 |
+
"d_model": 256,
|
| 19 |
+
"n_heads": 8,
|
| 20 |
+
"d_ff": 1024,
|
| 21 |
+
"n_layers": 4,
|
| 22 |
+
"max_seq_len": 256,
|
| 23 |
+
"dropout": 0.1
|
| 24 |
+
},
|
| 25 |
+
"tokenizer": "gpt2",
|
| 26 |
+
"wandb_run_id": "p2yw4r11",
|
| 27 |
+
"wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/p2yw4r11",
|
| 28 |
+
"evaluations": [
|
| 29 |
+
{
|
| 30 |
+
"step": 1804,
|
| 31 |
+
"tokens": 3694592,
|
| 32 |
+
"fraction": 0.10003119043493329,
|
| 33 |
+
"val_loss": 6.8086098829905195,
|
| 34 |
+
"val_perplexity": 905.6110262375989,
|
| 35 |
+
"val_bleu": 1.3151229296275404
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"step": 3607,
|
| 39 |
+
"tokens": 7387136,
|
| 40 |
+
"fraction": 0.20000693120776294,
|
| 41 |
+
"val_loss": 6.494788529630193,
|
| 42 |
+
"val_perplexity": 661.6842838933193,
|
| 43 |
+
"val_bleu": 1.7576461147222004
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"step": 5411,
|
| 47 |
+
"tokens": 11081728,
|
| 48 |
+
"fraction": 0.3000381216426962,
|
| 49 |
+
"val_loss": 6.3169474725475805,
|
| 50 |
+
"val_perplexity": 553.8796764018344,
|
| 51 |
+
"val_bleu": 1.8936975883421343
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"step": 7214,
|
| 55 |
+
"tokens": 14774272,
|
| 56 |
+
"fraction": 0.4000138624155259,
|
| 57 |
+
"val_loss": 6.130474798217743,
|
| 58 |
+
"val_perplexity": 459.6543519445046,
|
| 59 |
+
"val_bleu": 2.3554491109335634
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"step": 9018,
|
| 63 |
+
"tokens": 18468864,
|
| 64 |
+
"fraction": 0.5000450528504592,
|
| 65 |
+
"val_loss": 5.957174865547531,
|
| 66 |
+
"val_perplexity": 386.51661889878403,
|
| 67 |
+
"val_bleu": 2.703317966629297
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"step": 10821,
|
| 71 |
+
"tokens": 22161408,
|
| 72 |
+
"fraction": 0.6000207936232889,
|
| 73 |
+
"val_loss": 5.821500727278505,
|
| 74 |
+
"val_perplexity": 337.4781364342,
|
| 75 |
+
"val_bleu": 2.5474224881531287
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"step": 12625,
|
| 79 |
+
"tokens": 25856000,
|
| 80 |
+
"fraction": 0.7000519840582221,
|
| 81 |
+
"val_loss": 5.718654285647912,
|
| 82 |
+
"val_perplexity": 304.4948839856075,
|
| 83 |
+
"val_bleu": 2.507245238116674
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"step": 14428,
|
| 87 |
+
"tokens": 29548544,
|
| 88 |
+
"fraction": 0.8000277248310518,
|
| 89 |
+
"val_loss": 5.632188036532221,
|
| 90 |
+
"val_perplexity": 279.27250806132463,
|
| 91 |
+
"val_bleu": 2.6585916717692606
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"step": 16231,
|
| 95 |
+
"tokens": 33241088,
|
| 96 |
+
"fraction": 0.9000034656038814,
|
| 97 |
+
"val_loss": 5.5536918744831505,
|
| 98 |
+
"val_perplexity": 258.1889998997565,
|
| 99 |
+
"val_bleu": 2.400037540916668
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"step": 18035,
|
| 103 |
+
"tokens": 36935680,
|
| 104 |
+
"fraction": 1.0000346560388147,
|
| 105 |
+
"val_loss": 5.496509715944469,
|
| 106 |
+
"val_perplexity": 243.83937661071516,
|
| 107 |
+
"val_bleu": 2.5584566218227955
|
| 108 |
+
}
|
| 109 |
+
],
|
| 110 |
+
"best": {
|
| 111 |
+
"step": 18035,
|
| 112 |
+
"tokens": 36935680,
|
| 113 |
+
"fraction": 1.0000346560388147,
|
| 114 |
+
"val_loss": 5.496509715944469,
|
| 115 |
+
"val_perplexity": 243.83937661071516,
|
| 116 |
+
"val_bleu": 2.5584566218227955
|
| 117 |
+
},
|
| 118 |
+
"last": {
|
| 119 |
+
"step": 18035,
|
| 120 |
+
"tokens": 36935680,
|
| 121 |
+
"fraction": 1.0000346560388147,
|
| 122 |
+
"val_loss": 5.496509715944469,
|
| 123 |
+
"val_perplexity": 243.83937661071516,
|
| 124 |
+
"val_bleu": 2.5584566218227955
|
| 125 |
+
},
|
| 126 |
+
"last_checkpoint_step": 18035,
|
| 127 |
+
"last_checkpoint_tokens": 36935680,
|
| 128 |
+
"complete": true
|
| 129 |
+
}
|
checkpoints_part2/lion/ydyarkxk/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:408e2993d97778e30db06cfbd6c6b8ecefd053c426e03d9f517dd6a9b65ea04b
|
| 3 |
+
size 69249142
|
checkpoints_part2/lion/ydyarkxk/last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cd377b0467cdf5ab092d0200104c920f75e8653908cf626e7464a9e5e6e34a5b
|
| 3 |
+
size 69249142
|
checkpoints_part2/lion/ydyarkxk/metrics.json
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"optimizer": "lion",
|
| 4 |
+
"lr": 0.0002,
|
| 5 |
+
"weight_decay": 0.1,
|
| 6 |
+
"batch_size": 8,
|
| 7 |
+
"seq_len": 256,
|
| 8 |
+
"d_model": 256,
|
| 9 |
+
"n_heads": 8,
|
| 10 |
+
"d_ff": 1024,
|
| 11 |
+
"n_layers": 4,
|
| 12 |
+
"dropout": 0.1,
|
| 13 |
+
"epochs": 1,
|
| 14 |
+
"save_dir": "checkpoints_part2"
|
| 15 |
+
},
|
| 16 |
+
"model_config": {
|
| 17 |
+
"vocab_size": 50257,
|
| 18 |
+
"d_model": 256,
|
| 19 |
+
"n_heads": 8,
|
| 20 |
+
"d_ff": 1024,
|
| 21 |
+
"n_layers": 4,
|
| 22 |
+
"max_seq_len": 256,
|
| 23 |
+
"dropout": 0.1
|
| 24 |
+
},
|
| 25 |
+
"tokenizer": "gpt2",
|
| 26 |
+
"wandb_run_id": "ydyarkxk",
|
| 27 |
+
"wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/ydyarkxk",
|
| 28 |
+
"evaluations": [
|
| 29 |
+
{
|
| 30 |
+
"step": 1804,
|
| 31 |
+
"tokens": 3694592,
|
| 32 |
+
"fraction": 0.10003119043493329,
|
| 33 |
+
"val_loss": 7.238790913256342,
|
| 34 |
+
"val_perplexity": 1392.4094089228465,
|
| 35 |
+
"val_bleu": 1.1876307944768998
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"step": 3607,
|
| 39 |
+
"tokens": 7387136,
|
| 40 |
+
"fraction": 0.20000693120776294,
|
| 41 |
+
"val_loss": 6.508020398621549,
|
| 42 |
+
"val_perplexity": 670.4977846044866,
|
| 43 |
+
"val_bleu": 1.951173313307093
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"step": 5411,
|
| 47 |
+
"tokens": 11081728,
|
| 48 |
+
"fraction": 0.3000381216426962,
|
| 49 |
+
"val_loss": 5.978941541469977,
|
| 50 |
+
"val_perplexity": 395.0220323591368,
|
| 51 |
+
"val_bleu": 2.3198521517571797
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"step": 7214,
|
| 55 |
+
"tokens": 14774272,
|
| 56 |
+
"fraction": 0.4000138624155259,
|
| 57 |
+
"val_loss": 5.678261665051093,
|
| 58 |
+
"val_perplexity": 292.44062805241543,
|
| 59 |
+
"val_bleu": 2.431544539213501
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"step": 9018,
|
| 63 |
+
"tokens": 18468864,
|
| 64 |
+
"fraction": 0.5000450528504592,
|
| 65 |
+
"val_loss": 5.472018926443454,
|
| 66 |
+
"val_perplexity": 237.94009170092153,
|
| 67 |
+
"val_bleu": 2.5023165366302536
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"step": 10821,
|
| 71 |
+
"tokens": 22161408,
|
| 72 |
+
"fraction": 0.6000207936232889,
|
| 73 |
+
"val_loss": 5.327088786218456,
|
| 74 |
+
"val_perplexity": 205.8378630306369,
|
| 75 |
+
"val_bleu": 2.6355905792527023
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"step": 12625,
|
| 79 |
+
"tokens": 25856000,
|
| 80 |
+
"fraction": 0.7000519840582221,
|
| 81 |
+
"val_loss": 5.207654690790081,
|
| 82 |
+
"val_perplexity": 182.66514917149527,
|
| 83 |
+
"val_bleu": 2.438599562300715
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"step": 14428,
|
| 87 |
+
"tokens": 29548544,
|
| 88 |
+
"fraction": 0.8000277248310518,
|
| 89 |
+
"val_loss": 5.110067813458319,
|
| 90 |
+
"val_perplexity": 165.6815899343745,
|
| 91 |
+
"val_bleu": 2.1012652918887254
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"step": 16231,
|
| 95 |
+
"tokens": 33241088,
|
| 96 |
+
"fraction": 0.9000034656038814,
|
| 97 |
+
"val_loss": 5.023584961653231,
|
| 98 |
+
"val_perplexity": 151.9550816410011,
|
| 99 |
+
"val_bleu": 2.422311511198055
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"step": 18035,
|
| 103 |
+
"tokens": 36935680,
|
| 104 |
+
"fraction": 1.0000346560388147,
|
| 105 |
+
"val_loss": 4.940937335143784,
|
| 106 |
+
"val_perplexity": 139.90132254703283,
|
| 107 |
+
"val_bleu": 2.5706617588457146
|
| 108 |
+
}
|
| 109 |
+
],
|
| 110 |
+
"best": {
|
| 111 |
+
"step": 18035,
|
| 112 |
+
"tokens": 36935680,
|
| 113 |
+
"fraction": 1.0000346560388147,
|
| 114 |
+
"val_loss": 4.940937335143784,
|
| 115 |
+
"val_perplexity": 139.90132254703283,
|
| 116 |
+
"val_bleu": 2.5706617588457146
|
| 117 |
+
},
|
| 118 |
+
"last": {
|
| 119 |
+
"step": 18035,
|
| 120 |
+
"tokens": 36935680,
|
| 121 |
+
"fraction": 1.0000346560388147,
|
| 122 |
+
"val_loss": 4.940937335143784,
|
| 123 |
+
"val_perplexity": 139.90132254703283,
|
| 124 |
+
"val_bleu": 2.5706617588457146
|
| 125 |
+
},
|
| 126 |
+
"last_checkpoint_step": 18035,
|
| 127 |
+
"last_checkpoint_tokens": 36935680,
|
| 128 |
+
"complete": true
|
| 129 |
+
}
|
checkpoints_part2/scion/89yiklpr/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ee3ec5cc364107236664985d9bdc258e1627df169c1be718f46b9a2cf463fd1b
|
| 3 |
+
size 69249142
|
checkpoints_part2/scion/89yiklpr/last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3e2e19869d4ab0f853d61a56982410d80578e2ff17abb74527c96116de1bfe86
|
| 3 |
+
size 69249142
|
checkpoints_part2/scion/89yiklpr/metrics.json
ADDED
|
@@ -0,0 +1,129 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"optimizer": "scion",
|
| 4 |
+
"lr": 0.001,
|
| 5 |
+
"weight_decay": 0.01,
|
| 6 |
+
"batch_size": 8,
|
| 7 |
+
"seq_len": 256,
|
| 8 |
+
"d_model": 256,
|
| 9 |
+
"n_heads": 8,
|
| 10 |
+
"d_ff": 1024,
|
| 11 |
+
"n_layers": 4,
|
| 12 |
+
"dropout": 0.1,
|
| 13 |
+
"epochs": 1,
|
| 14 |
+
"save_dir": "checkpoints_part2"
|
| 15 |
+
},
|
| 16 |
+
"model_config": {
|
| 17 |
+
"vocab_size": 50257,
|
| 18 |
+
"d_model": 256,
|
| 19 |
+
"n_heads": 8,
|
| 20 |
+
"d_ff": 1024,
|
| 21 |
+
"n_layers": 4,
|
| 22 |
+
"max_seq_len": 256,
|
| 23 |
+
"dropout": 0.1
|
| 24 |
+
},
|
| 25 |
+
"tokenizer": "gpt2",
|
| 26 |
+
"wandb_run_id": "89yiklpr",
|
| 27 |
+
"wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/89yiklpr",
|
| 28 |
+
"evaluations": [
|
| 29 |
+
{
|
| 30 |
+
"step": 1804,
|
| 31 |
+
"tokens": 3694592,
|
| 32 |
+
"fraction": 0.10003119043493329,
|
| 33 |
+
"val_loss": 8.281115139791828,
|
| 34 |
+
"val_perplexity": 3948.5951633695317,
|
| 35 |
+
"val_bleu": 1.1122944171065337
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"step": 3607,
|
| 39 |
+
"tokens": 7387136,
|
| 40 |
+
"fraction": 0.20000693120776294,
|
| 41 |
+
"val_loss": 7.1758178436827516,
|
| 42 |
+
"val_perplexity": 1307.4289363561054,
|
| 43 |
+
"val_bleu": 2.5826001619044807
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"step": 5411,
|
| 47 |
+
"tokens": 11081728,
|
| 48 |
+
"fraction": 0.3000381216426962,
|
| 49 |
+
"val_loss": 6.6727667216531295,
|
| 50 |
+
"val_perplexity": 790.5798959226208,
|
| 51 |
+
"val_bleu": 2.5622937725329265
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"step": 7214,
|
| 55 |
+
"tokens": 14774272,
|
| 56 |
+
"fraction": 0.4000138624155259,
|
| 57 |
+
"val_loss": 6.378503823232746,
|
| 58 |
+
"val_perplexity": 589.0457314885668,
|
| 59 |
+
"val_bleu": 2.526161505466013
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"step": 9018,
|
| 63 |
+
"tokens": 18468864,
|
| 64 |
+
"fraction": 0.5000450528504592,
|
| 65 |
+
"val_loss": 6.168975746322297,
|
| 66 |
+
"val_perplexity": 477.69657295732503,
|
| 67 |
+
"val_bleu": 2.597034079802741
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"step": 10821,
|
| 71 |
+
"tokens": 22161408,
|
| 72 |
+
"fraction": 0.6000207936232889,
|
| 73 |
+
"val_loss": 5.992401353375403,
|
| 74 |
+
"val_perplexity": 400.3748980829875,
|
| 75 |
+
"val_bleu": 2.5429924513715405
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"step": 12625,
|
| 79 |
+
"tokens": 25856000,
|
| 80 |
+
"fraction": 0.7000519840582221,
|
| 81 |
+
"val_loss": 5.851240207335192,
|
| 82 |
+
"val_perplexity": 347.6652902579374,
|
| 83 |
+
"val_bleu": 2.4854103863453743
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"step": 14428,
|
| 87 |
+
"tokens": 29548544,
|
| 88 |
+
"fraction": 0.8000277248310518,
|
| 89 |
+
"val_loss": 5.733623129165101,
|
| 90 |
+
"val_perplexity": 309.0871046253544,
|
| 91 |
+
"val_bleu": 2.5172439325356755
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"step": 16231,
|
| 95 |
+
"tokens": 33241088,
|
| 96 |
+
"fraction": 0.9000034656038814,
|
| 97 |
+
"val_loss": 5.631224749806874,
|
| 98 |
+
"val_perplexity": 279.00361809141646,
|
| 99 |
+
"val_bleu": 2.8175740515027603
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"step": 18035,
|
| 103 |
+
"tokens": 36935680,
|
| 104 |
+
"fraction": 1.0000346560388147,
|
| 105 |
+
"val_loss": 5.533019167220521,
|
| 106 |
+
"val_perplexity": 252.9063260045474,
|
| 107 |
+
"val_bleu": 2.7522405185633163
|
| 108 |
+
}
|
| 109 |
+
],
|
| 110 |
+
"best": {
|
| 111 |
+
"step": 18035,
|
| 112 |
+
"tokens": 36935680,
|
| 113 |
+
"fraction": 1.0000346560388147,
|
| 114 |
+
"val_loss": 5.533019167220521,
|
| 115 |
+
"val_perplexity": 252.9063260045474,
|
| 116 |
+
"val_bleu": 2.7522405185633163
|
| 117 |
+
},
|
| 118 |
+
"last": {
|
| 119 |
+
"step": 18035,
|
| 120 |
+
"tokens": 36935680,
|
| 121 |
+
"fraction": 1.0000346560388147,
|
| 122 |
+
"val_loss": 5.533019167220521,
|
| 123 |
+
"val_perplexity": 252.9063260045474,
|
| 124 |
+
"val_bleu": 2.7522405185633163
|
| 125 |
+
},
|
| 126 |
+
"last_checkpoint_step": 18035,
|
| 127 |
+
"last_checkpoint_tokens": 36935680,
|
| 128 |
+
"complete": true
|
| 129 |
+
}
|
checkpoints_part2/sophia/3j32zwf3/best.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bf87bebcd359e661b0403c30ff57a06bdbbd42ed3fdfce7e84de7ce22d3892de
|
| 3 |
+
size 69249142
|
checkpoints_part2/sophia/3j32zwf3/last.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4ca778cca77a47b8ae02e0d3bf2303fb031f46ba0ced12b3df7296df69642388
|
| 3 |
+
size 69249142
|
checkpoints_part2/sophia/3j32zwf3/metrics.json
ADDED
|
@@ -0,0 +1,130 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"config": {
|
| 3 |
+
"optimizer": "sophia",
|
| 4 |
+
"lr": 0.0001,
|
| 5 |
+
"rho": 0.04,
|
| 6 |
+
"weight_decay": 0.1,
|
| 7 |
+
"batch_size": 8,
|
| 8 |
+
"seq_len": 256,
|
| 9 |
+
"d_model": 256,
|
| 10 |
+
"n_heads": 8,
|
| 11 |
+
"d_ff": 1024,
|
| 12 |
+
"n_layers": 4,
|
| 13 |
+
"dropout": 0.1,
|
| 14 |
+
"epochs": 1,
|
| 15 |
+
"save_dir": "checkpoints_part2"
|
| 16 |
+
},
|
| 17 |
+
"model_config": {
|
| 18 |
+
"vocab_size": 50257,
|
| 19 |
+
"d_model": 256,
|
| 20 |
+
"n_heads": 8,
|
| 21 |
+
"d_ff": 1024,
|
| 22 |
+
"n_layers": 4,
|
| 23 |
+
"max_seq_len": 256,
|
| 24 |
+
"dropout": 0.1
|
| 25 |
+
},
|
| 26 |
+
"tokenizer": "gpt2",
|
| 27 |
+
"wandb_run_id": "3j32zwf3",
|
| 28 |
+
"wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/3j32zwf3",
|
| 29 |
+
"evaluations": [
|
| 30 |
+
{
|
| 31 |
+
"step": 1804,
|
| 32 |
+
"tokens": 3694592,
|
| 33 |
+
"fraction": 0.10003119043493329,
|
| 34 |
+
"val_loss": 7.448652947972159,
|
| 35 |
+
"val_perplexity": 1717.5479599260464,
|
| 36 |
+
"val_bleu": 1.515281006968686
|
| 37 |
+
},
|
| 38 |
+
{
|
| 39 |
+
"step": 3607,
|
| 40 |
+
"tokens": 7387136,
|
| 41 |
+
"fraction": 0.20000693120776294,
|
| 42 |
+
"val_loss": 6.992933257611212,
|
| 43 |
+
"val_perplexity": 1088.910852313718,
|
| 44 |
+
"val_bleu": 1.3970472611186042
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"step": 5411,
|
| 48 |
+
"tokens": 11081728,
|
| 49 |
+
"fraction": 0.3000381216426962,
|
| 50 |
+
"val_loss": 6.619747597300364,
|
| 51 |
+
"val_perplexity": 749.7558328312051,
|
| 52 |
+
"val_bleu": 2.2005620954722893
|
| 53 |
+
},
|
| 54 |
+
{
|
| 55 |
+
"step": 7214,
|
| 56 |
+
"tokens": 14774272,
|
| 57 |
+
"fraction": 0.4000138624155259,
|
| 58 |
+
"val_loss": 6.32087761913231,
|
| 59 |
+
"val_perplexity": 556.0607879569982,
|
| 60 |
+
"val_bleu": 2.5882135890697797
|
| 61 |
+
},
|
| 62 |
+
{
|
| 63 |
+
"step": 9018,
|
| 64 |
+
"tokens": 18468864,
|
| 65 |
+
"fraction": 0.5000450528504592,
|
| 66 |
+
"val_loss": 6.0633629972111445,
|
| 67 |
+
"val_perplexity": 429.8184873861714,
|
| 68 |
+
"val_bleu": 2.042761820526234
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"step": 10821,
|
| 72 |
+
"tokens": 22161408,
|
| 73 |
+
"fraction": 0.6000207936232889,
|
| 74 |
+
"val_loss": 5.861090107116395,
|
| 75 |
+
"val_perplexity": 351.1066793695729,
|
| 76 |
+
"val_bleu": 2.2881800394766634
|
| 77 |
+
},
|
| 78 |
+
{
|
| 79 |
+
"step": 12625,
|
| 80 |
+
"tokens": 25856000,
|
| 81 |
+
"fraction": 0.7000519840582221,
|
| 82 |
+
"val_loss": 5.710041520124424,
|
| 83 |
+
"val_perplexity": 301.8836022673098,
|
| 84 |
+
"val_bleu": 2.4996404904784804
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"step": 14428,
|
| 88 |
+
"tokens": 29548544,
|
| 89 |
+
"fraction": 0.8000277248310518,
|
| 90 |
+
"val_loss": 5.5844624865792705,
|
| 91 |
+
"val_perplexity": 266.2571274895874,
|
| 92 |
+
"val_bleu": 2.5117622082097006
|
| 93 |
+
},
|
| 94 |
+
{
|
| 95 |
+
"step": 16231,
|
| 96 |
+
"tokens": 33241088,
|
| 97 |
+
"fraction": 0.9000034656038814,
|
| 98 |
+
"val_loss": 5.479142612087989,
|
| 99 |
+
"val_perplexity": 239.64115383876725,
|
| 100 |
+
"val_bleu": 2.522367710436995
|
| 101 |
+
},
|
| 102 |
+
{
|
| 103 |
+
"step": 18035,
|
| 104 |
+
"tokens": 36935680,
|
| 105 |
+
"fraction": 1.0000346560388147,
|
| 106 |
+
"val_loss": 5.389180385899877,
|
| 107 |
+
"val_perplexity": 219.02379697507902,
|
| 108 |
+
"val_bleu": 2.55340545320376
|
| 109 |
+
}
|
| 110 |
+
],
|
| 111 |
+
"best": {
|
| 112 |
+
"step": 18035,
|
| 113 |
+
"tokens": 36935680,
|
| 114 |
+
"fraction": 1.0000346560388147,
|
| 115 |
+
"val_loss": 5.389180385899877,
|
| 116 |
+
"val_perplexity": 219.02379697507902,
|
| 117 |
+
"val_bleu": 2.55340545320376
|
| 118 |
+
},
|
| 119 |
+
"last": {
|
| 120 |
+
"step": 18035,
|
| 121 |
+
"tokens": 36935680,
|
| 122 |
+
"fraction": 1.0000346560388147,
|
| 123 |
+
"val_loss": 5.389180385899877,
|
| 124 |
+
"val_perplexity": 219.02379697507902,
|
| 125 |
+
"val_bleu": 2.55340545320376
|
| 126 |
+
},
|
| 127 |
+
"last_checkpoint_step": 18035,
|
| 128 |
+
"last_checkpoint_tokens": 36935680,
|
| 129 |
+
"complete": true
|
| 130 |
+
}
|
src/part2/train.py
CHANGED
|
@@ -8,6 +8,8 @@ from transformers import AutoTokenizer
|
|
| 8 |
import sacrebleu
|
| 9 |
import wandb
|
| 10 |
import argparse
|
|
|
|
|
|
|
| 11 |
import yaml
|
| 12 |
from tqdm import tqdm
|
| 13 |
from .model import DecoderOnlyTransformer
|
|
@@ -86,7 +88,7 @@ def evaluate(model, val_loader, criterion, device = "cuda"):
|
|
| 86 |
|
| 87 |
def train(args):
|
| 88 |
|
| 89 |
-
wandb.init(
|
| 90 |
project = "ANLP-Assignment-2-Part2",
|
| 91 |
name = f"{args.optimizer}_lr{args.lr}_wd{args.weight_decay}",
|
| 92 |
config = vars(args),
|
|
@@ -118,6 +120,27 @@ def train(args):
|
|
| 118 |
dropout = args.dropout,
|
| 119 |
).to(device)
|
| 120 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
if args.optimizer == "adamw":
|
| 122 |
optimizer = CustomAdamW(model.parameters(), lr = args.lr, weight_decay = args.weight_decay)
|
| 123 |
elif args.optimizer == "cautious":
|
|
@@ -193,6 +216,23 @@ def train(args):
|
|
| 193 |
"fraction": fraction,
|
| 194 |
})
|
| 195 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 196 |
next_eval_token += eval_token_interval
|
| 197 |
model.train()
|
| 198 |
|
|
@@ -201,14 +241,27 @@ def train(args):
|
|
| 201 |
if total_tokens_processed >= total_tokens_budget:
|
| 202 |
break
|
| 203 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 204 |
wandb.finish()
|
| 205 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 206 |
def parse_args():
|
| 207 |
parser = argparse.ArgumentParser(description="Train transformer with specified optimizer config(s)")
|
| 208 |
parser.add_argument("--config", type=str, default=None, help="Path to YAML config file")
|
| 209 |
-
parser.add_argument("--all", action="store_true", help="Train all
|
| 210 |
parser.add_argument("--lr", type=float, default=None)
|
| 211 |
parser.add_argument("--batch_size", type=int, default=None)
|
|
|
|
| 212 |
cli_args = parser.parse_args()
|
| 213 |
if not cli_args.config and not cli_args.all:
|
| 214 |
parser.error("You must specify either --config <path> or --all")
|
|
@@ -221,6 +274,7 @@ def build_config(config_path, cli_args):
|
|
| 221 |
config["lr"] = cli_args.lr
|
| 222 |
if cli_args.batch_size is not None:
|
| 223 |
config["batch_size"] = cli_args.batch_size
|
|
|
|
| 224 |
return argparse.Namespace(**config)
|
| 225 |
|
| 226 |
if __name__ == "__main__":
|
|
@@ -244,4 +298,4 @@ if __name__ == "__main__":
|
|
| 244 |
torch.cuda.empty_cache()
|
| 245 |
else:
|
| 246 |
args = build_config(cli_args.config, cli_args)
|
| 247 |
-
train(args)
|
|
|
|
| 8 |
import sacrebleu
|
| 9 |
import wandb
|
| 10 |
import argparse
|
| 11 |
+
import json
|
| 12 |
+
from pathlib import Path
|
| 13 |
import yaml
|
| 14 |
from tqdm import tqdm
|
| 15 |
from .model import DecoderOnlyTransformer
|
|
|
|
| 88 |
|
| 89 |
def train(args):
|
| 90 |
|
| 91 |
+
run = wandb.init(
|
| 92 |
project = "ANLP-Assignment-2-Part2",
|
| 93 |
name = f"{args.optimizer}_lr{args.lr}_wd{args.weight_decay}",
|
| 94 |
config = vars(args),
|
|
|
|
| 120 |
dropout = args.dropout,
|
| 121 |
).to(device)
|
| 122 |
|
| 123 |
+
checkpoint_dir = Path(args.save_dir) / args.optimizer / run.id
|
| 124 |
+
checkpoint_dir.mkdir(parents = True, exist_ok = False)
|
| 125 |
+
metadata = {
|
| 126 |
+
"config": vars(args),
|
| 127 |
+
"model_config": {
|
| 128 |
+
"vocab_size": tokenizer.vocab_size,
|
| 129 |
+
"d_model": args.d_model,
|
| 130 |
+
"n_heads": args.n_heads,
|
| 131 |
+
"d_ff": args.d_ff,
|
| 132 |
+
"n_layers": args.n_layers,
|
| 133 |
+
"max_seq_len": args.seq_len,
|
| 134 |
+
"dropout": args.dropout,
|
| 135 |
+
},
|
| 136 |
+
"tokenizer": "gpt2",
|
| 137 |
+
"wandb_run_id": run.id,
|
| 138 |
+
"wandb_url": run.url,
|
| 139 |
+
"evaluations": [],
|
| 140 |
+
}
|
| 141 |
+
best_val_loss = float("inf")
|
| 142 |
+
print(f"Checkpoints: {checkpoint_dir}")
|
| 143 |
+
|
| 144 |
if args.optimizer == "adamw":
|
| 145 |
optimizer = CustomAdamW(model.parameters(), lr = args.lr, weight_decay = args.weight_decay)
|
| 146 |
elif args.optimizer == "cautious":
|
|
|
|
| 216 |
"fraction": fraction,
|
| 217 |
})
|
| 218 |
|
| 219 |
+
metrics = {
|
| 220 |
+
"step": step_count,
|
| 221 |
+
"tokens": total_tokens_processed,
|
| 222 |
+
"fraction": fraction,
|
| 223 |
+
"val_loss": val_loss,
|
| 224 |
+
"val_perplexity": val_ppl,
|
| 225 |
+
"val_bleu": bleu,
|
| 226 |
+
}
|
| 227 |
+
metadata["evaluations"].append(metrics)
|
| 228 |
+
if val_loss < best_val_loss:
|
| 229 |
+
best_val_loss = val_loss
|
| 230 |
+
save_weights(model, checkpoint_dir / "best.pt")
|
| 231 |
+
metadata["best"] = metrics
|
| 232 |
+
save_weights(model, checkpoint_dir / "last.pt")
|
| 233 |
+
metadata["last"] = metrics
|
| 234 |
+
(checkpoint_dir / "metrics.json").write_text(json.dumps(metadata, indent = 2) + "\n")
|
| 235 |
+
|
| 236 |
next_eval_token += eval_token_interval
|
| 237 |
model.train()
|
| 238 |
|
|
|
|
| 241 |
if total_tokens_processed >= total_tokens_budget:
|
| 242 |
break
|
| 243 |
|
| 244 |
+
save_weights(model, checkpoint_dir / "last.pt")
|
| 245 |
+
metadata["last_checkpoint_step"] = step_count
|
| 246 |
+
metadata["last_checkpoint_tokens"] = total_tokens_processed
|
| 247 |
+
metadata["complete"] = True
|
| 248 |
+
(checkpoint_dir / "metrics.json").write_text(json.dumps(metadata, indent = 2) + "\n")
|
| 249 |
+
print(f"Saved weights and metrics to {checkpoint_dir}")
|
| 250 |
wandb.finish()
|
| 251 |
|
| 252 |
+
def save_weights(model, path):
|
| 253 |
+
# Replace only after the entire state dict has been written.
|
| 254 |
+
temporary_path = path.with_suffix(".pt.tmp")
|
| 255 |
+
torch.save(model.state_dict(), temporary_path)
|
| 256 |
+
temporary_path.replace(path)
|
| 257 |
+
|
| 258 |
def parse_args():
|
| 259 |
parser = argparse.ArgumentParser(description="Train transformer with specified optimizer config(s)")
|
| 260 |
parser.add_argument("--config", type=str, default=None, help="Path to YAML config file")
|
| 261 |
+
parser.add_argument("--all", action="store_true", help="Train all 5 optimizer configs sequentially")
|
| 262 |
parser.add_argument("--lr", type=float, default=None)
|
| 263 |
parser.add_argument("--batch_size", type=int, default=None)
|
| 264 |
+
parser.add_argument("--save_dir", type=str, default="checkpoints_part2", help="Root directory for per-run weights and metrics")
|
| 265 |
cli_args = parser.parse_args()
|
| 266 |
if not cli_args.config and not cli_args.all:
|
| 267 |
parser.error("You must specify either --config <path> or --all")
|
|
|
|
| 274 |
config["lr"] = cli_args.lr
|
| 275 |
if cli_args.batch_size is not None:
|
| 276 |
config["batch_size"] = cli_args.batch_size
|
| 277 |
+
config["save_dir"] = cli_args.save_dir
|
| 278 |
return argparse.Namespace(**config)
|
| 279 |
|
| 280 |
if __name__ == "__main__":
|
|
|
|
| 298 |
torch.cuda.empty_cache()
|
| 299 |
else:
|
| 300 |
args = build_config(cli_args.config, cli_args)
|
| 301 |
+
train(args)
|