Chechaaa09 commited on
Commit
6c5b105
·
verified ·
1 Parent(s): ad63418

Add completed Task 2 checkpoints and evaluation metrics

Browse files

Five optimizer reruns, best/final model weights, full evaluation records, current training source, and loading instructions.

README.md CHANGED
@@ -1,8 +1,17 @@
 
 
 
 
 
 
 
 
 
1
  # ANLP Assignment 2
2
 
3
  ## Links
4
 
5
- - **Weights & Biases:** To be added.
6
  - **Checkpoints (Hugging Face):** https://huggingface.co/Chechaaa09/anlp-a2-models
7
 
8
  ## Setup
@@ -30,9 +39,48 @@ uv run python -m src.part3.evaluate --num_samples 1000 --max_new_tokens 40
30
 
31
  Task 1 checkpoints are written to `checkpoints_part1/best_variant_<n>.pt`,
32
  and evaluation results to `assets/part1_results/`. Task 2 logs metrics to
33
- Weights & Biases; its current training script does not save checkpoints.
 
 
34
  Task 3 results are written to `task3_results.json`.
35
 
36
- The Task 1 checkpoints are custom PyTorch state dictionaries. Load them using
37
- `src.part1.model.DecoderOnlyTransformer` and the matching configuration in
38
- `configs/part1_var<n>.yaml`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - pytorch
6
+ - mixture-of-experts
7
+ - optimizer-comparison
8
+ ---
9
+
10
  # ANLP Assignment 2
11
 
12
  ## Links
13
 
14
+ - **Weights & Biases:** [Task 1](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part1), [Task 2](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2), [Task 3](https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part3).
15
  - **Checkpoints (Hugging Face):** https://huggingface.co/Chechaaa09/anlp-a2-models
16
 
17
  ## Setup
 
39
 
40
  Task 1 checkpoints are written to `checkpoints_part1/best_variant_<n>.pt`,
41
  and evaluation results to `assets/part1_results/`. Task 2 logs metrics to
42
+ Weights & Biases and saves `best.pt`, `last.pt`, and `metrics.json` under
43
+ `checkpoints_part2/<optimizer>/<run_id>/`. The weights are PyTorch state
44
+ dictionaries; `metrics.json` includes the model configuration for loading them.
45
  Task 3 results are written to `task3_results.json`.
46
 
47
+ Task 3 uses the original pretrained [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m) checkpoint.
48
+
49
+ ## Loading a Task 2 checkpoint
50
+
51
+ These are state dictionaries for the custom PyTorch architecture in this repository.
52
+ Clone or download the source files, install the dependencies, and run from that checkout:
53
+
54
+ ```python
55
+ import json
56
+ import torch
57
+ from huggingface_hub import hf_hub_download
58
+ from src.part2.model import DecoderOnlyTransformer
59
+
60
+ repo_id = "Chechaaa09/anlp-a2-models"
61
+ folder = "checkpoints_part2/adamw/oj3opt7s"
62
+ with open(hf_hub_download(repo_id, f"{folder}/metrics.json")) as handle:
63
+ metadata = json.load(handle)
64
+ model = DecoderOnlyTransformer(**metadata["model_config"])
65
+ weights = torch.load(
66
+ hf_hub_download(repo_id, f"{folder}/best.pt"),
67
+ map_location="cpu", weights_only=True,
68
+ )
69
+ model.load_state_dict(weights)
70
+ model.eval()
71
+ ```
72
+
73
+ Use the GPT-2 tokenizer for Task 2. Each run includes `best.pt` (lowest
74
+ validation loss), `last.pt` (final weights), and `metrics.json` (configuration,
75
+ all ten evaluations, and its W&B link). Task 1 uses the saved custom tokenizer
76
+ in `tokenizer_part1/`.
77
+
78
+ | Optimizer | Checkpoint folder | Validation loss | Validation perplexity | Test sentence BLEU |
79
+ | --- | --- | ---: | ---: | ---: |
80
+ | adamw | `checkpoints_part2/adamw/oj3opt7s/` | 5.2531 | 191.15 | 2.9645 |
81
+ | cautious | `checkpoints_part2/cautious/p2yw4r11/` | 5.4965 | 243.84 | 2.5585 |
82
+ | lion | `checkpoints_part2/lion/ydyarkxk/` | 4.9409 | 139.90 | 2.5707 |
83
+ | scion | `checkpoints_part2/scion/89yiklpr/` | 5.5330 | 252.91 | 2.7522 |
84
+ | sophia | `checkpoints_part2/sophia/3j32zwf3/` | 5.3892 | 219.02 | 2.5534 |
85
+
86
+ Task 2 results are from the completed checkpoint-saving runs on 4 October 2026. BLEU is on the 0–100 scale.
checkpoints_part2/adamw/oj3opt7s/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:177d34e5a7d5820988c2e7b3d45b01e310c26d14b4a1a8fd816c7698ca90f20c
3
+ size 69249142
checkpoints_part2/adamw/oj3opt7s/last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2986940829e50d7b4a01d9b38f0cfb4078d9d4b438fab50035ed897451f2f437
3
+ size 69249142
checkpoints_part2/adamw/oj3opt7s/metrics.json ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "optimizer": "adamw",
4
+ "lr": 0.001,
5
+ "weight_decay": 0.01,
6
+ "batch_size": 8,
7
+ "seq_len": 256,
8
+ "d_model": 256,
9
+ "n_heads": 8,
10
+ "d_ff": 1024,
11
+ "n_layers": 4,
12
+ "dropout": 0.1,
13
+ "epochs": 1,
14
+ "save_dir": "checkpoints_part2"
15
+ },
16
+ "model_config": {
17
+ "vocab_size": 50257,
18
+ "d_model": 256,
19
+ "n_heads": 8,
20
+ "d_ff": 1024,
21
+ "n_layers": 4,
22
+ "max_seq_len": 256,
23
+ "dropout": 0.1
24
+ },
25
+ "tokenizer": "gpt2",
26
+ "wandb_run_id": "oj3opt7s",
27
+ "wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/oj3opt7s",
28
+ "evaluations": [
29
+ {
30
+ "step": 1804,
31
+ "tokens": 3694592,
32
+ "fraction": 0.10003119043493329,
33
+ "val_loss": 6.6341400855553605,
34
+ "val_perplexity": 760.6247123420002,
35
+ "val_bleu": 1.6350012582058588
36
+ },
37
+ {
38
+ "step": 3607,
39
+ "tokens": 7387136,
40
+ "fraction": 0.20000693120776294,
41
+ "val_loss": 6.198281208198227,
42
+ "val_perplexity": 491.9028355150239,
43
+ "val_bleu": 2.268582026127263
44
+ },
45
+ {
46
+ "step": 5411,
47
+ "tokens": 11081728,
48
+ "fraction": 0.3000381216426962,
49
+ "val_loss": 5.955291291672788,
50
+ "val_perplexity": 385.7892715145811,
51
+ "val_bleu": 2.5229026037852225
52
+ },
53
+ {
54
+ "step": 7214,
55
+ "tokens": 14774272,
56
+ "fraction": 0.4000138624155259,
57
+ "val_loss": 5.7803394494656315,
58
+ "val_perplexity": 323.8691089562624,
59
+ "val_bleu": 2.0795491380807083
60
+ },
61
+ {
62
+ "step": 9018,
63
+ "tokens": 18468864,
64
+ "fraction": 0.5000450528504592,
65
+ "val_loss": 5.637433009471246,
66
+ "val_perplexity": 280.7411328906744,
67
+ "val_bleu": 2.5986586550967212
68
+ },
69
+ {
70
+ "step": 10821,
71
+ "tokens": 22161408,
72
+ "fraction": 0.6000207936232889,
73
+ "val_loss": 5.5303351569794374,
74
+ "val_perplexity": 252.22843297838526,
75
+ "val_bleu": 2.516097418373473
76
+ },
77
+ {
78
+ "step": 12625,
79
+ "tokens": 25856000,
80
+ "fraction": 0.7000519840582221,
81
+ "val_loss": 5.437061781892758,
82
+ "val_perplexity": 229.76608780916973,
83
+ "val_bleu": 2.36290606336987
84
+ },
85
+ {
86
+ "step": 14428,
87
+ "tokens": 29548544,
88
+ "fraction": 0.8000277248310518,
89
+ "val_loss": 5.366506714782791,
90
+ "val_perplexity": 214.11359988989213,
91
+ "val_bleu": 2.6501718493221613
92
+ },
93
+ {
94
+ "step": 16231,
95
+ "tokens": 33241088,
96
+ "fraction": 0.9000034656038814,
97
+ "val_loss": 5.3192690577573645,
98
+ "val_perplexity": 204.234543765364,
99
+ "val_bleu": 2.5978208001168825
100
+ },
101
+ {
102
+ "step": 18035,
103
+ "tokens": 36935680,
104
+ "fraction": 1.0000346560388147,
105
+ "val_loss": 5.253066647790387,
106
+ "val_perplexity": 191.15156507531572,
107
+ "val_bleu": 2.9645072122027476
108
+ }
109
+ ],
110
+ "best": {
111
+ "step": 18035,
112
+ "tokens": 36935680,
113
+ "fraction": 1.0000346560388147,
114
+ "val_loss": 5.253066647790387,
115
+ "val_perplexity": 191.15156507531572,
116
+ "val_bleu": 2.9645072122027476
117
+ },
118
+ "last": {
119
+ "step": 18035,
120
+ "tokens": 36935680,
121
+ "fraction": 1.0000346560388147,
122
+ "val_loss": 5.253066647790387,
123
+ "val_perplexity": 191.15156507531572,
124
+ "val_bleu": 2.9645072122027476
125
+ },
126
+ "last_checkpoint_step": 18035,
127
+ "last_checkpoint_tokens": 36935680,
128
+ "complete": true
129
+ }
checkpoints_part2/cautious/p2yw4r11/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:09ef92900ad49774d121f8093bbbbab9deedb0f381f4e2e1cc5d84ad995201b7
3
+ size 69249142
checkpoints_part2/cautious/p2yw4r11/last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:818165692b91a95607893f71214bdd70d130cbc503a04d4fb2663735a5d5ebe9
3
+ size 69249142
checkpoints_part2/cautious/p2yw4r11/metrics.json ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "optimizer": "cautious",
4
+ "lr": 0.001,
5
+ "weight_decay": 0.01,
6
+ "batch_size": 8,
7
+ "seq_len": 256,
8
+ "d_model": 256,
9
+ "n_heads": 8,
10
+ "d_ff": 1024,
11
+ "n_layers": 4,
12
+ "dropout": 0.1,
13
+ "epochs": 1,
14
+ "save_dir": "checkpoints_part2"
15
+ },
16
+ "model_config": {
17
+ "vocab_size": 50257,
18
+ "d_model": 256,
19
+ "n_heads": 8,
20
+ "d_ff": 1024,
21
+ "n_layers": 4,
22
+ "max_seq_len": 256,
23
+ "dropout": 0.1
24
+ },
25
+ "tokenizer": "gpt2",
26
+ "wandb_run_id": "p2yw4r11",
27
+ "wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/p2yw4r11",
28
+ "evaluations": [
29
+ {
30
+ "step": 1804,
31
+ "tokens": 3694592,
32
+ "fraction": 0.10003119043493329,
33
+ "val_loss": 6.8086098829905195,
34
+ "val_perplexity": 905.6110262375989,
35
+ "val_bleu": 1.3151229296275404
36
+ },
37
+ {
38
+ "step": 3607,
39
+ "tokens": 7387136,
40
+ "fraction": 0.20000693120776294,
41
+ "val_loss": 6.494788529630193,
42
+ "val_perplexity": 661.6842838933193,
43
+ "val_bleu": 1.7576461147222004
44
+ },
45
+ {
46
+ "step": 5411,
47
+ "tokens": 11081728,
48
+ "fraction": 0.3000381216426962,
49
+ "val_loss": 6.3169474725475805,
50
+ "val_perplexity": 553.8796764018344,
51
+ "val_bleu": 1.8936975883421343
52
+ },
53
+ {
54
+ "step": 7214,
55
+ "tokens": 14774272,
56
+ "fraction": 0.4000138624155259,
57
+ "val_loss": 6.130474798217743,
58
+ "val_perplexity": 459.6543519445046,
59
+ "val_bleu": 2.3554491109335634
60
+ },
61
+ {
62
+ "step": 9018,
63
+ "tokens": 18468864,
64
+ "fraction": 0.5000450528504592,
65
+ "val_loss": 5.957174865547531,
66
+ "val_perplexity": 386.51661889878403,
67
+ "val_bleu": 2.703317966629297
68
+ },
69
+ {
70
+ "step": 10821,
71
+ "tokens": 22161408,
72
+ "fraction": 0.6000207936232889,
73
+ "val_loss": 5.821500727278505,
74
+ "val_perplexity": 337.4781364342,
75
+ "val_bleu": 2.5474224881531287
76
+ },
77
+ {
78
+ "step": 12625,
79
+ "tokens": 25856000,
80
+ "fraction": 0.7000519840582221,
81
+ "val_loss": 5.718654285647912,
82
+ "val_perplexity": 304.4948839856075,
83
+ "val_bleu": 2.507245238116674
84
+ },
85
+ {
86
+ "step": 14428,
87
+ "tokens": 29548544,
88
+ "fraction": 0.8000277248310518,
89
+ "val_loss": 5.632188036532221,
90
+ "val_perplexity": 279.27250806132463,
91
+ "val_bleu": 2.6585916717692606
92
+ },
93
+ {
94
+ "step": 16231,
95
+ "tokens": 33241088,
96
+ "fraction": 0.9000034656038814,
97
+ "val_loss": 5.5536918744831505,
98
+ "val_perplexity": 258.1889998997565,
99
+ "val_bleu": 2.400037540916668
100
+ },
101
+ {
102
+ "step": 18035,
103
+ "tokens": 36935680,
104
+ "fraction": 1.0000346560388147,
105
+ "val_loss": 5.496509715944469,
106
+ "val_perplexity": 243.83937661071516,
107
+ "val_bleu": 2.5584566218227955
108
+ }
109
+ ],
110
+ "best": {
111
+ "step": 18035,
112
+ "tokens": 36935680,
113
+ "fraction": 1.0000346560388147,
114
+ "val_loss": 5.496509715944469,
115
+ "val_perplexity": 243.83937661071516,
116
+ "val_bleu": 2.5584566218227955
117
+ },
118
+ "last": {
119
+ "step": 18035,
120
+ "tokens": 36935680,
121
+ "fraction": 1.0000346560388147,
122
+ "val_loss": 5.496509715944469,
123
+ "val_perplexity": 243.83937661071516,
124
+ "val_bleu": 2.5584566218227955
125
+ },
126
+ "last_checkpoint_step": 18035,
127
+ "last_checkpoint_tokens": 36935680,
128
+ "complete": true
129
+ }
checkpoints_part2/lion/ydyarkxk/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:408e2993d97778e30db06cfbd6c6b8ecefd053c426e03d9f517dd6a9b65ea04b
3
+ size 69249142
checkpoints_part2/lion/ydyarkxk/last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd377b0467cdf5ab092d0200104c920f75e8653908cf626e7464a9e5e6e34a5b
3
+ size 69249142
checkpoints_part2/lion/ydyarkxk/metrics.json ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "optimizer": "lion",
4
+ "lr": 0.0002,
5
+ "weight_decay": 0.1,
6
+ "batch_size": 8,
7
+ "seq_len": 256,
8
+ "d_model": 256,
9
+ "n_heads": 8,
10
+ "d_ff": 1024,
11
+ "n_layers": 4,
12
+ "dropout": 0.1,
13
+ "epochs": 1,
14
+ "save_dir": "checkpoints_part2"
15
+ },
16
+ "model_config": {
17
+ "vocab_size": 50257,
18
+ "d_model": 256,
19
+ "n_heads": 8,
20
+ "d_ff": 1024,
21
+ "n_layers": 4,
22
+ "max_seq_len": 256,
23
+ "dropout": 0.1
24
+ },
25
+ "tokenizer": "gpt2",
26
+ "wandb_run_id": "ydyarkxk",
27
+ "wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/ydyarkxk",
28
+ "evaluations": [
29
+ {
30
+ "step": 1804,
31
+ "tokens": 3694592,
32
+ "fraction": 0.10003119043493329,
33
+ "val_loss": 7.238790913256342,
34
+ "val_perplexity": 1392.4094089228465,
35
+ "val_bleu": 1.1876307944768998
36
+ },
37
+ {
38
+ "step": 3607,
39
+ "tokens": 7387136,
40
+ "fraction": 0.20000693120776294,
41
+ "val_loss": 6.508020398621549,
42
+ "val_perplexity": 670.4977846044866,
43
+ "val_bleu": 1.951173313307093
44
+ },
45
+ {
46
+ "step": 5411,
47
+ "tokens": 11081728,
48
+ "fraction": 0.3000381216426962,
49
+ "val_loss": 5.978941541469977,
50
+ "val_perplexity": 395.0220323591368,
51
+ "val_bleu": 2.3198521517571797
52
+ },
53
+ {
54
+ "step": 7214,
55
+ "tokens": 14774272,
56
+ "fraction": 0.4000138624155259,
57
+ "val_loss": 5.678261665051093,
58
+ "val_perplexity": 292.44062805241543,
59
+ "val_bleu": 2.431544539213501
60
+ },
61
+ {
62
+ "step": 9018,
63
+ "tokens": 18468864,
64
+ "fraction": 0.5000450528504592,
65
+ "val_loss": 5.472018926443454,
66
+ "val_perplexity": 237.94009170092153,
67
+ "val_bleu": 2.5023165366302536
68
+ },
69
+ {
70
+ "step": 10821,
71
+ "tokens": 22161408,
72
+ "fraction": 0.6000207936232889,
73
+ "val_loss": 5.327088786218456,
74
+ "val_perplexity": 205.8378630306369,
75
+ "val_bleu": 2.6355905792527023
76
+ },
77
+ {
78
+ "step": 12625,
79
+ "tokens": 25856000,
80
+ "fraction": 0.7000519840582221,
81
+ "val_loss": 5.207654690790081,
82
+ "val_perplexity": 182.66514917149527,
83
+ "val_bleu": 2.438599562300715
84
+ },
85
+ {
86
+ "step": 14428,
87
+ "tokens": 29548544,
88
+ "fraction": 0.8000277248310518,
89
+ "val_loss": 5.110067813458319,
90
+ "val_perplexity": 165.6815899343745,
91
+ "val_bleu": 2.1012652918887254
92
+ },
93
+ {
94
+ "step": 16231,
95
+ "tokens": 33241088,
96
+ "fraction": 0.9000034656038814,
97
+ "val_loss": 5.023584961653231,
98
+ "val_perplexity": 151.9550816410011,
99
+ "val_bleu": 2.422311511198055
100
+ },
101
+ {
102
+ "step": 18035,
103
+ "tokens": 36935680,
104
+ "fraction": 1.0000346560388147,
105
+ "val_loss": 4.940937335143784,
106
+ "val_perplexity": 139.90132254703283,
107
+ "val_bleu": 2.5706617588457146
108
+ }
109
+ ],
110
+ "best": {
111
+ "step": 18035,
112
+ "tokens": 36935680,
113
+ "fraction": 1.0000346560388147,
114
+ "val_loss": 4.940937335143784,
115
+ "val_perplexity": 139.90132254703283,
116
+ "val_bleu": 2.5706617588457146
117
+ },
118
+ "last": {
119
+ "step": 18035,
120
+ "tokens": 36935680,
121
+ "fraction": 1.0000346560388147,
122
+ "val_loss": 4.940937335143784,
123
+ "val_perplexity": 139.90132254703283,
124
+ "val_bleu": 2.5706617588457146
125
+ },
126
+ "last_checkpoint_step": 18035,
127
+ "last_checkpoint_tokens": 36935680,
128
+ "complete": true
129
+ }
checkpoints_part2/scion/89yiklpr/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee3ec5cc364107236664985d9bdc258e1627df169c1be718f46b9a2cf463fd1b
3
+ size 69249142
checkpoints_part2/scion/89yiklpr/last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e2e19869d4ab0f853d61a56982410d80578e2ff17abb74527c96116de1bfe86
3
+ size 69249142
checkpoints_part2/scion/89yiklpr/metrics.json ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "optimizer": "scion",
4
+ "lr": 0.001,
5
+ "weight_decay": 0.01,
6
+ "batch_size": 8,
7
+ "seq_len": 256,
8
+ "d_model": 256,
9
+ "n_heads": 8,
10
+ "d_ff": 1024,
11
+ "n_layers": 4,
12
+ "dropout": 0.1,
13
+ "epochs": 1,
14
+ "save_dir": "checkpoints_part2"
15
+ },
16
+ "model_config": {
17
+ "vocab_size": 50257,
18
+ "d_model": 256,
19
+ "n_heads": 8,
20
+ "d_ff": 1024,
21
+ "n_layers": 4,
22
+ "max_seq_len": 256,
23
+ "dropout": 0.1
24
+ },
25
+ "tokenizer": "gpt2",
26
+ "wandb_run_id": "89yiklpr",
27
+ "wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/89yiklpr",
28
+ "evaluations": [
29
+ {
30
+ "step": 1804,
31
+ "tokens": 3694592,
32
+ "fraction": 0.10003119043493329,
33
+ "val_loss": 8.281115139791828,
34
+ "val_perplexity": 3948.5951633695317,
35
+ "val_bleu": 1.1122944171065337
36
+ },
37
+ {
38
+ "step": 3607,
39
+ "tokens": 7387136,
40
+ "fraction": 0.20000693120776294,
41
+ "val_loss": 7.1758178436827516,
42
+ "val_perplexity": 1307.4289363561054,
43
+ "val_bleu": 2.5826001619044807
44
+ },
45
+ {
46
+ "step": 5411,
47
+ "tokens": 11081728,
48
+ "fraction": 0.3000381216426962,
49
+ "val_loss": 6.6727667216531295,
50
+ "val_perplexity": 790.5798959226208,
51
+ "val_bleu": 2.5622937725329265
52
+ },
53
+ {
54
+ "step": 7214,
55
+ "tokens": 14774272,
56
+ "fraction": 0.4000138624155259,
57
+ "val_loss": 6.378503823232746,
58
+ "val_perplexity": 589.0457314885668,
59
+ "val_bleu": 2.526161505466013
60
+ },
61
+ {
62
+ "step": 9018,
63
+ "tokens": 18468864,
64
+ "fraction": 0.5000450528504592,
65
+ "val_loss": 6.168975746322297,
66
+ "val_perplexity": 477.69657295732503,
67
+ "val_bleu": 2.597034079802741
68
+ },
69
+ {
70
+ "step": 10821,
71
+ "tokens": 22161408,
72
+ "fraction": 0.6000207936232889,
73
+ "val_loss": 5.992401353375403,
74
+ "val_perplexity": 400.3748980829875,
75
+ "val_bleu": 2.5429924513715405
76
+ },
77
+ {
78
+ "step": 12625,
79
+ "tokens": 25856000,
80
+ "fraction": 0.7000519840582221,
81
+ "val_loss": 5.851240207335192,
82
+ "val_perplexity": 347.6652902579374,
83
+ "val_bleu": 2.4854103863453743
84
+ },
85
+ {
86
+ "step": 14428,
87
+ "tokens": 29548544,
88
+ "fraction": 0.8000277248310518,
89
+ "val_loss": 5.733623129165101,
90
+ "val_perplexity": 309.0871046253544,
91
+ "val_bleu": 2.5172439325356755
92
+ },
93
+ {
94
+ "step": 16231,
95
+ "tokens": 33241088,
96
+ "fraction": 0.9000034656038814,
97
+ "val_loss": 5.631224749806874,
98
+ "val_perplexity": 279.00361809141646,
99
+ "val_bleu": 2.8175740515027603
100
+ },
101
+ {
102
+ "step": 18035,
103
+ "tokens": 36935680,
104
+ "fraction": 1.0000346560388147,
105
+ "val_loss": 5.533019167220521,
106
+ "val_perplexity": 252.9063260045474,
107
+ "val_bleu": 2.7522405185633163
108
+ }
109
+ ],
110
+ "best": {
111
+ "step": 18035,
112
+ "tokens": 36935680,
113
+ "fraction": 1.0000346560388147,
114
+ "val_loss": 5.533019167220521,
115
+ "val_perplexity": 252.9063260045474,
116
+ "val_bleu": 2.7522405185633163
117
+ },
118
+ "last": {
119
+ "step": 18035,
120
+ "tokens": 36935680,
121
+ "fraction": 1.0000346560388147,
122
+ "val_loss": 5.533019167220521,
123
+ "val_perplexity": 252.9063260045474,
124
+ "val_bleu": 2.7522405185633163
125
+ },
126
+ "last_checkpoint_step": 18035,
127
+ "last_checkpoint_tokens": 36935680,
128
+ "complete": true
129
+ }
checkpoints_part2/sophia/3j32zwf3/best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf87bebcd359e661b0403c30ff57a06bdbbd42ed3fdfce7e84de7ce22d3892de
3
+ size 69249142
checkpoints_part2/sophia/3j32zwf3/last.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ca778cca77a47b8ae02e0d3bf2303fb031f46ba0ced12b3df7296df69642388
3
+ size 69249142
checkpoints_part2/sophia/3j32zwf3/metrics.json ADDED
@@ -0,0 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "optimizer": "sophia",
4
+ "lr": 0.0001,
5
+ "rho": 0.04,
6
+ "weight_decay": 0.1,
7
+ "batch_size": 8,
8
+ "seq_len": 256,
9
+ "d_model": 256,
10
+ "n_heads": 8,
11
+ "d_ff": 1024,
12
+ "n_layers": 4,
13
+ "dropout": 0.1,
14
+ "epochs": 1,
15
+ "save_dir": "checkpoints_part2"
16
+ },
17
+ "model_config": {
18
+ "vocab_size": 50257,
19
+ "d_model": 256,
20
+ "n_heads": 8,
21
+ "d_ff": 1024,
22
+ "n_layers": 4,
23
+ "max_seq_len": 256,
24
+ "dropout": 0.1
25
+ },
26
+ "tokenizer": "gpt2",
27
+ "wandb_run_id": "3j32zwf3",
28
+ "wandb_url": "https://wandb.ai/chechashreyas-international-institute-of-information-tec/ANLP-Assignment-2-Part2/runs/3j32zwf3",
29
+ "evaluations": [
30
+ {
31
+ "step": 1804,
32
+ "tokens": 3694592,
33
+ "fraction": 0.10003119043493329,
34
+ "val_loss": 7.448652947972159,
35
+ "val_perplexity": 1717.5479599260464,
36
+ "val_bleu": 1.515281006968686
37
+ },
38
+ {
39
+ "step": 3607,
40
+ "tokens": 7387136,
41
+ "fraction": 0.20000693120776294,
42
+ "val_loss": 6.992933257611212,
43
+ "val_perplexity": 1088.910852313718,
44
+ "val_bleu": 1.3970472611186042
45
+ },
46
+ {
47
+ "step": 5411,
48
+ "tokens": 11081728,
49
+ "fraction": 0.3000381216426962,
50
+ "val_loss": 6.619747597300364,
51
+ "val_perplexity": 749.7558328312051,
52
+ "val_bleu": 2.2005620954722893
53
+ },
54
+ {
55
+ "step": 7214,
56
+ "tokens": 14774272,
57
+ "fraction": 0.4000138624155259,
58
+ "val_loss": 6.32087761913231,
59
+ "val_perplexity": 556.0607879569982,
60
+ "val_bleu": 2.5882135890697797
61
+ },
62
+ {
63
+ "step": 9018,
64
+ "tokens": 18468864,
65
+ "fraction": 0.5000450528504592,
66
+ "val_loss": 6.0633629972111445,
67
+ "val_perplexity": 429.8184873861714,
68
+ "val_bleu": 2.042761820526234
69
+ },
70
+ {
71
+ "step": 10821,
72
+ "tokens": 22161408,
73
+ "fraction": 0.6000207936232889,
74
+ "val_loss": 5.861090107116395,
75
+ "val_perplexity": 351.1066793695729,
76
+ "val_bleu": 2.2881800394766634
77
+ },
78
+ {
79
+ "step": 12625,
80
+ "tokens": 25856000,
81
+ "fraction": 0.7000519840582221,
82
+ "val_loss": 5.710041520124424,
83
+ "val_perplexity": 301.8836022673098,
84
+ "val_bleu": 2.4996404904784804
85
+ },
86
+ {
87
+ "step": 14428,
88
+ "tokens": 29548544,
89
+ "fraction": 0.8000277248310518,
90
+ "val_loss": 5.5844624865792705,
91
+ "val_perplexity": 266.2571274895874,
92
+ "val_bleu": 2.5117622082097006
93
+ },
94
+ {
95
+ "step": 16231,
96
+ "tokens": 33241088,
97
+ "fraction": 0.9000034656038814,
98
+ "val_loss": 5.479142612087989,
99
+ "val_perplexity": 239.64115383876725,
100
+ "val_bleu": 2.522367710436995
101
+ },
102
+ {
103
+ "step": 18035,
104
+ "tokens": 36935680,
105
+ "fraction": 1.0000346560388147,
106
+ "val_loss": 5.389180385899877,
107
+ "val_perplexity": 219.02379697507902,
108
+ "val_bleu": 2.55340545320376
109
+ }
110
+ ],
111
+ "best": {
112
+ "step": 18035,
113
+ "tokens": 36935680,
114
+ "fraction": 1.0000346560388147,
115
+ "val_loss": 5.389180385899877,
116
+ "val_perplexity": 219.02379697507902,
117
+ "val_bleu": 2.55340545320376
118
+ },
119
+ "last": {
120
+ "step": 18035,
121
+ "tokens": 36935680,
122
+ "fraction": 1.0000346560388147,
123
+ "val_loss": 5.389180385899877,
124
+ "val_perplexity": 219.02379697507902,
125
+ "val_bleu": 2.55340545320376
126
+ },
127
+ "last_checkpoint_step": 18035,
128
+ "last_checkpoint_tokens": 36935680,
129
+ "complete": true
130
+ }
src/part2/train.py CHANGED
@@ -8,6 +8,8 @@ from transformers import AutoTokenizer
8
  import sacrebleu
9
  import wandb
10
  import argparse
 
 
11
  import yaml
12
  from tqdm import tqdm
13
  from .model import DecoderOnlyTransformer
@@ -86,7 +88,7 @@ def evaluate(model, val_loader, criterion, device = "cuda"):
86
 
87
  def train(args):
88
 
89
- wandb.init(
90
  project = "ANLP-Assignment-2-Part2",
91
  name = f"{args.optimizer}_lr{args.lr}_wd{args.weight_decay}",
92
  config = vars(args),
@@ -118,6 +120,27 @@ def train(args):
118
  dropout = args.dropout,
119
  ).to(device)
120
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
  if args.optimizer == "adamw":
122
  optimizer = CustomAdamW(model.parameters(), lr = args.lr, weight_decay = args.weight_decay)
123
  elif args.optimizer == "cautious":
@@ -193,6 +216,23 @@ def train(args):
193
  "fraction": fraction,
194
  })
195
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
196
  next_eval_token += eval_token_interval
197
  model.train()
198
 
@@ -201,14 +241,27 @@ def train(args):
201
  if total_tokens_processed >= total_tokens_budget:
202
  break
203
 
 
 
 
 
 
 
204
  wandb.finish()
205
 
 
 
 
 
 
 
206
  def parse_args():
207
  parser = argparse.ArgumentParser(description="Train transformer with specified optimizer config(s)")
208
  parser.add_argument("--config", type=str, default=None, help="Path to YAML config file")
209
- parser.add_argument("--all", action="store_true", help="Train all 4 optimizer configs sequentially")
210
  parser.add_argument("--lr", type=float, default=None)
211
  parser.add_argument("--batch_size", type=int, default=None)
 
212
  cli_args = parser.parse_args()
213
  if not cli_args.config and not cli_args.all:
214
  parser.error("You must specify either --config <path> or --all")
@@ -221,6 +274,7 @@ def build_config(config_path, cli_args):
221
  config["lr"] = cli_args.lr
222
  if cli_args.batch_size is not None:
223
  config["batch_size"] = cli_args.batch_size
 
224
  return argparse.Namespace(**config)
225
 
226
  if __name__ == "__main__":
@@ -244,4 +298,4 @@ if __name__ == "__main__":
244
  torch.cuda.empty_cache()
245
  else:
246
  args = build_config(cli_args.config, cli_args)
247
- train(args)
 
8
  import sacrebleu
9
  import wandb
10
  import argparse
11
+ import json
12
+ from pathlib import Path
13
  import yaml
14
  from tqdm import tqdm
15
  from .model import DecoderOnlyTransformer
 
88
 
89
  def train(args):
90
 
91
+ run = wandb.init(
92
  project = "ANLP-Assignment-2-Part2",
93
  name = f"{args.optimizer}_lr{args.lr}_wd{args.weight_decay}",
94
  config = vars(args),
 
120
  dropout = args.dropout,
121
  ).to(device)
122
 
123
+ checkpoint_dir = Path(args.save_dir) / args.optimizer / run.id
124
+ checkpoint_dir.mkdir(parents = True, exist_ok = False)
125
+ metadata = {
126
+ "config": vars(args),
127
+ "model_config": {
128
+ "vocab_size": tokenizer.vocab_size,
129
+ "d_model": args.d_model,
130
+ "n_heads": args.n_heads,
131
+ "d_ff": args.d_ff,
132
+ "n_layers": args.n_layers,
133
+ "max_seq_len": args.seq_len,
134
+ "dropout": args.dropout,
135
+ },
136
+ "tokenizer": "gpt2",
137
+ "wandb_run_id": run.id,
138
+ "wandb_url": run.url,
139
+ "evaluations": [],
140
+ }
141
+ best_val_loss = float("inf")
142
+ print(f"Checkpoints: {checkpoint_dir}")
143
+
144
  if args.optimizer == "adamw":
145
  optimizer = CustomAdamW(model.parameters(), lr = args.lr, weight_decay = args.weight_decay)
146
  elif args.optimizer == "cautious":
 
216
  "fraction": fraction,
217
  })
218
 
219
+ metrics = {
220
+ "step": step_count,
221
+ "tokens": total_tokens_processed,
222
+ "fraction": fraction,
223
+ "val_loss": val_loss,
224
+ "val_perplexity": val_ppl,
225
+ "val_bleu": bleu,
226
+ }
227
+ metadata["evaluations"].append(metrics)
228
+ if val_loss < best_val_loss:
229
+ best_val_loss = val_loss
230
+ save_weights(model, checkpoint_dir / "best.pt")
231
+ metadata["best"] = metrics
232
+ save_weights(model, checkpoint_dir / "last.pt")
233
+ metadata["last"] = metrics
234
+ (checkpoint_dir / "metrics.json").write_text(json.dumps(metadata, indent = 2) + "\n")
235
+
236
  next_eval_token += eval_token_interval
237
  model.train()
238
 
 
241
  if total_tokens_processed >= total_tokens_budget:
242
  break
243
 
244
+ save_weights(model, checkpoint_dir / "last.pt")
245
+ metadata["last_checkpoint_step"] = step_count
246
+ metadata["last_checkpoint_tokens"] = total_tokens_processed
247
+ metadata["complete"] = True
248
+ (checkpoint_dir / "metrics.json").write_text(json.dumps(metadata, indent = 2) + "\n")
249
+ print(f"Saved weights and metrics to {checkpoint_dir}")
250
  wandb.finish()
251
 
252
+ def save_weights(model, path):
253
+ # Replace only after the entire state dict has been written.
254
+ temporary_path = path.with_suffix(".pt.tmp")
255
+ torch.save(model.state_dict(), temporary_path)
256
+ temporary_path.replace(path)
257
+
258
  def parse_args():
259
  parser = argparse.ArgumentParser(description="Train transformer with specified optimizer config(s)")
260
  parser.add_argument("--config", type=str, default=None, help="Path to YAML config file")
261
+ parser.add_argument("--all", action="store_true", help="Train all 5 optimizer configs sequentially")
262
  parser.add_argument("--lr", type=float, default=None)
263
  parser.add_argument("--batch_size", type=int, default=None)
264
+ parser.add_argument("--save_dir", type=str, default="checkpoints_part2", help="Root directory for per-run weights and metrics")
265
  cli_args = parser.parse_args()
266
  if not cli_args.config and not cli_args.all:
267
  parser.error("You must specify either --config <path> or --all")
 
274
  config["lr"] = cli_args.lr
275
  if cli_args.batch_size is not None:
276
  config["batch_size"] = cli_args.batch_size
277
+ config["save_dir"] = cli_args.save_dir
278
  return argparse.Namespace(**config)
279
 
280
  if __name__ == "__main__":
 
298
  torch.cuda.empty_cache()
299
  else:
300
  args = build_config(cli_args.config, cli_args)
301
+ train(args)