Hugging Face's logo

coeiico
/
tiny-transformer

PyTorch
Safetensors
transformer
interpretability
from-scratch
Model card Files
xet
 Community
coeiico commited on
Commit
4a570eb
·
verified ·
1 Parent(s): 9cdf072

Add every run in its own folder and the model card

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +34 -0
  2. bigram-seed0/analysis/figures/figure-data/curves-bigram-seed0.csv +17 -0
  3. bigram-seed0/analysis/figures/results.json +29 -0
  4. bigram-seed0/analysis/figures/source-snapshot.tar.gz +3 -0
  5. bigram-seed0/analysis/final/results.json +29 -0
  6. bigram-seed0/analysis/final/source-snapshot.tar.gz +3 -0
  7. bigram-seed0/metrics.json +228 -0
  8. bigram-seed0/model.safetensors +3 -0
  9. bigram-seed0/model_config.json +81 -0
  10. bigram-seed0/source-snapshot.tar.gz +3 -0
  11. bigram-seed0/training.png +0 -0
  12. default-seed0/analysis/figures/critical_pattern.png +0 -0
  13. default-seed0/analysis/figures/figure-data/curves-default-seed0.csv +17 -0
  14. default-seed0/analysis/figures/figure-data/heads-default-seed0.csv +33 -0
  15. default-seed0/analysis/figures/figure-data/layers-default-seed0.csv +5 -0
  16. default-seed0/analysis/figures/figure-data/pattern-default-seed0.json +1279 -0
  17. default-seed0/analysis/figures/head_ablation.png +0 -0
  18. default-seed0/analysis/figures/lookup_current.png +0 -0
  19. default-seed0/analysis/figures/lookup_next.png +0 -0
  20. default-seed0/analysis/figures/lookup_next_pattern.png +0 -0
  21. default-seed0/analysis/figures/results.json +219 -0
  22. default-seed0/analysis/figures/source-snapshot.tar.gz +3 -0
  23. default-seed0/analysis/final/critical_pattern.png +0 -0
  24. default-seed0/analysis/final/head_ablation.png +0 -0
  25. default-seed0/analysis/final/lookup_current.png +0 -0
  26. default-seed0/analysis/final/lookup_next.png +0 -0
  27. default-seed0/analysis/final/lookup_next_pattern.png +0 -0
  28. default-seed0/analysis/final/results.json +219 -0
  29. default-seed0/analysis/final/source-snapshot.tar.gz +3 -0
  30. default-seed0/metrics.json +228 -0
  31. default-seed0/model.safetensors +3 -0
  32. default-seed0/model_config.json +81 -0
  33. default-seed0/source-snapshot.tar.gz +3 -0
  34. default-seed0/training.png +0 -0
  35. default-seed1/analysis/figures/critical_pattern.png +0 -0
  36. default-seed1/analysis/figures/figure-data/curves-default-seed1.csv +17 -0
  37. default-seed1/analysis/figures/figure-data/heads-default-seed1.csv +33 -0
  38. default-seed1/analysis/figures/figure-data/layers-default-seed1.csv +5 -0
  39. default-seed1/analysis/figures/figure-data/pattern-default-seed1.json +1279 -0
  40. default-seed1/analysis/figures/head_ablation.png +0 -0
  41. default-seed1/analysis/figures/lookup_current.png +0 -0
  42. default-seed1/analysis/figures/lookup_next.png +0 -0
  43. default-seed1/analysis/figures/lookup_next_pattern.png +0 -0
  44. default-seed1/analysis/figures/results.json +219 -0
  45. default-seed1/analysis/figures/source-snapshot.tar.gz +3 -0
  46. default-seed1/analysis/final/critical_pattern.png +0 -0
  47. default-seed1/analysis/final/head_ablation.png +0 -0
  48. default-seed1/analysis/final/lookup_current.png +0 -0
  49. default-seed1/analysis/final/lookup_next.png +0 -0
  50. default-seed1/analysis/final/lookup_next_pattern.png +0 -0
README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - transformer
5
+ - interpretability
6
+ - from-scratch
7
+ library_name: pytorch
8
+ ---
9
+
10
+ # tiny-transformer
11
+
12
+ Trained runs for [tiny-transformer](https://github.com/coeiico/tiny-transformer): a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub.
13
+
14
+ Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/`.
15
+
16
+ | Run | What it is | Lists sorted exactly (test, 2,048 lists) |
17
+ | --- | --- | --- |
18
+ | `default-seed0` | 4 layers, 3.2M parameters | 100% |
19
+ | `default-seed1` | same, seed 1 | 100% |
20
+ | `default-seed2` | same, seed 2 | 99.8% |
21
+ | `bigram-seed0` | baseline: no layers (`--model.n-layers 0`), sees only the previous token | 0% |
22
+
23
+ To load one, with the [tiny-transformer](https://github.com/coeiico/tiny-transformer) package installed:
24
+
25
+ ```python
26
+ from pathlib import Path
27
+ from huggingface_hub import snapshot_download
28
+ from tiny_transformer.hub import load_model
29
+
30
+ folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0'
31
+ model, vocab = load_model(folder)
32
+ ```
33
+
34
+ These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device.
bigram-seed0/analysis/figures/figure-data/curves-bigram-seed0.csv ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ step,train_loss,val_loss,val_token_accuracy,val_exact_match
2
+ 250,2.346252202987671,2.3416833877563477,0.19580078125,0.0
3
+ 500,2.3324460983276367,2.33697247505188,0.19647216796875,0.0
4
+ 750,2.339632511138916,2.337479591369629,0.1966552734375,0.0
5
+ 1000,2.33378267288208,2.336371421813965,0.193359375,0.0
6
+ 1250,2.3311407566070557,2.3351407051086426,0.1973876953125,0.0
7
+ 1500,2.3376550674438477,2.3326027393341064,0.2003173828125,0.0
8
+ 1750,2.3303802013397217,2.3325517177581787,0.19921875,0.0
9
+ 2000,2.333252429962158,2.3325724601745605,0.1982421875,0.0
10
+ 2250,2.329752206802368,2.3292136192321777,0.19915771484375,0.0
11
+ 2500,2.325176239013672,2.329281806945801,0.19976806640625,0.0
12
+ 2750,2.3266186714172363,2.3286523818969727,0.200927734375,0.0
13
+ 3000,2.3282289505004883,2.3266916275024414,0.20263671875,0.0
14
+ 3250,2.3204801082611084,2.3267407417297363,0.20269775390625,0.0
15
+ 3500,2.3247458934783936,2.3262388706207275,0.20135498046875,0.0
16
+ 3750,2.3221328258514404,2.325857162475586,0.2020263671875,0.0
17
+ 4000,2.3267335891723633,2.324476718902588,0.20184326171875,0.0
bigram-seed0/analysis/figures/results.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "runs/bigram-seed0",
3
+ "code": {
4
+ "commit": "42933bd7a4e2e13fd737d0d48d489f72fbeed8ac",
5
+ "dirty": true,
6
+ "snapshot": "source-snapshot.tar.gz"
7
+ },
8
+ "environment": {
9
+ "python": "3.12.3",
10
+ "torch": "2.15.0+cu132",
11
+ "device": "cuda",
12
+ "precision": "torch.float32",
13
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
14
+ "gpu": "NVIDIA GeForce RTX 3060",
15
+ "cuda": "13.2"
16
+ },
17
+ "test_size": 2048,
18
+ "chance_token_accuracy": 0.015625,
19
+ "trained": {
20
+ "loss": 2.323464870452881,
21
+ "token_accuracy": 0.20025634765625,
22
+ "exact_match": 0.0
23
+ },
24
+ "untrained": {
25
+ "loss": 4.254000663757324,
26
+ "token_accuracy": 0.011566162109375,
27
+ "exact_match": 0.0
28
+ }
29
+ }
bigram-seed0/analysis/figures/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ade11e9af3c8d5c2d74eaab0390b06cc6580dcebb2fa5fe3a3d64b4974b3ae70
3
+ size 347464
bigram-seed0/analysis/final/results.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "runs/bigram-seed0",
3
+ "code": {
4
+ "commit": "19276f75392bee9d5f6f61820395002a0746e349",
5
+ "dirty": true,
6
+ "snapshot": "source-snapshot.tar.gz"
7
+ },
8
+ "environment": {
9
+ "python": "3.12.3",
10
+ "torch": "2.15.0+cu132",
11
+ "device": "cuda",
12
+ "precision": "torch.float32",
13
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
14
+ "gpu": "NVIDIA GeForce RTX 3060",
15
+ "cuda": "13.2"
16
+ },
17
+ "test_size": 2048,
18
+ "chance_token_accuracy": 0.015625,
19
+ "trained": {
20
+ "loss": 2.323464870452881,
21
+ "token_accuracy": 0.20025634765625,
22
+ "exact_match": 0.0
23
+ },
24
+ "untrained": {
25
+ "loss": 4.254000663757324,
26
+ "token_accuracy": 0.011566162109375,
27
+ "exact_match": 0.0
28
+ }
29
+ }
bigram-seed0/analysis/final/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc7e4cff4eccbd18081051577563726d7aa7a1642c5680295777bc9ce1c17ce5
3
+ size 189601
bigram-seed0/metrics.json ADDED
@@ -0,0 +1,228 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "bigram-seed0",
4
+ "seed": 0,
5
+ "quick": false,
6
+ "device": "auto",
7
+ "eval_every": 250,
8
+ "task": {
9
+ "n_values": 64,
10
+ "length": 16
11
+ },
12
+ "splits": {
13
+ "val_seed": 1000003,
14
+ "val_size": 1024,
15
+ "test_seed": 2000003,
16
+ "test_size": 2048
17
+ },
18
+ "model": {
19
+ "n_layers": 0,
20
+ "d_model": 256,
21
+ "n_heads": 8,
22
+ "d_mlp": 1024,
23
+ "mlp": "gelu",
24
+ "norm": "layernorm",
25
+ "dropout": 0.0
26
+ },
27
+ "optim": {
28
+ "steps": 4000,
29
+ "batch_size": 256,
30
+ "lr": 0.001,
31
+ "warmup_steps": 100,
32
+ "min_lr_ratio": 0.1,
33
+ "weight_decay": 0.1,
34
+ "grad_clip": 1.0
35
+ }
36
+ },
37
+ "model_config": {
38
+ "vocab_size": 65,
39
+ "n_layers": 0,
40
+ "d_model": 256,
41
+ "n_heads": 8,
42
+ "d_mlp": 1024,
43
+ "max_seq_len": 33,
44
+ "mlp": "gelu",
45
+ "norm": "layernorm",
46
+ "dropout": 0.0
47
+ },
48
+ "parameters": 42240,
49
+ "data": {
50
+ "task": "sorting",
51
+ "version": 1,
52
+ "n_values": 64,
53
+ "length": 16,
54
+ "val_seed": 1000003,
55
+ "val_size": 1024,
56
+ "test_seed": 2000003,
57
+ "test_size": 2048,
58
+ "train_seed": 0
59
+ },
60
+ "code": {
61
+ "commit": "19276f75392bee9d5f6f61820395002a0746e349",
62
+ "dirty": true,
63
+ "snapshot": "source-snapshot.tar.gz"
64
+ },
65
+ "environment": {
66
+ "python": "3.12.3",
67
+ "torch": "2.15.0+cu132",
68
+ "device": "cuda",
69
+ "precision": "torch.float32",
70
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
71
+ "gpu": "NVIDIA GeForce RTX 3060",
72
+ "cuda": "13.2"
73
+ },
74
+ "resumable": false,
75
+ "status": "finished",
76
+ "seconds": 141.9,
77
+ "history": [
78
+ {
79
+ "step": 250,
80
+ "train_loss": 2.346252202987671,
81
+ "val_loss": 2.3416833877563477,
82
+ "val_token_accuracy": 0.19580078125,
83
+ "val_exact_match": 0.0,
84
+ "grad_norm": 0.4313671886920929,
85
+ "lr": 0.0009967189933441244
86
+ },
87
+ {
88
+ "step": 500,
89
+ "train_loss": 2.3324460983276367,
90
+ "val_loss": 2.33697247505188,
91
+ "val_token_accuracy": 0.19647216796875,
92
+ "val_exact_match": 0.0,
93
+ "grad_norm": 0.3702669143676758,
94
+ "lr": 0.0009768413988762155
95
+ },
96
+ {
97
+ "step": 750,
98
+ "train_loss": 2.339632511138916,
99
+ "val_loss": 2.337479591369629,
100
+ "val_token_accuracy": 0.1966552734375,
101
+ "val_exact_match": 0.0,
102
+ "grad_norm": 0.3715057969093323,
103
+ "lr": 0.0009397114317029974
104
+ },
105
+ {
106
+ "step": 1000,
107
+ "train_loss": 2.33378267288208,
108
+ "val_loss": 2.336371421813965,
109
+ "val_token_accuracy": 0.193359375,
110
+ "val_exact_match": 0.0,
111
+ "grad_norm": 0.35880762338638306,
112
+ "lr": 0.0008868298366769955
113
+ },
114
+ {
115
+ "step": 1250,
116
+ "train_loss": 2.3311407566070557,
117
+ "val_loss": 2.3351407051086426,
118
+ "val_token_accuracy": 0.1973876953125,
119
+ "val_exact_match": 0.0,
120
+ "grad_norm": 0.3279813230037689,
121
+ "lr": 0.0008203340189070906
122
+ },
123
+ {
124
+ "step": 1500,
125
+ "train_loss": 2.3376550674438477,
126
+ "val_loss": 2.3326027393341064,
127
+ "val_token_accuracy": 0.2003173828125,
128
+ "val_exact_match": 0.0,
129
+ "grad_norm": 0.34029656648635864,
130
+ "lr": 0.0007429116526313744
131
+ },
132
+ {
133
+ "step": 1750,
134
+ "train_loss": 2.3303802013397217,
135
+ "val_loss": 2.3325517177581787,
136
+ "val_token_accuracy": 0.19921875,
137
+ "val_exact_match": 0.0,
138
+ "grad_norm": 0.3435227870941162,
139
+ "lr": 0.0006576920489294011
140
+ },
141
+ {
142
+ "step": 2000,
143
+ "train_loss": 2.333252429962158,
144
+ "val_loss": 2.3325724601745605,
145
+ "val_token_accuracy": 0.1982421875,
146
+ "val_exact_match": 0.0,
147
+ "grad_norm": 0.31740111112594604,
148
+ "lr": 0.0005681196730492369
149
+ },
150
+ {
151
+ "step": 2250,
152
+ "train_loss": 2.329752206802368,
153
+ "val_loss": 2.3292136192321777,
154
+ "val_token_accuracy": 0.19915771484375,
155
+ "val_exact_match": 0.0,
156
+ "grad_norm": 0.31328055262565613,
157
+ "lr": 0.00047781492361400794
158
+ },
159
+ {
160
+ "step": 2500,
161
+ "train_loss": 2.325176239013672,
162
+ "val_loss": 2.329281806945801,
163
+ "val_token_accuracy": 0.19976806640625,
164
+ "val_exact_match": 0.0,
165
+ "grad_norm": 0.3233284652233124,
166
+ "lr": 0.0003904278008308589
167
+ },
168
+ {
169
+ "step": 2750,
170
+ "train_loss": 2.3266186714172363,
171
+ "val_loss": 2.3286523818969727,
172
+ "val_token_accuracy": 0.200927734375,
173
+ "val_exact_match": 0.0,
174
+ "grad_norm": 0.32487940788269043,
175
+ "lr": 0.0003094903782424895
176
+ },
177
+ {
178
+ "step": 3000,
179
+ "train_loss": 2.3282289505004883,
180
+ "val_loss": 2.3266916275024414,
181
+ "val_token_accuracy": 0.20263671875,
182
+ "val_exact_match": 0.0,
183
+ "grad_norm": 0.30471721291542053,
184
+ "lr": 0.0002382740409206803
185
+ },
186
+ {
187
+ "step": 3250,
188
+ "train_loss": 2.3204801082611084,
189
+ "val_loss": 2.3267407417297363,
190
+ "val_token_accuracy": 0.20269775390625,
191
+ "val_exact_match": 0.0,
192
+ "grad_norm": 0.31849977374076843,
193
+ "lr": 0.00017965726034785466
194
+ },
195
+ {
196
+ "step": 3500,
197
+ "train_loss": 2.3247458934783936,
198
+ "val_loss": 2.3262388706207275,
199
+ "val_token_accuracy": 0.20135498046875,
200
+ "val_exact_match": 0.0,
201
+ "grad_norm": 0.29355937242507935,
202
+ "lr": 0.00013600925035352912
203
+ },
204
+ {
205
+ "step": 3750,
206
+ "train_loss": 2.3221328258514404,
207
+ "val_loss": 2.325857162475586,
208
+ "val_token_accuracy": 0.2020263671875,
209
+ "val_exact_match": 0.0,
210
+ "grad_norm": 0.30890047550201416,
211
+ "lr": 0.00010909420658097956
212
+ },
213
+ {
214
+ "step": 4000,
215
+ "train_loss": 2.3267335891723633,
216
+ "val_loss": 2.324476718902588,
217
+ "val_token_accuracy": 0.20184326171875,
218
+ "val_exact_match": 0.0,
219
+ "grad_norm": 0.3154265582561493,
220
+ "lr": 0.0001
221
+ }
222
+ ],
223
+ "final": {
224
+ "val_loss": 2.324476718902588,
225
+ "val_token_accuracy": 0.20184326171875,
226
+ "val_exact_match": 0.0
227
+ }
228
+ }
bigram-seed0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b51f191672160991e25bc1cbb7e37178444fa3a3fba846188aca9d896a972215
3
+ size 169360
bigram-seed0/model_config.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "format_version": 1,
3
+ "transformer": {
4
+ "vocab_size": 65,
5
+ "n_layers": 0,
6
+ "d_model": 256,
7
+ "n_heads": 8,
8
+ "d_mlp": 1024,
9
+ "max_seq_len": 33,
10
+ "mlp": "gelu",
11
+ "norm": "layernorm",
12
+ "dropout": 0.0
13
+ },
14
+ "vocab": [
15
+ "0",
16
+ "1",
17
+ "2",
18
+ "3",
19
+ "4",
20
+ "5",
21
+ "6",
22
+ "7",
23
+ "8",
24
+ "9",
25
+ "10",
26
+ "11",
27
+ "12",
28
+ "13",
29
+ "14",
30
+ "15",
31
+ "16",
32
+ "17",
33
+ "18",
34
+ "19",
35
+ "20",
36
+ "21",
37
+ "22",
38
+ "23",
39
+ "24",
40
+ "25",
41
+ "26",
42
+ "27",
43
+ "28",
44
+ "29",
45
+ "30",
46
+ "31",
47
+ "32",
48
+ "33",
49
+ "34",
50
+ "35",
51
+ "36",
52
+ "37",
53
+ "38",
54
+ "39",
55
+ "40",
56
+ "41",
57
+ "42",
58
+ "43",
59
+ "44",
60
+ "45",
61
+ "46",
62
+ "47",
63
+ "48",
64
+ "49",
65
+ "50",
66
+ "51",
67
+ "52",
68
+ "53",
69
+ "54",
70
+ "55",
71
+ "56",
72
+ "57",
73
+ "58",
74
+ "59",
75
+ "60",
76
+ "61",
77
+ "62",
78
+ "63",
79
+ "SEP"
80
+ ]
81
+ }
bigram-seed0/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e963742ccde0aca2462ea11e294c6b2504355b7b78eda8738885e210454496e
3
+ size 187056
bigram-seed0/training.png ADDED
default-seed0/analysis/figures/critical_pattern.png ADDED
default-seed0/analysis/figures/figure-data/curves-default-seed0.csv ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ step,train_loss,val_loss,val_token_accuracy,val_exact_match
2
+ 250,0.07291533797979355,0.0985519140958786,0.9683837890625,0.595703125
3
+ 500,0.014207392930984497,0.031683966517448425,0.99053955078125,0.8720703125
4
+ 750,0.02052781730890274,0.023844445124268532,0.99273681640625,0.890625
5
+ 1000,0.009082821197807789,0.01231020875275135,0.996826171875,0.951171875
6
+ 1250,0.00922011211514473,0.019146941602230072,0.994873046875,0.9208984375
7
+ 1500,0.01364622451364994,0.005576150491833687,0.998779296875,0.9814453125
8
+ 1750,0.003266311716288328,0.001853839261457324,0.9996337890625,0.994140625
9
+ 2000,0.005052485503256321,0.0006970376125536859,0.9998779296875,0.998046875
10
+ 2250,0.00020874346955679357,0.0009217980550602078,0.99981689453125,0.9970703125
11
+ 2500,0.001243826001882553,0.0005396516062319279,0.99981689453125,0.9970703125
12
+ 2750,0.00022199669911060482,0.00017582655709702522,0.99993896484375,0.9990234375
13
+ 3000,0.00021956569980829954,8.435545896645635e-05,1.0,1.0
14
+ 3250,3.6058951081940904e-05,8.97766076377593e-05,0.99993896484375,0.9990234375
15
+ 3500,4.75610613648314e-06,9.887960004562046e-06,1.0,1.0
16
+ 3750,0.00010177387594012544,0.00023778356262482703,0.99993896484375,0.9990234375
17
+ 4000,1.727779454085976e-05,1.847820749389939e-05,1.0,1.0
default-seed0/analysis/figures/figure-data/heads-default-seed0.csv ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ layer,head,loss_rise,token_accuracy_drop,lookup_current,lookup_next
2
+ 0,0,0.09456062089066108,0.021453857421875,0.58856201171875,0.196746826171875
3
+ 0,1,0.006693977446957433,0.001556396484375,0.199432373046875,0.322418212890625
4
+ 0,2,2.235693497167631,0.322906494140625,0.859954833984375,0.204315185546875
5
+ 0,3,0.15027083228687843,0.017242431640625,0.282135009765625,0.372955322265625
6
+ 0,4,0.07599250744442543,0.010345458984375,0.2462158203125,0.4195556640625
7
+ 0,5,0.00419125804091891,0.00079345703125,0.19976806640625,0.290557861328125
8
+ 0,6,0.10219017933468422,0.010589599609375,0.03662109375,0.11737060546875
9
+ 0,7,5.169036907659574,0.764434814453125,0.988250732421875,0.136383056640625
10
+ 1,0,0.00025813918091444066,6.103515625e-05,0.286376953125,0.53216552734375
11
+ 1,1,2.973878781631356e-05,3.0517578125e-05,0.254669189453125,0.451263427734375
12
+ 1,2,0.00024059577026491752,9.1552734375e-05,0.499359130859375,0.471405029296875
13
+ 1,3,0.0009068534045582055,0.0003662109375,0.36083984375,0.4954833984375
14
+ 1,4,0.003771953800423944,0.001220703125,0.21551513671875,0.45703125
15
+ 1,5,0.0006174362997626304,0.00018310546875,0.195831298828125,0.5029296875
16
+ 1,6,0.000752416023715341,0.000274658203125,0.14984130859375,0.46435546875
17
+ 1,7,0.004184644253655279,0.00128173828125,0.171142578125,0.466583251953125
18
+ 2,0,0.00014806245508225402,3.0517578125e-05,0.3846435546875,0.43756103515625
19
+ 2,1,4.894880476058461e-06,0.0,0.24420166015625,0.59722900390625
20
+ 2,2,2.7085926376457792e-05,0.0,0.2579345703125,0.504119873046875
21
+ 2,3,0.00017150774419860682,6.103515625e-05,0.267059326171875,0.58270263671875
22
+ 2,4,0.00021039024341007462,0.0001220703125,0.279937744140625,0.451690673828125
23
+ 2,5,0.0007992379169081687,0.000152587890625,0.378326416015625,0.468170166015625
24
+ 2,6,0.0008647132117403089,0.00030517578125,0.229736328125,0.674285888671875
25
+ 2,7,1.1404487850086298e-05,0.0,0.234710693359375,0.673553466796875
26
+ 3,0,1.613488984730793e-05,0.0,0.243560791015625,0.269866943359375
27
+ 3,1,8.192873792722821e-07,0.0,0.227264404296875,0.328369140625
28
+ 3,2,1.7713737179292366e-06,0.0,0.204864501953125,0.2083740234375
29
+ 3,3,2.291507462359732e-05,0.0,0.244720458984375,0.380889892578125
30
+ 3,4,0.0001252591091542854,0.0,0.27691650390625,0.379302978515625
31
+ 3,5,5.263272214506287e-06,0.0,0.246490478515625,0.265625
32
+ 3,6,4.800411898031598e-05,0.0,0.29443359375,0.322174072265625
33
+ 3,7,7.388479389192071e-06,0.0,0.200347900390625,0.201141357421875
default-seed0/analysis/figures/figure-data/layers-default-seed0.csv ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ layer,loss_rise,token_accuracy_drop
2
+ 0,13.57202391098599,0.88677978515625
3
+ 1,0.19787378381352028,0.04681396484375
4
+ 2,0.04730297024707397,0.016082763671875
5
+ 3,0.0025761095375855803,0.000274658203125
default-seed0/analysis/figures/figure-data/pattern-default-seed0.json ADDED
@@ -0,0 +1,1279 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "layer": 0,
3
+ "head": 7,
4
+ "tokens": [
5
+ "21",
6
+ "2",
7
+ "15",
8
+ "4",
9
+ "51",
10
+ "35",
11
+ "37",
12
+ "52",
13
+ "31",
14
+ "62",
15
+ "5",
16
+ "38",
17
+ "9",
18
+ "37",
19
+ "4",
20
+ "45",
21
+ "SEP",
22
+ "2",
23
+ "4",
24
+ "4",
25
+ "5",
26
+ "9",
27
+ "15",
28
+ "21",
29
+ "31",
30
+ "35",
31
+ "37",
32
+ "37",
33
+ "38",
34
+ "45",
35
+ "51",
36
+ "52",
37
+ "62"
38
+ ],
39
+ "input_length": 16,
40
+ "pattern": [
41
+ [
42
+ 1.0,
43
+ 0.0,
44
+ 0.0,
45
+ 0.0,
46
+ 0.0,
47
+ 0.0,
48
+ 0.0,
49
+ 0.0,
50
+ 0.0,
51
+ 0.0,
52
+ 0.0,
53
+ 0.0,
54
+ 0.0,
55
+ 0.0,
56
+ 0.0,
57
+ 0.0,
58
+ 0.0,
59
+ 0.0,
60
+ 0.0,
61
+ 0.0,
62
+ 0.0,
63
+ 0.0,
64
+ 0.0,
65
+ 0.0,
66
+ 0.0,
67
+ 0.0,
68
+ 0.0,
69
+ 0.0,
70
+ 0.0,
71
+ 0.0,
72
+ 0.0,
73
+ 0.0,
74
+ 0.0
75
+ ],
76
+ [
77
+ 0.023018356412649155,
78
+ 0.9769816994667053,
79
+ 0.0,
80
+ 0.0,
81
+ 0.0,
82
+ 0.0,
83
+ 0.0,
84
+ 0.0,
85
+ 0.0,
86
+ 0.0,
87
+ 0.0,
88
+ 0.0,
89
+ 0.0,
90
+ 0.0,
91
+ 0.0,
92
+ 0.0,
93
+ 0.0,
94
+ 0.0,
95
+ 0.0,
96
+ 0.0,
97
+ 0.0,
98
+ 0.0,
99
+ 0.0,
100
+ 0.0,
101
+ 0.0,
102
+ 0.0,
103
+ 0.0,
104
+ 0.0,
105
+ 0.0,
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0
110
+ ],
111
+ [
112
+ 0.0045615979470312595,
113
+ 0.002258468186482787,
114
+ 0.9931799173355103,
115
+ 0.0,
116
+ 0.0,
117
+ 0.0,
118
+ 0.0,
119
+ 0.0,
120
+ 0.0,
121
+ 0.0,
122
+ 0.0,
123
+ 0.0,
124
+ 0.0,
125
+ 0.0,
126
+ 0.0,
127
+ 0.0,
128
+ 0.0,
129
+ 0.0,
130
+ 0.0,
131
+ 0.0,
132
+ 0.0,
133
+ 0.0,
134
+ 0.0,
135
+ 0.0,
136
+ 0.0,
137
+ 0.0,
138
+ 0.0,
139
+ 0.0,
140
+ 0.0,
141
+ 0.0,
142
+ 0.0,
143
+ 0.0,
144
+ 0.0
145
+ ],
146
+ [
147
+ 0.03559539467096329,
148
+ 0.06540653854608536,
149
+ 0.011447257362306118,
150
+ 0.8875508904457092,
151
+ 0.0,
152
+ 0.0,
153
+ 0.0,
154
+ 0.0,
155
+ 0.0,
156
+ 0.0,
157
+ 0.0,
158
+ 0.0,
159
+ 0.0,
160
+ 0.0,
161
+ 0.0,
162
+ 0.0,
163
+ 0.0,
164
+ 0.0,
165
+ 0.0,
166
+ 0.0,
167
+ 0.0,
168
+ 0.0,
169
+ 0.0,
170
+ 0.0,
171
+ 0.0,
172
+ 0.0,
173
+ 0.0,
174
+ 0.0,
175
+ 0.0,
176
+ 0.0,
177
+ 0.0,
178
+ 0.0,
179
+ 0.0
180
+ ],
181
+ [
182
+ 0.021527551114559174,
183
+ 0.006806362420320511,
184
+ 0.012866882607340813,
185
+ 0.010566228069365025,
186
+ 0.9482329487800598,
187
+ 0.0,
188
+ 0.0,
189
+ 0.0,
190
+ 0.0,
191
+ 0.0,
192
+ 0.0,
193
+ 0.0,
194
+ 0.0,
195
+ 0.0,
196
+ 0.0,
197
+ 0.0,
198
+ 0.0,
199
+ 0.0,
200
+ 0.0,
201
+ 0.0,
202
+ 0.0,
203
+ 0.0,
204
+ 0.0,
205
+ 0.0,
206
+ 0.0,
207
+ 0.0,
208
+ 0.0,
209
+ 0.0,
210
+ 0.0,
211
+ 0.0,
212
+ 0.0,
213
+ 0.0,
214
+ 0.0
215
+ ],
216
+ [
217
+ 0.007833131588995457,
218
+ 0.0024875393137335777,
219
+ 0.01399846188724041,
220
+ 0.00461894366890192,
221
+ 0.0051492964848876,
222
+ 0.9659126996994019,
223
+ 0.0,
224
+ 0.0,
225
+ 0.0,
226
+ 0.0,
227
+ 0.0,
228
+ 0.0,
229
+ 0.0,
230
+ 0.0,
231
+ 0.0,
232
+ 0.0,
233
+ 0.0,
234
+ 0.0,
235
+ 0.0,
236
+ 0.0,
237
+ 0.0,
238
+ 0.0,
239
+ 0.0,
240
+ 0.0,
241
+ 0.0,
242
+ 0.0,
243
+ 0.0,
244
+ 0.0,
245
+ 0.0,
246
+ 0.0,
247
+ 0.0,
248
+ 0.0,
249
+ 0.0
250
+ ],
251
+ [
252
+ 0.0009257665951736271,
253
+ 0.006245102733373642,
254
+ 0.0074912612326443195,
255
+ 0.008376035839319229,
256
+ 0.006875417195260525,
257
+ 0.055685821920633316,
258
+ 0.9144006371498108,
259
+ 0.0,
260
+ 0.0,
261
+ 0.0,
262
+ 0.0,
263
+ 0.0,
264
+ 0.0,
265
+ 0.0,
266
+ 0.0,
267
+ 0.0,
268
+ 0.0,
269
+ 0.0,
270
+ 0.0,
271
+ 0.0,
272
+ 0.0,
273
+ 0.0,
274
+ 0.0,
275
+ 0.0,
276
+ 0.0,
277
+ 0.0,
278
+ 0.0,
279
+ 0.0,
280
+ 0.0,
281
+ 0.0,
282
+ 0.0,
283
+ 0.0,
284
+ 0.0
285
+ ],
286
+ [
287
+ 0.003037565154954791,
288
+ 0.005996344611048698,
289
+ 0.0092690484598279,
290
+ 0.011296346783638,
291
+ 0.06189500913023949,
292
+ 0.008469926193356514,
293
+ 0.01204671896994114,
294
+ 0.8879890441894531,
295
+ 0.0,
296
+ 0.0,
297
+ 0.0,
298
+ 0.0,
299
+ 0.0,
300
+ 0.0,
301
+ 0.0,
302
+ 0.0,
303
+ 0.0,
304
+ 0.0,
305
+ 0.0,
306
+ 0.0,
307
+ 0.0,
308
+ 0.0,
309
+ 0.0,
310
+ 0.0,
311
+ 0.0,
312
+ 0.0,
313
+ 0.0,
314
+ 0.0,
315
+ 0.0,
316
+ 0.0,
317
+ 0.0,
318
+ 0.0,
319
+ 0.0
320
+ ],
321
+ [
322
+ 0.0025577344931662083,
323
+ 0.0022750389762222767,
324
+ 0.0011446349089965224,
325
+ 0.0034309474285691977,
326
+ 0.006817675661295652,
327
+ 0.004609635099768639,
328
+ 0.002528931712731719,
329
+ 0.009095567278563976,
330
+ 0.9675397276878357,
331
+ 0.0,
332
+ 0.0,
333
+ 0.0,
334
+ 0.0,
335
+ 0.0,
336
+ 0.0,
337
+ 0.0,
338
+ 0.0,
339
+ 0.0,
340
+ 0.0,
341
+ 0.0,
342
+ 0.0,
343
+ 0.0,
344
+ 0.0,
345
+ 0.0,
346
+ 0.0,
347
+ 0.0,
348
+ 0.0,
349
+ 0.0,
350
+ 0.0,
351
+ 0.0,
352
+ 0.0,
353
+ 0.0,
354
+ 0.0
355
+ ],
356
+ [
357
+ 0.026828112080693245,
358
+ 0.011514825746417046,
359
+ 0.030997384339571,
360
+ 0.06020127609372139,
361
+ 0.12983357906341553,
362
+ 0.04715048521757126,
363
+ 0.023005908355116844,
364
+ 0.03150458261370659,
365
+ 0.005654371343553066,
366
+ 0.6333094239234924,
367
+ 0.0,
368
+ 0.0,
369
+ 0.0,
370
+ 0.0,
371
+ 0.0,
372
+ 0.0,
373
+ 0.0,
374
+ 0.0,
375
+ 0.0,
376
+ 0.0,
377
+ 0.0,
378
+ 0.0,
379
+ 0.0,
380
+ 0.0,
381
+ 0.0,
382
+ 0.0,
383
+ 0.0,
384
+ 0.0,
385
+ 0.0,
386
+ 0.0,
387
+ 0.0,
388
+ 0.0,
389
+ 0.0
390
+ ],
391
+ [
392
+ 0.01655244454741478,
393
+ 0.014901078306138515,
394
+ 0.007255906239151955,
395
+ 0.04292556270956993,
396
+ 0.011578263714909554,
397
+ 0.04288385063409805,
398
+ 0.01748061738908291,
399
+ 0.010024511255323887,
400
+ 0.029632017016410828,
401
+ 0.015783263370394707,
402
+ 0.7909825444221497,
403
+ 0.0,
404
+ 0.0,
405
+ 0.0,
406
+ 0.0,
407
+ 0.0,
408
+ 0.0,
409
+ 0.0,
410
+ 0.0,
411
+ 0.0,
412
+ 0.0,
413
+ 0.0,
414
+ 0.0,
415
+ 0.0,
416
+ 0.0,
417
+ 0.0,
418
+ 0.0,
419
+ 0.0,
420
+ 0.0,
421
+ 0.0,
422
+ 0.0,
423
+ 0.0,
424
+ 0.0
425
+ ],
426
+ [
427
+ 0.012589594349265099,
428
+ 0.0015876422403380275,
429
+ 0.0017474794294685125,
430
+ 0.0020861185621470213,
431
+ 0.0043601891957223415,
432
+ 0.0357985682785511,
433
+ 0.02455979399383068,
434
+ 0.001521193073131144,
435
+ 0.009579543955624104,
436
+ 0.011912292800843716,
437
+ 0.009621202014386654,
438
+ 0.8846362829208374,
439
+ 0.0,
440
+ 0.0,
441
+ 0.0,
442
+ 0.0,
443
+ 0.0,
444
+ 0.0,
445
+ 0.0,
446
+ 0.0,
447
+ 0.0,
448
+ 0.0,
449
+ 0.0,
450
+ 0.0,
451
+ 0.0,
452
+ 0.0,
453
+ 0.0,
454
+ 0.0,
455
+ 0.0,
456
+ 0.0,
457
+ 0.0,
458
+ 0.0,
459
+ 0.0
460
+ ],
461
+ [
462
+ 0.008168579079210758,
463
+ 0.003204863518476486,
464
+ 0.015219288878142834,
465
+ 0.003438628278672695,
466
+ 0.0027168861124664545,
467
+ 0.004375458229333162,
468
+ 0.0015034880489110947,
469
+ 0.004062159452587366,
470
+ 0.012900643050670624,
471
+ 0.011661041527986526,
472
+ 0.0040309918113052845,
473
+ 0.006697934120893478,
474
+ 0.9220201969146729,
475
+ 0.0,
476
+ 0.0,
477
+ 0.0,
478
+ 0.0,
479
+ 0.0,
480
+ 0.0,
481
+ 0.0,
482
+ 0.0,
483
+ 0.0,
484
+ 0.0,
485
+ 0.0,
486
+ 0.0,
487
+ 0.0,
488
+ 0.0,
489
+ 0.0,
490
+ 0.0,
491
+ 0.0,
492
+ 0.0,
493
+ 0.0,
494
+ 0.0
495
+ ],
496
+ [
497
+ 0.0005252237897366285,
498
+ 0.0031085822265595198,
499
+ 0.004390801303088665,
500
+ 0.0035702104214578867,
501
+ 0.0035593381617218256,
502
+ 0.030331918969750404,
503
+ 0.4591764509677887,
504
+ 0.004388758447021246,
505
+ 0.0011947763850912452,
506
+ 0.0026341581251472235,
507
+ 0.0035157559905201197,
508
+ 0.019338253885507584,
509
+ 0.0003783679858315736,
510
+ 0.46388736367225647,
511
+ 0.0,
512
+ 0.0,
513
+ 0.0,
514
+ 0.0,
515
+ 0.0,
516
+ 0.0,
517
+ 0.0,
518
+ 0.0,
519
+ 0.0,
520
+ 0.0,
521
+ 0.0,
522
+ 0.0,
523
+ 0.0,
524
+ 0.0,
525
+ 0.0,
526
+ 0.0,
527
+ 0.0,
528
+ 0.0,
529
+ 0.0
530
+ ],
531
+ [
532
+ 0.024496333673596382,
533
+ 0.02895524352788925,
534
+ 0.007409717887639999,
535
+ 0.3809566795825958,
536
+ 0.025879722088575363,
537
+ 0.01931842789053917,
538
+ 0.015419346280395985,
539
+ 0.021070702001452446,
540
+ 0.019366836175322533,
541
+ 0.024991635233163834,
542
+ 0.023570610210299492,
543
+ 0.012358313426375389,
544
+ 0.007372889667749405,
545
+ 0.015129401348531246,
546
+ 0.37370413541793823,
547
+ 0.0,
548
+ 0.0,
549
+ 0.0,
550
+ 0.0,
551
+ 0.0,
552
+ 0.0,
553
+ 0.0,
554
+ 0.0,
555
+ 0.0,
556
+ 0.0,
557
+ 0.0,
558
+ 0.0,
559
+ 0.0,
560
+ 0.0,
561
+ 0.0,
562
+ 0.0,
563
+ 0.0,
564
+ 0.0
565
+ ],
566
+ [
567
+ 0.012154011987149715,
568
+ 0.006682737730443478,
569
+ 0.007665995042771101,
570
+ 0.028195885941386223,
571
+ 0.011137477122247219,
572
+ 0.013211286626756191,
573
+ 0.003523231018334627,
574
+ 0.02074238285422325,
575
+ 0.016506455838680267,
576
+ 0.015658063814044,
577
+ 0.0018506235210224986,
578
+ 0.0026229280047118664,
579
+ 0.011027158237993717,
580
+ 0.0032886748667806387,
581
+ 0.02927999757230282,
582
+ 0.8164531588554382,
583
+ 0.0,
584
+ 0.0,
585
+ 0.0,
586
+ 0.0,
587
+ 0.0,
588
+ 0.0,
589
+ 0.0,
590
+ 0.0,
591
+ 0.0,
592
+ 0.0,
593
+ 0.0,
594
+ 0.0,
595
+ 0.0,
596
+ 0.0,
597
+ 0.0,
598
+ 0.0,
599
+ 0.0
600
+ ],
601
+ [
602
+ 0.02281976491212845,
603
+ 0.10939843952655792,
604
+ 0.01768209971487522,
605
+ 0.0379922017455101,
606
+ 0.05958489701151848,
607
+ 0.037575434893369675,
608
+ 0.05534495413303375,
609
+ 0.07548356056213379,
610
+ 0.085883229970932,
611
+ 0.03505440801382065,
612
+ 0.03271886333823204,
613
+ 0.06485980749130249,
614
+ 0.03420936316251755,
615
+ 0.053685955703258514,
616
+ 0.03465070575475693,
617
+ 0.03298822045326233,
618
+ 0.21006812155246735,
619
+ 0.0,
620
+ 0.0,
621
+ 0.0,
622
+ 0.0,
623
+ 0.0,
624
+ 0.0,
625
+ 0.0,
626
+ 0.0,
627
+ 0.0,
628
+ 0.0,
629
+ 0.0,
630
+ 0.0,
631
+ 0.0,
632
+ 0.0,
633
+ 0.0,
634
+ 0.0
635
+ ],
636
+ [
637
+ 0.007137803360819817,
638
+ 0.43834009766578674,
639
+ 0.007111180108040571,
640
+ 0.02079029753804207,
641
+ 0.007220623083412647,
642
+ 0.006627264432609081,
643
+ 0.01151738129556179,
644
+ 0.007246346678584814,
645
+ 0.006320110987871885,
646
+ 0.005185392685234547,
647
+ 0.009528160095214844,
648
+ 0.007866409607231617,
649
+ 0.00646289624273777,
650
+ 0.011828177608549595,
651
+ 0.019085045903921127,
652
+ 0.00794859230518341,
653
+ 0.025521352887153625,
654
+ 0.3942629396915436,
655
+ 0.0,
656
+ 0.0,
657
+ 0.0,
658
+ 0.0,
659
+ 0.0,
660
+ 0.0,
661
+ 0.0,
662
+ 0.0,
663
+ 0.0,
664
+ 0.0,
665
+ 0.0,
666
+ 0.0,
667
+ 0.0,
668
+ 0.0,
669
+ 0.0
670
+ ],
671
+ [
672
+ 0.009250459261238575,
673
+ 0.031928885728120804,
674
+ 0.003431153018027544,
675
+ 0.2892793118953705,
676
+ 0.011380745097994804,
677
+ 0.007772959768772125,
678
+ 0.008837981149554253,
679
+ 0.0097163962200284,
680
+ 0.009380745701491833,
681
+ 0.012278328649699688,
682
+ 0.016086222603917122,
683
+ 0.007512957323342562,
684
+ 0.0047597577795386314,
685
+ 0.008751295506954193,
686
+ 0.2750386893749237,
687
+ 0.02218909002840519,
688
+ 0.012468640692532063,
689
+ 0.02915336936712265,
690
+ 0.23078303039073944,
691
+ 0.0,
692
+ 0.0,
693
+ 0.0,
694
+ 0.0,
695
+ 0.0,
696
+ 0.0,
697
+ 0.0,
698
+ 0.0,
699
+ 0.0,
700
+ 0.0,
701
+ 0.0,
702
+ 0.0,
703
+ 0.0,
704
+ 0.0
705
+ ],
706
+ [
707
+ 0.008872050791978836,
708
+ 0.019518502056598663,
709
+ 0.003353310748934746,
710
+ 0.24201583862304688,
711
+ 0.010490399785339832,
712
+ 0.006393995601683855,
713
+ 0.006850871257483959,
714
+ 0.008608594536781311,
715
+ 0.004944298416376114,
716
+ 0.010344484820961952,
717
+ 0.015715332701802254,
718
+ 0.004991887602955103,
719
+ 0.0037644386757165194,
720
+ 0.006783084012567997,
721
+ 0.2317233681678772,
722
+ 0.01787676475942135,
723
+ 0.008699280209839344,
724
+ 0.01878373883664608,
725
+ 0.1803499013185501,
726
+ 0.18991988897323608,
727
+ 0.0,
728
+ 0.0,
729
+ 0.0,
730
+ 0.0,
731
+ 0.0,
732
+ 0.0,
733
+ 0.0,
734
+ 0.0,
735
+ 0.0,
736
+ 0.0,
737
+ 0.0,
738
+ 0.0,
739
+ 0.0
740
+ ],
741
+ [
742
+ 0.006691065616905689,
743
+ 0.01196309458464384,
744
+ 0.0037312544882297516,
745
+ 0.03853219747543335,
746
+ 0.004294651560485363,
747
+ 0.018225621432065964,
748
+ 0.009463095106184483,
749
+ 0.0042219506576657295,
750
+ 0.010560271330177784,
751
+ 0.009044119156897068,
752
+ 0.3927554190158844,
753
+ 0.022279053926467896,
754
+ 0.009899655357003212,
755
+ 0.009482008405029774,
756
+ 0.039267648011446,
757
+ 0.005388212855905294,
758
+ 0.008469431661069393,
759
+ 0.012121939100325108,
760
+ 0.03598523139953613,
761
+ 0.037726208567619324,
762
+ 0.30989784002304077,
763
+ 0.0,
764
+ 0.0,
765
+ 0.0,
766
+ 0.0,
767
+ 0.0,
768
+ 0.0,
769
+ 0.0,
770
+ 0.0,
771
+ 0.0,
772
+ 0.0,
773
+ 0.0,
774
+ 0.0
775
+ ],
776
+ [
777
+ 0.005792838055640459,
778
+ 0.003233768744394183,
779
+ 0.010633621364831924,
780
+ 0.0039171804673969746,
781
+ 0.002460538176819682,
782
+ 0.0030334817711263895,
783
+ 0.0013890591217204928,
784
+ 0.0034110224805772305,
785
+ 0.006884894799441099,
786
+ 0.009242535568773746,
787
+ 0.004236869513988495,
788
+ 0.005316397175192833,
789
+ 0.5288781523704529,
790
+ 0.001391792669892311,
791
+ 0.0040800850838422775,
792
+ 0.006198899354785681,
793
+ 0.004258950240910053,
794
+ 0.0034238765947520733,
795
+ 0.004500585608184338,
796
+ 0.0042763021774590015,
797
+ 0.004853626247495413,
798
+ 0.37858548760414124,
799
+ 0.0,
800
+ 0.0,
801
+ 0.0,
802
+ 0.0,
803
+ 0.0,
804
+ 0.0,
805
+ 0.0,
806
+ 0.0,
807
+ 0.0,
808
+ 0.0,
809
+ 0.0
810
+ ],
811
+ [
812
+ 0.004203260876238346,
813
+ 0.002793460851535201,
814
+ 0.5633641481399536,
815
+ 0.001922506489790976,
816
+ 0.0030416715890169144,
817
+ 0.004999521188437939,
818
+ 0.0027974105905741453,
819
+ 0.003516203723847866,
820
+ 0.001284533180296421,
821
+ 0.002191777341067791,
822
+ 0.0018217077013105154,
823
+ 0.0010901825735345483,
824
+ 0.009667942300438881,
825
+ 0.002757111331447959,
826
+ 0.0019903217907994986,
827
+ 0.0031780314166098833,
828
+ 0.00159900460857898,
829
+ 0.003045066026970744,
830
+ 0.0019833394326269627,
831
+ 0.0022355637047439814,
832
+ 0.0017322205239906907,
833
+ 0.008563856594264507,
834
+ 0.37022119760513306,
835
+ 0.0,
836
+ 0.0,
837
+ 0.0,
838
+ 0.0,
839
+ 0.0,
840
+ 0.0,
841
+ 0.0,
842
+ 0.0,
843
+ 0.0,
844
+ 0.0
845
+ ],
846
+ [
847
+ 0.5568448305130005,
848
+ 0.0028602159582078457,
849
+ 0.008501074276864529,
850
+ 0.002472684718668461,
851
+ 0.003511592745780945,
852
+ 0.002653622068464756,
853
+ 0.0005315084126777947,
854
+ 0.0015206747921183705,
855
+ 0.0015745942946523428,
856
+ 0.0022094366140663624,
857
+ 0.0016722797881811857,
858
+ 0.0049180155619978905,
859
+ 0.0031623856630176306,
860
+ 0.0005430039600469172,
861
+ 0.002721830504015088,
862
+ 0.0032658791169524193,
863
+ 0.002483119023963809,
864
+ 0.0031409449875354767,
865
+ 0.0026567354798316956,
866
+ 0.002672189613804221,
867
+ 0.0018375213257968426,
868
+ 0.0033643117640167475,
869
+ 0.007786066737025976,
870
+ 0.3770953416824341,
871
+ 0.0,
872
+ 0.0,
873
+ 0.0,
874
+ 0.0,
875
+ 0.0,
876
+ 0.0,
877
+ 0.0,
878
+ 0.0,
879
+ 0.0
880
+ ],
881
+ [
882
+ 0.0024161776527762413,
883
+ 0.0036724423989653587,
884
+ 0.0013728196499869227,
885
+ 0.004530532285571098,
886
+ 0.007108110003173351,
887
+ 0.004414150957018137,
888
+ 0.00348899164237082,
889
+ 0.00970153696835041,
890
+ 0.5030159950256348,
891
+ 0.003252868540585041,
892
+ 0.0052609797567129135,
893
+ 0.007975067012012005,
894
+ 0.01143447682261467,
895
+ 0.0035413485020399094,
896
+ 0.00464472034946084,
897
+ 0.004634521901607513,
898
+ 0.008493144065141678,
899
+ 0.00388615345582366,
900
+ 0.005385011434555054,
901
+ 0.004555848427116871,
902
+ 0.005924721714109182,
903
+ 0.011706513352692127,
904
+ 0.0016336682019755244,
905
+ 0.0026681125164031982,
906
+ 0.3752821385860443,
907
+ 0.0,
908
+ 0.0,
909
+ 0.0,
910
+ 0.0,
911
+ 0.0,
912
+ 0.0,
913
+ 0.0,
914
+ 0.0
915
+ ],
916
+ [
917
+ 0.006361646112054586,
918
+ 0.0030678503680974245,
919
+ 0.009892310947179794,
920
+ 0.0061208149418234825,
921
+ 0.0044769528321921825,
922
+ 0.4815216064453125,
923
+ 0.01844535768032074,
924
+ 0.0061722612008452415,
925
+ 0.004645282402634621,
926
+ 0.010261544957756996,
927
+ 0.009851057082414627,
928
+ 0.02187376469373703,
929
+ 0.0017814149614423513,
930
+ 0.018494635820388794,
931
+ 0.006336814258247614,
932
+ 0.005621344316750765,
933
+ 0.005525233689695597,
934
+ 0.0028296757955104113,
935
+ 0.0059134080074727535,
936
+ 0.006024481263011694,
937
+ 0.009826972149312496,
938
+ 0.0022001052275300026,
939
+ 0.009621761739253998,
940
+ 0.006355932913720608,
941
+ 0.004840776789933443,
942
+ 0.3319369852542877,
943
+ 0.0,
944
+ 0.0,
945
+ 0.0,
946
+ 0.0,
947
+ 0.0,
948
+ 0.0,
949
+ 0.0
950
+ ],
951
+ [
952
+ 0.000425170233938843,
953
+ 0.004094802308827639,
954
+ 0.0031548335682600737,
955
+ 0.0058814301155507565,
956
+ 0.0034635127522051334,
957
+ 0.02076704613864422,
958
+ 0.32310518622398376,
959
+ 0.004715237766504288,
960
+ 0.0009895049734041095,
961
+ 0.0027204174548387527,
962
+ 0.004136086907237768,
963
+ 0.01667216047644615,
964
+ 0.0004302041488699615,
965
+ 0.32536986470222473,
966
+ 0.005149839445948601,
967
+ 0.001225978252477944,
968
+ 0.003524137893691659,
969
+ 0.0038776169531047344,
970
+ 0.005054942797869444,
971
+ 0.004758174065500498,
972
+ 0.0035773315466940403,
973
+ 0.0005115001695230603,
974
+ 0.0031328077893704176,
975
+ 0.00047373436973430216,
976
+ 0.0012087365612387657,
977
+ 0.015949063003063202,
978
+ 0.23563075065612793,
979
+ 0.0,
980
+ 0.0,
981
+ 0.0,
982
+ 0.0,
983
+ 0.0,
984
+ 0.0
985
+ ],
986
+ [
987
+ 0.0003504737396724522,
988
+ 0.0029342903289943933,
989
+ 0.0024841963313519955,
990
+ 0.004752310924232006,
991
+ 0.0029739539604634047,
992
+ 0.01688055507838726,
993
+ 0.2607404887676239,
994
+ 0.004170580301433802,
995
+ 0.0009409718331880867,
996
+ 0.0024588305968791246,
997
+ 0.0034691242035478354,
998
+ 0.013757032342255116,
999
+ 0.0003214033495169133,
1000
+ 0.26270395517349243,
1001
+ 0.004193123895674944,
1002
+ 0.0010010853875428438,
1003
+ 0.0028727527242153883,
1004
+ 0.002769015496596694,
1005
+ 0.004081565886735916,
1006
+ 0.003849680069833994,
1007
+ 0.0029822972137480974,
1008
+ 0.00038473180029541254,
1009
+ 0.002470763400197029,
1010
+ 0.0003893456014338881,
1011
+ 0.0011314790463075042,
1012
+ 0.01302724052220583,
1013
+ 0.18942537903785706,
1014
+ 0.19248336553573608,
1015
+ 0.0,
1016
+ 0.0,
1017
+ 0.0,
1018
+ 0.0,
1019
+ 0.0
1020
+ ],
1021
+ [
1022
+ 0.007859006524085999,
1023
+ 0.0016575781628489494,
1024
+ 0.0011552342912182212,
1025
+ 0.002312729600816965,
1026
+ 0.003259645774960518,
1027
+ 0.017806673422455788,
1028
+ 0.01716545969247818,
1029
+ 0.001212490489706397,
1030
+ 0.005413018632680178,
1031
+ 0.00896664708852768,
1032
+ 0.00777075020596385,
1033
+ 0.46369069814682007,
1034
+ 0.00310750724747777,
1035
+ 0.017114248126745224,
1036
+ 0.0020787613466382027,
1037
+ 0.0011019748635590076,
1038
+ 0.004998395219445229,
1039
+ 0.001606945414096117,
1040
+ 0.0024974036496132612,
1041
+ 0.001989792799577117,
1042
+ 0.007642480544745922,
1043
+ 0.003726074006408453,
1044
+ 0.0012539394665509462,
1045
+ 0.007231967989355326,
1046
+ 0.005606942344456911,
1047
+ 0.015894239768385887,
1048
+ 0.0155118927359581,
1049
+ 0.015764642506837845,
1050
+ 0.35460278391838074,
1051
+ 0.0,
1052
+ 0.0,
1053
+ 0.0,
1054
+ 0.0
1055
+ ],
1056
+ [
1057
+ 0.006357785314321518,
1058
+ 0.006078907288610935,
1059
+ 0.0043388856574893,
1060
+ 0.02823742851614952,
1061
+ 0.00836637057363987,
1062
+ 0.007889148779213428,
1063
+ 0.002819335088133812,
1064
+ 0.015533188357949257,
1065
+ 0.009232904762029648,
1066
+ 0.013993765227496624,
1067
+ 0.0017917501972988248,
1068
+ 0.002151534194126725,
1069
+ 0.0071965414099395275,
1070
+ 0.0026568507310003042,
1071
+ 0.028629090636968613,
1072
+ 0.41004717350006104,
1073
+ 0.005909791216254234,
1074
+ 0.006158337462693453,
1075
+ 0.026087140664458275,
1076
+ 0.02569008246064186,
1077
+ 0.0021707997657358646,
1078
+ 0.006381053477525711,
1079
+ 0.0049550761468708515,
1080
+ 0.006111223250627518,
1081
+ 0.008736412972211838,
1082
+ 0.00805989932268858,
1083
+ 0.0031875991262495518,
1084
+ 0.003034937893971801,
1085
+ 0.0023294412530958652,
1086
+ 0.3358675241470337,
1087
+ 0.0,
1088
+ 0.0,
1089
+ 0.0
1090
+ ],
1091
+ [
1092
+ 0.013543812558054924,
1093
+ 0.004420229233801365,
1094
+ 0.007792192045599222,
1095
+ 0.005869054701179266,
1096
+ 0.4126547873020172,
1097
+ 0.006657087709754705,
1098
+ 0.008062448352575302,
1099
+ 0.03802299126982689,
1100
+ 0.013350061140954494,
1101
+ 0.03719658404588699,
1102
+ 0.001816951553337276,
1103
+ 0.0070841689594089985,
1104
+ 0.001960590248927474,
1105
+ 0.0080854007974267,
1106
+ 0.005973675288259983,
1107
+ 0.011459790170192719,
1108
+ 0.014922630973160267,
1109
+ 0.003987274132668972,
1110
+ 0.005299489013850689,
1111
+ 0.006025535985827446,
1112
+ 0.0016457047313451767,
1113
+ 0.0024295023176819086,
1114
+ 0.007024877704679966,
1115
+ 0.01224868930876255,
1116
+ 0.0130916191264987,
1117
+ 0.006404382176697254,
1118
+ 0.008179416880011559,
1119
+ 0.008356597274541855,
1120
+ 0.007054286077618599,
1121
+ 0.011059602722525597,
1122
+ 0.30832064151763916,
1123
+ 0.0,
1124
+ 0.0
1125
+ ],
1126
+ [
1127
+ 0.004401464015245438,
1128
+ 0.005623731296509504,
1129
+ 0.011734258383512497,
1130
+ 0.007039395160973072,
1131
+ 0.05771010369062424,
1132
+ 0.010057630948722363,
1133
+ 0.010448037646710873,
1134
+ 0.3336871564388275,
1135
+ 0.010435361415147781,
1136
+ 0.02165687456727028,
1137
+ 0.002097832504659891,
1138
+ 0.0021464028395712376,
1139
+ 0.006190245505422354,
1140
+ 0.010710052214562893,
1141
+ 0.007371100597083569,
1142
+ 0.027940666303038597,
1143
+ 0.013196882791817188,
1144
+ 0.004852048121392727,
1145
+ 0.006140248384326696,
1146
+ 0.007695483043789864,
1147
+ 0.002024211920797825,
1148
+ 0.006342147011309862,
1149
+ 0.010344588197767735,
1150
+ 0.004595689009875059,
1151
+ 0.00991485733538866,
1152
+ 0.009901775978505611,
1153
+ 0.011316884309053421,
1154
+ 0.01131411176174879,
1155
+ 0.0023516900837421417,
1156
+ 0.027719907462596893,
1157
+ 0.05792964622378349,
1158
+ 0.2851095199584961,
1159
+ 0.0
1160
+ ],
1161
+ [
1162
+ 0.011347686871886253,
1163
+ 0.0054034581407904625,
1164
+ 0.008515670895576477,
1165
+ 0.028890598565340042,
1166
+ 0.05831068381667137,
1167
+ 0.016622332856059074,
1168
+ 0.009163033217191696,
1169
+ 0.014324112795293331,
1170
+ 0.002134349662810564,
1171
+ 0.22702908515930176,
1172
+ 0.007055801805108786,
1173
+ 0.02273165062069893,
1174
+ 0.030010566115379333,
1175
+ 0.009357254020869732,
1176
+ 0.02768339030444622,
1177
+ 0.019783705472946167,
1178
+ 0.012823608703911304,
1179
+ 0.004220066126435995,
1180
+ 0.02361401356756687,
1181
+ 0.025145919993519783,
1182
+ 0.006622320506721735,
1183
+ 0.028714947402477264,
1184
+ 0.007441896945238113,
1185
+ 0.010096932761371136,
1186
+ 0.002382578095421195,
1187
+ 0.015767429023981094,
1188
+ 0.009112813510000706,
1189
+ 0.009395239874720573,
1190
+ 0.02087339758872986,
1191
+ 0.02110687829554081,
1192
+ 0.04987165331840515,
1193
+ 0.022752759978175163,
1194
+ 0.2316940724849701
1195
+ ]
1196
+ ],
1197
+ "expected": [
1198
+ [
1199
+ 17,
1200
+ 1
1201
+ ],
1202
+ [
1203
+ 18,
1204
+ 3
1205
+ ],
1206
+ [
1207
+ 18,
1208
+ 14
1209
+ ],
1210
+ [
1211
+ 19,
1212
+ 3
1213
+ ],
1214
+ [
1215
+ 19,
1216
+ 14
1217
+ ],
1218
+ [
1219
+ 20,
1220
+ 10
1221
+ ],
1222
+ [
1223
+ 21,
1224
+ 12
1225
+ ],
1226
+ [
1227
+ 22,
1228
+ 2
1229
+ ],
1230
+ [
1231
+ 23,
1232
+ 0
1233
+ ],
1234
+ [
1235
+ 24,
1236
+ 8
1237
+ ],
1238
+ [
1239
+ 25,
1240
+ 5
1241
+ ],
1242
+ [
1243
+ 26,
1244
+ 6
1245
+ ],
1246
+ [
1247
+ 26,
1248
+ 13
1249
+ ],
1250
+ [
1251
+ 27,
1252
+ 6
1253
+ ],
1254
+ [
1255
+ 27,
1256
+ 13
1257
+ ],
1258
+ [
1259
+ 28,
1260
+ 11
1261
+ ],
1262
+ [
1263
+ 29,
1264
+ 15
1265
+ ],
1266
+ [
1267
+ 30,
1268
+ 4
1269
+ ],
1270
+ [
1271
+ 31,
1272
+ 7
1273
+ ],
1274
+ [
1275
+ 32,
1276
+ 9
1277
+ ]
1278
+ ]
1279
+ }
default-seed0/analysis/figures/head_ablation.png ADDED
default-seed0/analysis/figures/lookup_current.png ADDED
default-seed0/analysis/figures/lookup_next.png ADDED
default-seed0/analysis/figures/lookup_next_pattern.png ADDED
default-seed0/analysis/figures/results.json ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "runs/default-seed0",
3
+ "code": {
4
+ "commit": "42933bd7a4e2e13fd737d0d48d489f72fbeed8ac",
5
+ "dirty": true,
6
+ "snapshot": "source-snapshot.tar.gz"
7
+ },
8
+ "environment": {
9
+ "python": "3.12.3",
10
+ "torch": "2.15.0+cu132",
11
+ "device": "cuda",
12
+ "precision": "torch.float32",
13
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
14
+ "gpu": "NVIDIA GeForce RTX 3060",
15
+ "cuda": "13.2"
16
+ },
17
+ "test_size": 2048,
18
+ "chance_token_accuracy": 0.015625,
19
+ "trained": {
20
+ "loss": 4.249109224474523e-06,
21
+ "token_accuracy": 1.0,
22
+ "exact_match": 1.0
23
+ },
24
+ "untrained": {
25
+ "loss": 4.214900970458984,
26
+ "token_accuracy": 0.01812744140625,
27
+ "exact_match": 0.0
28
+ },
29
+ "head_ablation": {
30
+ "clean_loss": 4.249109224474523e-06,
31
+ "identity_patch_max_logit_change": 0.0,
32
+ "loss_rise": [
33
+ [
34
+ 0.09456062089066108,
35
+ 0.006693977446957433,
36
+ 2.235693497167631,
37
+ 0.15027083228687843,
38
+ 0.07599250744442543,
39
+ 0.00419125804091891,
40
+ 0.10219017933468422,
41
+ 5.169036907659574
42
+ ],
43
+ [
44
+ 0.00025813918091444066,
45
+ 2.973878781631356e-05,
46
+ 0.00024059577026491752,
47
+ 0.0009068534045582055,
48
+ 0.003771953800423944,
49
+ 0.0006174362997626304,
50
+ 0.000752416023715341,
51
+ 0.004184644253655279
52
+ ],
53
+ [
54
+ 0.00014806245508225402,
55
+ 4.894880476058461e-06,
56
+ 2.7085926376457792e-05,
57
+ 0.00017150774419860682,
58
+ 0.00021039024341007462,
59
+ 0.0007992379169081687,
60
+ 0.0008647132117403089,
61
+ 1.1404487850086298e-05
62
+ ],
63
+ [
64
+ 1.613488984730793e-05,
65
+ 8.192873792722821e-07,
66
+ 1.7713737179292366e-06,
67
+ 2.291507462359732e-05,
68
+ 0.0001252591091542854,
69
+ 5.263272214506287e-06,
70
+ 4.800411898031598e-05,
71
+ 7.388479389192071e-06
72
+ ]
73
+ ],
74
+ "token_accuracy_drop": [
75
+ [
76
+ 0.021453857421875,
77
+ 0.001556396484375,
78
+ 0.322906494140625,
79
+ 0.017242431640625,
80
+ 0.010345458984375,
81
+ 0.00079345703125,
82
+ 0.010589599609375,
83
+ 0.764434814453125
84
+ ],
85
+ [
86
+ 6.103515625e-05,
87
+ 3.0517578125e-05,
88
+ 9.1552734375e-05,
89
+ 0.0003662109375,
90
+ 0.001220703125,
91
+ 0.00018310546875,
92
+ 0.000274658203125,
93
+ 0.00128173828125
94
+ ],
95
+ [
96
+ 3.0517578125e-05,
97
+ 0.0,
98
+ 0.0,
99
+ 6.103515625e-05,
100
+ 0.0001220703125,
101
+ 0.000152587890625,
102
+ 0.00030517578125,
103
+ 0.0
104
+ ],
105
+ [
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0,
110
+ 0.0,
111
+ 0.0,
112
+ 0.0,
113
+ 0.0
114
+ ]
115
+ ],
116
+ "layer_loss_rise": [
117
+ 13.57202391098599,
118
+ 0.19787378381352028,
119
+ 0.04730297024707397,
120
+ 0.0025761095375855803
121
+ ],
122
+ "layer_token_accuracy_drop": [
123
+ 0.88677978515625,
124
+ 0.04681396484375,
125
+ 0.016082763671875,
126
+ 0.000274658203125
127
+ ]
128
+ },
129
+ "lookup_next": {
130
+ "score": [
131
+ [
132
+ 0.196746826171875,
133
+ 0.322418212890625,
134
+ 0.204315185546875,
135
+ 0.372955322265625,
136
+ 0.4195556640625,
137
+ 0.290557861328125,
138
+ 0.11737060546875,
139
+ 0.136383056640625
140
+ ],
141
+ [
142
+ 0.53216552734375,
143
+ 0.451263427734375,
144
+ 0.471405029296875,
145
+ 0.4954833984375,
146
+ 0.45703125,
147
+ 0.5029296875,
148
+ 0.46435546875,
149
+ 0.466583251953125
150
+ ],
151
+ [
152
+ 0.43756103515625,
153
+ 0.59722900390625,
154
+ 0.504119873046875,
155
+ 0.58270263671875,
156
+ 0.451690673828125,
157
+ 0.468170166015625,
158
+ 0.674285888671875,
159
+ 0.673553466796875
160
+ ],
161
+ [
162
+ 0.269866943359375,
163
+ 0.328369140625,
164
+ 0.2083740234375,
165
+ 0.380889892578125,
166
+ 0.379302978515625,
167
+ 0.265625,
168
+ 0.322174072265625,
169
+ 0.201141357421875
170
+ ]
171
+ ],
172
+ "chance": 0.07715225219726562
173
+ },
174
+ "lookup_current": {
175
+ "score": [
176
+ [
177
+ 0.58856201171875,
178
+ 0.199432373046875,
179
+ 0.859954833984375,
180
+ 0.282135009765625,
181
+ 0.2462158203125,
182
+ 0.19976806640625,
183
+ 0.03662109375,
184
+ 0.988250732421875
185
+ ],
186
+ [
187
+ 0.286376953125,
188
+ 0.254669189453125,
189
+ 0.499359130859375,
190
+ 0.36083984375,
191
+ 0.21551513671875,
192
+ 0.195831298828125,
193
+ 0.14984130859375,
194
+ 0.171142578125
195
+ ],
196
+ [
197
+ 0.3846435546875,
198
+ 0.24420166015625,
199
+ 0.2579345703125,
200
+ 0.267059326171875,
201
+ 0.279937744140625,
202
+ 0.378326416015625,
203
+ 0.229736328125,
204
+ 0.234710693359375
205
+ ],
206
+ [
207
+ 0.243560791015625,
208
+ 0.227264404296875,
209
+ 0.204864501953125,
210
+ 0.244720458984375,
211
+ 0.27691650390625,
212
+ 0.246490478515625,
213
+ 0.29443359375,
214
+ 0.200347900390625
215
+ ]
216
+ ],
217
+ "chance": 0.07715225219726562
218
+ }
219
+ }
default-seed0/analysis/figures/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:987265c7625afba46c48ceb93bc7be63100d3e67a00f8fed1385a794487a09ab
3
+ size 347464
default-seed0/analysis/final/critical_pattern.png ADDED
default-seed0/analysis/final/head_ablation.png ADDED
default-seed0/analysis/final/lookup_current.png ADDED
default-seed0/analysis/final/lookup_next.png ADDED
default-seed0/analysis/final/lookup_next_pattern.png ADDED
default-seed0/analysis/final/results.json ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "runs/default-seed0",
3
+ "code": {
4
+ "commit": "19276f75392bee9d5f6f61820395002a0746e349",
5
+ "dirty": true,
6
+ "snapshot": "source-snapshot.tar.gz"
7
+ },
8
+ "environment": {
9
+ "python": "3.12.3",
10
+ "torch": "2.15.0+cu132",
11
+ "device": "cuda",
12
+ "precision": "torch.float32",
13
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
14
+ "gpu": "NVIDIA GeForce RTX 3060",
15
+ "cuda": "13.2"
16
+ },
17
+ "test_size": 2048,
18
+ "chance_token_accuracy": 0.015625,
19
+ "trained": {
20
+ "loss": 4.249109224474523e-06,
21
+ "token_accuracy": 1.0,
22
+ "exact_match": 1.0
23
+ },
24
+ "untrained": {
25
+ "loss": 4.214900970458984,
26
+ "token_accuracy": 0.01812744140625,
27
+ "exact_match": 0.0
28
+ },
29
+ "head_ablation": {
30
+ "clean_loss": 4.249109224474523e-06,
31
+ "identity_patch_max_logit_change": 0.0,
32
+ "loss_rise": [
33
+ [
34
+ 0.09456062089066108,
35
+ 0.006693977446957433,
36
+ 2.235693497167631,
37
+ 0.15027083228687843,
38
+ 0.07599250744442543,
39
+ 0.00419125804091891,
40
+ 0.10219017933468422,
41
+ 5.169036907659574
42
+ ],
43
+ [
44
+ 0.00025813918091444066,
45
+ 2.973878781631356e-05,
46
+ 0.00024059577026491752,
47
+ 0.0009068534045582055,
48
+ 0.003771953800423944,
49
+ 0.0006174362997626304,
50
+ 0.000752416023715341,
51
+ 0.004184644253655279
52
+ ],
53
+ [
54
+ 0.00014806245508225402,
55
+ 4.894880476058461e-06,
56
+ 2.7085926376457792e-05,
57
+ 0.00017150774419860682,
58
+ 0.00021039024341007462,
59
+ 0.0007992379169081687,
60
+ 0.0008647132117403089,
61
+ 1.1404487850086298e-05
62
+ ],
63
+ [
64
+ 1.613488984730793e-05,
65
+ 8.192873792722821e-07,
66
+ 1.7713737179292366e-06,
67
+ 2.291507462359732e-05,
68
+ 0.0001252591091542854,
69
+ 5.263272214506287e-06,
70
+ 4.800411898031598e-05,
71
+ 7.388479389192071e-06
72
+ ]
73
+ ],
74
+ "token_accuracy_drop": [
75
+ [
76
+ 0.021453857421875,
77
+ 0.001556396484375,
78
+ 0.322906494140625,
79
+ 0.017242431640625,
80
+ 0.010345458984375,
81
+ 0.00079345703125,
82
+ 0.010589599609375,
83
+ 0.764434814453125
84
+ ],
85
+ [
86
+ 6.103515625e-05,
87
+ 3.0517578125e-05,
88
+ 9.1552734375e-05,
89
+ 0.0003662109375,
90
+ 0.001220703125,
91
+ 0.00018310546875,
92
+ 0.000274658203125,
93
+ 0.00128173828125
94
+ ],
95
+ [
96
+ 3.0517578125e-05,
97
+ 0.0,
98
+ 0.0,
99
+ 6.103515625e-05,
100
+ 0.0001220703125,
101
+ 0.000152587890625,
102
+ 0.00030517578125,
103
+ 0.0
104
+ ],
105
+ [
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0,
110
+ 0.0,
111
+ 0.0,
112
+ 0.0,
113
+ 0.0
114
+ ]
115
+ ],
116
+ "layer_loss_rise": [
117
+ 13.57202391098599,
118
+ 0.19787378381352028,
119
+ 0.04730297024707397,
120
+ 0.0025761095375855803
121
+ ],
122
+ "layer_token_accuracy_drop": [
123
+ 0.88677978515625,
124
+ 0.04681396484375,
125
+ 0.016082763671875,
126
+ 0.000274658203125
127
+ ]
128
+ },
129
+ "lookup_next": {
130
+ "score": [
131
+ [
132
+ 0.196746826171875,
133
+ 0.322418212890625,
134
+ 0.204315185546875,
135
+ 0.372955322265625,
136
+ 0.4195556640625,
137
+ 0.290557861328125,
138
+ 0.11737060546875,
139
+ 0.136383056640625
140
+ ],
141
+ [
142
+ 0.53216552734375,
143
+ 0.451263427734375,
144
+ 0.471405029296875,
145
+ 0.4954833984375,
146
+ 0.45703125,
147
+ 0.5029296875,
148
+ 0.46435546875,
149
+ 0.466583251953125
150
+ ],
151
+ [
152
+ 0.43756103515625,
153
+ 0.59722900390625,
154
+ 0.504119873046875,
155
+ 0.58270263671875,
156
+ 0.451690673828125,
157
+ 0.468170166015625,
158
+ 0.674285888671875,
159
+ 0.673553466796875
160
+ ],
161
+ [
162
+ 0.269866943359375,
163
+ 0.328369140625,
164
+ 0.2083740234375,
165
+ 0.380889892578125,
166
+ 0.379302978515625,
167
+ 0.265625,
168
+ 0.322174072265625,
169
+ 0.201141357421875
170
+ ]
171
+ ],
172
+ "chance": 0.07715225219726562
173
+ },
174
+ "lookup_current": {
175
+ "score": [
176
+ [
177
+ 0.58856201171875,
178
+ 0.199432373046875,
179
+ 0.859954833984375,
180
+ 0.282135009765625,
181
+ 0.2462158203125,
182
+ 0.19976806640625,
183
+ 0.03662109375,
184
+ 0.988250732421875
185
+ ],
186
+ [
187
+ 0.286376953125,
188
+ 0.254669189453125,
189
+ 0.499359130859375,
190
+ 0.36083984375,
191
+ 0.21551513671875,
192
+ 0.195831298828125,
193
+ 0.14984130859375,
194
+ 0.171142578125
195
+ ],
196
+ [
197
+ 0.3846435546875,
198
+ 0.24420166015625,
199
+ 0.2579345703125,
200
+ 0.267059326171875,
201
+ 0.279937744140625,
202
+ 0.378326416015625,
203
+ 0.229736328125,
204
+ 0.234710693359375
205
+ ],
206
+ [
207
+ 0.243560791015625,
208
+ 0.227264404296875,
209
+ 0.204864501953125,
210
+ 0.244720458984375,
211
+ 0.27691650390625,
212
+ 0.246490478515625,
213
+ 0.29443359375,
214
+ 0.200347900390625
215
+ ]
216
+ ],
217
+ "chance": 0.07715225219726562
218
+ }
219
+ }
default-seed0/analysis/final/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:495dc85a840188753c5988108593ae96365fd22456f9f8e412970d33522e4f36
3
+ size 189601
default-seed0/metrics.json ADDED
@@ -0,0 +1,228 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "run_name": "default-seed0",
4
+ "seed": 0,
5
+ "quick": false,
6
+ "device": "auto",
7
+ "eval_every": 250,
8
+ "task": {
9
+ "n_values": 64,
10
+ "length": 16
11
+ },
12
+ "splits": {
13
+ "val_seed": 1000003,
14
+ "val_size": 1024,
15
+ "test_seed": 2000003,
16
+ "test_size": 2048
17
+ },
18
+ "model": {
19
+ "n_layers": 4,
20
+ "d_model": 256,
21
+ "n_heads": 8,
22
+ "d_mlp": 1024,
23
+ "mlp": "gelu",
24
+ "norm": "layernorm",
25
+ "dropout": 0.0
26
+ },
27
+ "optim": {
28
+ "steps": 4000,
29
+ "batch_size": 256,
30
+ "lr": 0.001,
31
+ "warmup_steps": 100,
32
+ "min_lr_ratio": 0.1,
33
+ "weight_decay": 0.1,
34
+ "grad_clip": 1.0
35
+ }
36
+ },
37
+ "model_config": {
38
+ "vocab_size": 65,
39
+ "n_layers": 4,
40
+ "d_model": 256,
41
+ "n_heads": 8,
42
+ "d_mlp": 1024,
43
+ "max_seq_len": 33,
44
+ "mlp": "gelu",
45
+ "norm": "layernorm",
46
+ "dropout": 0.0
47
+ },
48
+ "parameters": 3201280,
49
+ "data": {
50
+ "task": "sorting",
51
+ "version": 1,
52
+ "n_values": 64,
53
+ "length": 16,
54
+ "val_seed": 1000003,
55
+ "val_size": 1024,
56
+ "test_seed": 2000003,
57
+ "test_size": 2048,
58
+ "train_seed": 0
59
+ },
60
+ "code": {
61
+ "commit": "19276f75392bee9d5f6f61820395002a0746e349",
62
+ "dirty": true,
63
+ "snapshot": "source-snapshot.tar.gz"
64
+ },
65
+ "environment": {
66
+ "python": "3.12.3",
67
+ "torch": "2.15.0+cu132",
68
+ "device": "cuda",
69
+ "precision": "torch.float32",
70
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
71
+ "gpu": "NVIDIA GeForce RTX 3060",
72
+ "cuda": "13.2"
73
+ },
74
+ "resumable": false,
75
+ "status": "finished",
76
+ "seconds": 646.7,
77
+ "history": [
78
+ {
79
+ "step": 250,
80
+ "train_loss": 0.07291533797979355,
81
+ "val_loss": 0.0985519140958786,
82
+ "val_token_accuracy": 0.9683837890625,
83
+ "val_exact_match": 0.595703125,
84
+ "grad_norm": 2.3497650623321533,
85
+ "lr": 0.0009967189933441244
86
+ },
87
+ {
88
+ "step": 500,
89
+ "train_loss": 0.014207392930984497,
90
+ "val_loss": 0.031683966517448425,
91
+ "val_token_accuracy": 0.99053955078125,
92
+ "val_exact_match": 0.8720703125,
93
+ "grad_norm": 0.7331729531288147,
94
+ "lr": 0.0009768413988762155
95
+ },
96
+ {
97
+ "step": 750,
98
+ "train_loss": 0.02052781730890274,
99
+ "val_loss": 0.023844445124268532,
100
+ "val_token_accuracy": 0.99273681640625,
101
+ "val_exact_match": 0.890625,
102
+ "grad_norm": 1.3566714525222778,
103
+ "lr": 0.0009397114317029974
104
+ },
105
+ {
106
+ "step": 1000,
107
+ "train_loss": 0.009082821197807789,
108
+ "val_loss": 0.01231020875275135,
109
+ "val_token_accuracy": 0.996826171875,
110
+ "val_exact_match": 0.951171875,
111
+ "grad_norm": 0.7233428359031677,
112
+ "lr": 0.0008868298366769955
113
+ },
114
+ {
115
+ "step": 1250,
116
+ "train_loss": 0.00922011211514473,
117
+ "val_loss": 0.019146941602230072,
118
+ "val_token_accuracy": 0.994873046875,
119
+ "val_exact_match": 0.9208984375,
120
+ "grad_norm": 0.5756050944328308,
121
+ "lr": 0.0008203340189070906
122
+ },
123
+ {
124
+ "step": 1500,
125
+ "train_loss": 0.01364622451364994,
126
+ "val_loss": 0.005576150491833687,
127
+ "val_token_accuracy": 0.998779296875,
128
+ "val_exact_match": 0.9814453125,
129
+ "grad_norm": 1.3124696016311646,
130
+ "lr": 0.0007429116526313744
131
+ },
132
+ {
133
+ "step": 1750,
134
+ "train_loss": 0.003266311716288328,
135
+ "val_loss": 0.001853839261457324,
136
+ "val_token_accuracy": 0.9996337890625,
137
+ "val_exact_match": 0.994140625,
138
+ "grad_norm": 0.28147438168525696,
139
+ "lr": 0.0006576920489294011
140
+ },
141
+ {
142
+ "step": 2000,
143
+ "train_loss": 0.005052485503256321,
144
+ "val_loss": 0.0006970376125536859,
145
+ "val_token_accuracy": 0.9998779296875,
146
+ "val_exact_match": 0.998046875,
147
+ "grad_norm": 1.1249206066131592,
148
+ "lr": 0.0005681196730492369
149
+ },
150
+ {
151
+ "step": 2250,
152
+ "train_loss": 0.00020874346955679357,
153
+ "val_loss": 0.0009217980550602078,
154
+ "val_token_accuracy": 0.99981689453125,
155
+ "val_exact_match": 0.9970703125,
156
+ "grad_norm": 0.03951434791088104,
157
+ "lr": 0.00047781492361400794
158
+ },
159
+ {
160
+ "step": 2500,
161
+ "train_loss": 0.001243826001882553,
162
+ "val_loss": 0.0005396516062319279,
163
+ "val_token_accuracy": 0.99981689453125,
164
+ "val_exact_match": 0.9970703125,
165
+ "grad_norm": 0.23733462393283844,
166
+ "lr": 0.0003904278008308589
167
+ },
168
+ {
169
+ "step": 2750,
170
+ "train_loss": 0.00022199669911060482,
171
+ "val_loss": 0.00017582655709702522,
172
+ "val_token_accuracy": 0.99993896484375,
173
+ "val_exact_match": 0.9990234375,
174
+ "grad_norm": 0.11565989255905151,
175
+ "lr": 0.0003094903782424895
176
+ },
177
+ {
178
+ "step": 3000,
179
+ "train_loss": 0.00021956569980829954,
180
+ "val_loss": 8.435545896645635e-05,
181
+ "val_token_accuracy": 1.0,
182
+ "val_exact_match": 1.0,
183
+ "grad_norm": 0.061819761991500854,
184
+ "lr": 0.0002382740409206803
185
+ },
186
+ {
187
+ "step": 3250,
188
+ "train_loss": 3.6058951081940904e-05,
189
+ "val_loss": 8.97766076377593e-05,
190
+ "val_token_accuracy": 0.99993896484375,
191
+ "val_exact_match": 0.9990234375,
192
+ "grad_norm": 0.004151737317442894,
193
+ "lr": 0.00017965726034785466
194
+ },
195
+ {
196
+ "step": 3500,
197
+ "train_loss": 4.75610613648314e-06,
198
+ "val_loss": 9.887960004562046e-06,
199
+ "val_token_accuracy": 1.0,
200
+ "val_exact_match": 1.0,
201
+ "grad_norm": 0.0010888022370636463,
202
+ "lr": 0.00013600925035352912
203
+ },
204
+ {
205
+ "step": 3750,
206
+ "train_loss": 0.00010177387594012544,
207
+ "val_loss": 0.00023778356262482703,
208
+ "val_token_accuracy": 0.99993896484375,
209
+ "val_exact_match": 0.9990234375,
210
+ "grad_norm": 0.09358467906713486,
211
+ "lr": 0.00010909420658097956
212
+ },
213
+ {
214
+ "step": 4000,
215
+ "train_loss": 1.727779454085976e-05,
216
+ "val_loss": 1.847820749389939e-05,
217
+ "val_token_accuracy": 1.0,
218
+ "val_exact_match": 1.0,
219
+ "grad_norm": 0.014979858882725239,
220
+ "lr": 0.0001
221
+ }
222
+ ],
223
+ "final": {
224
+ "val_loss": 1.847820749389939e-05,
225
+ "val_token_accuracy": 1.0,
226
+ "val_exact_match": 1.0
227
+ }
228
+ }
default-seed0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c56f9f38dbed7db80d810f7cbec73980f4303018991b46a0edb1bdf9f962fb68
3
+ size 12811264
default-seed0/model_config.json ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "format_version": 1,
3
+ "transformer": {
4
+ "vocab_size": 65,
5
+ "n_layers": 4,
6
+ "d_model": 256,
7
+ "n_heads": 8,
8
+ "d_mlp": 1024,
9
+ "max_seq_len": 33,
10
+ "mlp": "gelu",
11
+ "norm": "layernorm",
12
+ "dropout": 0.0
13
+ },
14
+ "vocab": [
15
+ "0",
16
+ "1",
17
+ "2",
18
+ "3",
19
+ "4",
20
+ "5",
21
+ "6",
22
+ "7",
23
+ "8",
24
+ "9",
25
+ "10",
26
+ "11",
27
+ "12",
28
+ "13",
29
+ "14",
30
+ "15",
31
+ "16",
32
+ "17",
33
+ "18",
34
+ "19",
35
+ "20",
36
+ "21",
37
+ "22",
38
+ "23",
39
+ "24",
40
+ "25",
41
+ "26",
42
+ "27",
43
+ "28",
44
+ "29",
45
+ "30",
46
+ "31",
47
+ "32",
48
+ "33",
49
+ "34",
50
+ "35",
51
+ "36",
52
+ "37",
53
+ "38",
54
+ "39",
55
+ "40",
56
+ "41",
57
+ "42",
58
+ "43",
59
+ "44",
60
+ "45",
61
+ "46",
62
+ "47",
63
+ "48",
64
+ "49",
65
+ "50",
66
+ "51",
67
+ "52",
68
+ "53",
69
+ "54",
70
+ "55",
71
+ "56",
72
+ "57",
73
+ "58",
74
+ "59",
75
+ "60",
76
+ "61",
77
+ "62",
78
+ "63",
79
+ "SEP"
80
+ ]
81
+ }
default-seed0/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e963742ccde0aca2462ea11e294c6b2504355b7b78eda8738885e210454496e
3
+ size 187056
default-seed0/training.png ADDED
default-seed1/analysis/figures/critical_pattern.png ADDED
default-seed1/analysis/figures/figure-data/curves-default-seed1.csv ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ step,train_loss,val_loss,val_token_accuracy,val_exact_match
2
+ 250,0.06796592473983765,0.08403412997722626,0.97271728515625,0.650390625
3
+ 500,0.03714346885681152,0.044457193464040756,0.98626708984375,0.8046875
4
+ 750,0.03594381362199783,0.028060290962457657,0.99298095703125,0.900390625
5
+ 1000,0.008170297369360924,0.012762336991727352,0.9959716796875,0.94140625
6
+ 1250,0.006513639353215694,0.018579615280032158,0.99481201171875,0.9208984375
7
+ 1500,0.008571112528443336,0.00933537445962429,0.997802734375,0.9658203125
8
+ 1750,0.008580565452575684,0.011725695803761482,0.997314453125,0.9580078125
9
+ 2000,0.0038308706134557724,0.003425079397857189,0.9991455078125,0.986328125
10
+ 2250,0.002150578424334526,0.001471434603445232,0.9996337890625,0.994140625
11
+ 2500,0.0023180556017905474,0.0010354401310905814,0.9996337890625,0.994140625
12
+ 2750,0.0008889065939001739,0.000499909685458988,0.9998779296875,0.998046875
13
+ 3000,0.00015005430032033473,0.00048269086983054876,0.99993896484375,0.9990234375
14
+ 3250,0.0008362794178538024,0.000332412077113986,0.99993896484375,0.9990234375
15
+ 3500,0.001066122087650001,2.008096817007754e-05,1.0,1.0
16
+ 3750,1.979812077479437e-05,1.038504888128955e-05,1.0,1.0
17
+ 4000,5.5594000514247455e-06,0.00019783558673225343,0.9998779296875,0.998046875
default-seed1/analysis/figures/figure-data/heads-default-seed1.csv ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ layer,head,loss_rise,token_accuracy_drop,lookup_current,lookup_next
2
+ 0,0,0.009939101075360668,0.00244140625,0.274139404296875,0.325408935546875
3
+ 0,1,5.683765084540937e-05,3.0517578125e-05,0.13470458984375,0.19158935546875
4
+ 0,2,0.22735828768236388,0.02294921875,0.109710693359375,0.24566650390625
5
+ 0,3,6.165035268773863,0.757965087890625,0.994598388671875,0.129638671875
6
+ 0,4,4.92795243189903e-05,3.0517578125e-05,0.22613525390625,0.229583740234375
7
+ 0,5,0.07720187168342818,0.01043701171875,0.29803466796875,0.35150146484375
8
+ 0,6,0.25589226793272246,0.043701171875,0.48126220703125,0.112030029296875
9
+ 0,7,0.04213370989782561,0.006103515625,0.26092529296875,0.3319091796875
10
+ 1,0,0.0015396755352412583,0.000518798828125,0.665252685546875,0.3260498046875
11
+ 1,1,0.017370990028211963,0.0048828125,0.162628173828125,0.425262451171875
12
+ 1,2,0.00043754249236371834,9.1552734375e-05,0.197509765625,0.4627685546875
13
+ 1,3,0.000748932261558366,0.000335693359375,0.362091064453125,0.472564697265625
14
+ 1,4,0.010392792073616874,0.00274658203125,0.314239501953125,0.45611572265625
15
+ 1,5,0.003005323482284439,0.001007080078125,0.163330078125,0.444305419921875
16
+ 1,6,0.003752188773432863,0.001312255859375,0.19354248046875,0.487762451171875
17
+ 1,7,0.0001520529494882794,3.0517578125e-05,0.289886474609375,0.45831298828125
18
+ 2,0,0.0002516775730327936,0.0,0.279388427734375,0.43133544921875
19
+ 2,1,8.41566961753415e-05,3.0517578125e-05,0.2215576171875,0.589935302734375
20
+ 2,2,0.0002615795692690881,9.1552734375e-05,0.33441162109375,0.513824462890625
21
+ 2,3,0.005035566182414186,0.002288818359375,0.267822265625,0.5626220703125
22
+ 2,4,0.0005736572366004111,0.00018310546875,0.23614501953125,0.60723876953125
23
+ 2,5,0.00016033766041800845,3.0517578125e-05,0.194366455078125,0.51190185546875
24
+ 2,6,0.0008616791619715514,0.000244140625,0.251312255859375,0.536346435546875
25
+ 2,7,0.0014845907153357984,0.0006103515625,0.311798095703125,0.489349365234375
26
+ 3,0,8.289516699733213e-06,0.0,0.230010986328125,0.354034423828125
27
+ 3,1,0.0002178412123612361,0.0,0.256103515625,0.410614013671875
28
+ 3,2,5.374424290494062e-06,0.0,0.21990966796875,0.4019775390625
29
+ 3,3,-1.4844681572867557e-07,0.0,0.166107177734375,0.220245361328125
30
+ 3,4,-1.2055061233695596e-06,0.0,0.149200439453125,0.181854248046875
31
+ 3,5,-4.305347829358652e-07,0.0,0.243621826171875,0.370697021484375
32
+ 3,6,3.061305505980272e-06,0.0,0.219970703125,0.2957763671875
33
+ 3,7,3.428203854127787e-07,0.0,0.25,0.246856689453125
default-seed1/analysis/figures/figure-data/layers-default-seed1.csv ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ layer,loss_rise,token_accuracy_drop
2
+ 0,7.488848230352232,0.88482666015625
3
+ 1,0.20484483849031676,0.056793212890625
4
+ 2,0.17635237287504424,0.0625
5
+ 3,0.004784376842508209,0.001190185546875
default-seed1/analysis/figures/figure-data/pattern-default-seed1.json ADDED
@@ -0,0 +1,1279 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "layer": 0,
3
+ "head": 3,
4
+ "tokens": [
5
+ "21",
6
+ "2",
7
+ "15",
8
+ "4",
9
+ "51",
10
+ "35",
11
+ "37",
12
+ "52",
13
+ "31",
14
+ "62",
15
+ "5",
16
+ "38",
17
+ "9",
18
+ "37",
19
+ "4",
20
+ "45",
21
+ "SEP",
22
+ "2",
23
+ "4",
24
+ "4",
25
+ "5",
26
+ "9",
27
+ "15",
28
+ "21",
29
+ "31",
30
+ "35",
31
+ "37",
32
+ "37",
33
+ "38",
34
+ "45",
35
+ "51",
36
+ "52",
37
+ "62"
38
+ ],
39
+ "input_length": 16,
40
+ "pattern": [
41
+ [
42
+ 1.0,
43
+ 0.0,
44
+ 0.0,
45
+ 0.0,
46
+ 0.0,
47
+ 0.0,
48
+ 0.0,
49
+ 0.0,
50
+ 0.0,
51
+ 0.0,
52
+ 0.0,
53
+ 0.0,
54
+ 0.0,
55
+ 0.0,
56
+ 0.0,
57
+ 0.0,
58
+ 0.0,
59
+ 0.0,
60
+ 0.0,
61
+ 0.0,
62
+ 0.0,
63
+ 0.0,
64
+ 0.0,
65
+ 0.0,
66
+ 0.0,
67
+ 0.0,
68
+ 0.0,
69
+ 0.0,
70
+ 0.0,
71
+ 0.0,
72
+ 0.0,
73
+ 0.0,
74
+ 0.0
75
+ ],
76
+ [
77
+ 0.0056287674233317375,
78
+ 0.994371235370636,
79
+ 0.0,
80
+ 0.0,
81
+ 0.0,
82
+ 0.0,
83
+ 0.0,
84
+ 0.0,
85
+ 0.0,
86
+ 0.0,
87
+ 0.0,
88
+ 0.0,
89
+ 0.0,
90
+ 0.0,
91
+ 0.0,
92
+ 0.0,
93
+ 0.0,
94
+ 0.0,
95
+ 0.0,
96
+ 0.0,
97
+ 0.0,
98
+ 0.0,
99
+ 0.0,
100
+ 0.0,
101
+ 0.0,
102
+ 0.0,
103
+ 0.0,
104
+ 0.0,
105
+ 0.0,
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0
110
+ ],
111
+ [
112
+ 0.0337347686290741,
113
+ 0.003753614379093051,
114
+ 0.9625115990638733,
115
+ 0.0,
116
+ 0.0,
117
+ 0.0,
118
+ 0.0,
119
+ 0.0,
120
+ 0.0,
121
+ 0.0,
122
+ 0.0,
123
+ 0.0,
124
+ 0.0,
125
+ 0.0,
126
+ 0.0,
127
+ 0.0,
128
+ 0.0,
129
+ 0.0,
130
+ 0.0,
131
+ 0.0,
132
+ 0.0,
133
+ 0.0,
134
+ 0.0,
135
+ 0.0,
136
+ 0.0,
137
+ 0.0,
138
+ 0.0,
139
+ 0.0,
140
+ 0.0,
141
+ 0.0,
142
+ 0.0,
143
+ 0.0,
144
+ 0.0
145
+ ],
146
+ [
147
+ 0.009305569343268871,
148
+ 0.046217598021030426,
149
+ 0.00931719783693552,
150
+ 0.9351596236228943,
151
+ 0.0,
152
+ 0.0,
153
+ 0.0,
154
+ 0.0,
155
+ 0.0,
156
+ 0.0,
157
+ 0.0,
158
+ 0.0,
159
+ 0.0,
160
+ 0.0,
161
+ 0.0,
162
+ 0.0,
163
+ 0.0,
164
+ 0.0,
165
+ 0.0,
166
+ 0.0,
167
+ 0.0,
168
+ 0.0,
169
+ 0.0,
170
+ 0.0,
171
+ 0.0,
172
+ 0.0,
173
+ 0.0,
174
+ 0.0,
175
+ 0.0,
176
+ 0.0,
177
+ 0.0,
178
+ 0.0,
179
+ 0.0
180
+ ],
181
+ [
182
+ 0.0029410384595394135,
183
+ 0.026337452232837677,
184
+ 0.0046421801671385765,
185
+ 0.019023653119802475,
186
+ 0.9470557570457458,
187
+ 0.0,
188
+ 0.0,
189
+ 0.0,
190
+ 0.0,
191
+ 0.0,
192
+ 0.0,
193
+ 0.0,
194
+ 0.0,
195
+ 0.0,
196
+ 0.0,
197
+ 0.0,
198
+ 0.0,
199
+ 0.0,
200
+ 0.0,
201
+ 0.0,
202
+ 0.0,
203
+ 0.0,
204
+ 0.0,
205
+ 0.0,
206
+ 0.0,
207
+ 0.0,
208
+ 0.0,
209
+ 0.0,
210
+ 0.0,
211
+ 0.0,
212
+ 0.0,
213
+ 0.0,
214
+ 0.0
215
+ ],
216
+ [
217
+ 0.006296431180089712,
218
+ 0.0067995390854775906,
219
+ 0.0024467534385621548,
220
+ 0.010534998029470444,
221
+ 0.008756743744015694,
222
+ 0.9651655554771423,
223
+ 0.0,
224
+ 0.0,
225
+ 0.0,
226
+ 0.0,
227
+ 0.0,
228
+ 0.0,
229
+ 0.0,
230
+ 0.0,
231
+ 0.0,
232
+ 0.0,
233
+ 0.0,
234
+ 0.0,
235
+ 0.0,
236
+ 0.0,
237
+ 0.0,
238
+ 0.0,
239
+ 0.0,
240
+ 0.0,
241
+ 0.0,
242
+ 0.0,
243
+ 0.0,
244
+ 0.0,
245
+ 0.0,
246
+ 0.0,
247
+ 0.0,
248
+ 0.0,
249
+ 0.0
250
+ ],
251
+ [
252
+ 0.00730488495901227,
253
+ 0.003240293124690652,
254
+ 0.022617008537054062,
255
+ 0.0014084362192079425,
256
+ 0.002127093030139804,
257
+ 0.018666155636310577,
258
+ 0.9446361660957336,
259
+ 0.0,
260
+ 0.0,
261
+ 0.0,
262
+ 0.0,
263
+ 0.0,
264
+ 0.0,
265
+ 0.0,
266
+ 0.0,
267
+ 0.0,
268
+ 0.0,
269
+ 0.0,
270
+ 0.0,
271
+ 0.0,
272
+ 0.0,
273
+ 0.0,
274
+ 0.0,
275
+ 0.0,
276
+ 0.0,
277
+ 0.0,
278
+ 0.0,
279
+ 0.0,
280
+ 0.0,
281
+ 0.0,
282
+ 0.0,
283
+ 0.0,
284
+ 0.0
285
+ ],
286
+ [
287
+ 0.029554974287748337,
288
+ 0.002137552248314023,
289
+ 0.008525188080966473,
290
+ 0.0029284912161529064,
291
+ 0.023835990577936172,
292
+ 0.00267363921739161,
293
+ 0.0034024526830762625,
294
+ 0.9269417524337769,
295
+ 0.0,
296
+ 0.0,
297
+ 0.0,
298
+ 0.0,
299
+ 0.0,
300
+ 0.0,
301
+ 0.0,
302
+ 0.0,
303
+ 0.0,
304
+ 0.0,
305
+ 0.0,
306
+ 0.0,
307
+ 0.0,
308
+ 0.0,
309
+ 0.0,
310
+ 0.0,
311
+ 0.0,
312
+ 0.0,
313
+ 0.0,
314
+ 0.0,
315
+ 0.0,
316
+ 0.0,
317
+ 0.0,
318
+ 0.0,
319
+ 0.0
320
+ ],
321
+ [
322
+ 0.0016302927397191525,
323
+ 0.0036944225430488586,
324
+ 0.007505543064326048,
325
+ 0.0017552506178617477,
326
+ 0.00807818491011858,
327
+ 0.017201170325279236,
328
+ 0.00792906153947115,
329
+ 0.000540276407264173,
330
+ 0.9516658186912537,
331
+ 0.0,
332
+ 0.0,
333
+ 0.0,
334
+ 0.0,
335
+ 0.0,
336
+ 0.0,
337
+ 0.0,
338
+ 0.0,
339
+ 0.0,
340
+ 0.0,
341
+ 0.0,
342
+ 0.0,
343
+ 0.0,
344
+ 0.0,
345
+ 0.0,
346
+ 0.0,
347
+ 0.0,
348
+ 0.0,
349
+ 0.0,
350
+ 0.0,
351
+ 0.0,
352
+ 0.0,
353
+ 0.0,
354
+ 0.0
355
+ ],
356
+ [
357
+ 0.032543014734983444,
358
+ 0.003915418870747089,
359
+ 0.026386000216007233,
360
+ 0.048863545060157776,
361
+ 0.02208738960325718,
362
+ 0.012656260281801224,
363
+ 0.011211938224732876,
364
+ 0.010928686708211899,
365
+ 0.03544507548213005,
366
+ 0.7959626913070679,
367
+ 0.0,
368
+ 0.0,
369
+ 0.0,
370
+ 0.0,
371
+ 0.0,
372
+ 0.0,
373
+ 0.0,
374
+ 0.0,
375
+ 0.0,
376
+ 0.0,
377
+ 0.0,
378
+ 0.0,
379
+ 0.0,
380
+ 0.0,
381
+ 0.0,
382
+ 0.0,
383
+ 0.0,
384
+ 0.0,
385
+ 0.0,
386
+ 0.0,
387
+ 0.0,
388
+ 0.0,
389
+ 0.0
390
+ ],
391
+ [
392
+ 0.020697344094514847,
393
+ 0.01915159448981285,
394
+ 0.004889092408120632,
395
+ 0.055012255907058716,
396
+ 0.007652617059648037,
397
+ 0.01024616975337267,
398
+ 0.01014277059584856,
399
+ 0.0414687804877758,
400
+ 0.003158681793138385,
401
+ 0.007078006397932768,
402
+ 0.8205026984214783,
403
+ 0.0,
404
+ 0.0,
405
+ 0.0,
406
+ 0.0,
407
+ 0.0,
408
+ 0.0,
409
+ 0.0,
410
+ 0.0,
411
+ 0.0,
412
+ 0.0,
413
+ 0.0,
414
+ 0.0,
415
+ 0.0,
416
+ 0.0,
417
+ 0.0,
418
+ 0.0,
419
+ 0.0,
420
+ 0.0,
421
+ 0.0,
422
+ 0.0,
423
+ 0.0,
424
+ 0.0
425
+ ],
426
+ [
427
+ 0.0038216663524508476,
428
+ 0.005144505295902491,
429
+ 0.01843036711215973,
430
+ 0.0011346811661496758,
431
+ 0.0014980246778577566,
432
+ 0.016808388754725456,
433
+ 0.0513039268553257,
434
+ 0.00504545820876956,
435
+ 0.004138512536883354,
436
+ 0.0010691492352634668,
437
+ 0.0025283447466790676,
438
+ 0.8890769481658936,
439
+ 0.0,
440
+ 0.0,
441
+ 0.0,
442
+ 0.0,
443
+ 0.0,
444
+ 0.0,
445
+ 0.0,
446
+ 0.0,
447
+ 0.0,
448
+ 0.0,
449
+ 0.0,
450
+ 0.0,
451
+ 0.0,
452
+ 0.0,
453
+ 0.0,
454
+ 0.0,
455
+ 0.0,
456
+ 0.0,
457
+ 0.0,
458
+ 0.0,
459
+ 0.0
460
+ ],
461
+ [
462
+ 0.014788861386477947,
463
+ 0.016526665538549423,
464
+ 0.006457935087382793,
465
+ 0.00567302480340004,
466
+ 0.030961129814386368,
467
+ 0.0375581756234169,
468
+ 0.008722063153982162,
469
+ 0.0053113168105483055,
470
+ 0.02551380544900894,
471
+ 0.013848965056240559,
472
+ 0.010869098827242851,
473
+ 0.032610658556222916,
474
+ 0.7911583185195923,
475
+ 0.0,
476
+ 0.0,
477
+ 0.0,
478
+ 0.0,
479
+ 0.0,
480
+ 0.0,
481
+ 0.0,
482
+ 0.0,
483
+ 0.0,
484
+ 0.0,
485
+ 0.0,
486
+ 0.0,
487
+ 0.0,
488
+ 0.0,
489
+ 0.0,
490
+ 0.0,
491
+ 0.0,
492
+ 0.0,
493
+ 0.0,
494
+ 0.0
495
+ ],
496
+ [
497
+ 0.003465716727077961,
498
+ 0.0013363217003643513,
499
+ 0.01127171516418457,
500
+ 0.000512742146383971,
501
+ 0.001051116967573762,
502
+ 0.008836048655211926,
503
+ 0.463044673204422,
504
+ 0.002462194301187992,
505
+ 0.007184683345258236,
506
+ 0.0022532211150974035,
507
+ 0.0023318100720643997,
508
+ 0.024335172027349472,
509
+ 0.0014328984543681145,
510
+ 0.4704817235469818,
511
+ 0.0,
512
+ 0.0,
513
+ 0.0,
514
+ 0.0,
515
+ 0.0,
516
+ 0.0,
517
+ 0.0,
518
+ 0.0,
519
+ 0.0,
520
+ 0.0,
521
+ 0.0,
522
+ 0.0,
523
+ 0.0,
524
+ 0.0,
525
+ 0.0,
526
+ 0.0,
527
+ 0.0,
528
+ 0.0,
529
+ 0.0
530
+ ],
531
+ [
532
+ 0.005401943810284138,
533
+ 0.018947286531329155,
534
+ 0.0065261246636509895,
535
+ 0.41978543996810913,
536
+ 0.01645742543041706,
537
+ 0.017205355688929558,
538
+ 0.0026961721014231443,
539
+ 0.006079786457121372,
540
+ 0.01181725412607193,
541
+ 0.029080737382173538,
542
+ 0.023446323350071907,
543
+ 0.0040831658989191055,
544
+ 0.00655824039131403,
545
+ 0.00259238388389349,
546
+ 0.42932233214378357,
547
+ 0.0,
548
+ 0.0,
549
+ 0.0,
550
+ 0.0,
551
+ 0.0,
552
+ 0.0,
553
+ 0.0,
554
+ 0.0,
555
+ 0.0,
556
+ 0.0,
557
+ 0.0,
558
+ 0.0,
559
+ 0.0,
560
+ 0.0,
561
+ 0.0,
562
+ 0.0,
563
+ 0.0,
564
+ 0.0
565
+ ],
566
+ [
567
+ 0.012805987149477005,
568
+ 0.013161402195692062,
569
+ 0.011426457203924656,
570
+ 0.008020521141588688,
571
+ 0.007964019663631916,
572
+ 0.014548967592418194,
573
+ 0.00676193181425333,
574
+ 0.012627669610083103,
575
+ 0.004576542880386114,
576
+ 0.007411624304950237,
577
+ 0.011817420832812786,
578
+ 0.00585333164781332,
579
+ 0.00526558980345726,
580
+ 0.006768215447664261,
581
+ 0.007559095975011587,
582
+ 0.8634310960769653,
583
+ 0.0,
584
+ 0.0,
585
+ 0.0,
586
+ 0.0,
587
+ 0.0,
588
+ 0.0,
589
+ 0.0,
590
+ 0.0,
591
+ 0.0,
592
+ 0.0,
593
+ 0.0,
594
+ 0.0,
595
+ 0.0,
596
+ 0.0,
597
+ 0.0,
598
+ 0.0,
599
+ 0.0
600
+ ],
601
+ [
602
+ 0.016494469717144966,
603
+ 0.026330171152949333,
604
+ 0.01767500676214695,
605
+ 0.024986015632748604,
606
+ 0.055334217846393585,
607
+ 0.025864768773317337,
608
+ 0.057951100170612335,
609
+ 0.07052712142467499,
610
+ 0.05733547732234001,
611
+ 0.08134312927722931,
612
+ 0.022683078423142433,
613
+ 0.03443731367588043,
614
+ 0.010515427216887474,
615
+ 0.060338545590639114,
616
+ 0.025953534990549088,
617
+ 0.126624196767807,
618
+ 0.28560641407966614,
619
+ 0.0,
620
+ 0.0,
621
+ 0.0,
622
+ 0.0,
623
+ 0.0,
624
+ 0.0,
625
+ 0.0,
626
+ 0.0,
627
+ 0.0,
628
+ 0.0,
629
+ 0.0,
630
+ 0.0,
631
+ 0.0,
632
+ 0.0,
633
+ 0.0,
634
+ 0.0
635
+ ],
636
+ [
637
+ 0.004350163042545319,
638
+ 0.5013648867607117,
639
+ 0.00400053383782506,
640
+ 0.01300878170877695,
641
+ 0.0232920590788126,
642
+ 0.008887429721653461,
643
+ 0.004573792684823275,
644
+ 0.004271232057362795,
645
+ 0.012335696257650852,
646
+ 0.004431537352502346,
647
+ 0.008035087957978249,
648
+ 0.008351154625415802,
649
+ 0.01756184734404087,
650
+ 0.004471444524824619,
651
+ 0.011388544924557209,
652
+ 0.012922966852784157,
653
+ 0.008924380876123905,
654
+ 0.34782838821411133,
655
+ 0.0,
656
+ 0.0,
657
+ 0.0,
658
+ 0.0,
659
+ 0.0,
660
+ 0.0,
661
+ 0.0,
662
+ 0.0,
663
+ 0.0,
664
+ 0.0,
665
+ 0.0,
666
+ 0.0,
667
+ 0.0,
668
+ 0.0,
669
+ 0.0
670
+ ],
671
+ [
672
+ 0.004142140503972769,
673
+ 0.016624469310045242,
674
+ 0.0034141079522669315,
675
+ 0.31197237968444824,
676
+ 0.012067227624356747,
677
+ 0.01185158547013998,
678
+ 0.0018036093097180128,
679
+ 0.004930904600769281,
680
+ 0.0060422481037676334,
681
+ 0.013310776092112064,
682
+ 0.0245559923350811,
683
+ 0.0026703388430178165,
684
+ 0.006114610470831394,
685
+ 0.001724808244034648,
686
+ 0.29331013560295105,
687
+ 0.011686134152114391,
688
+ 0.007224114146083593,
689
+ 0.015601199120283127,
690
+ 0.25095316767692566,
691
+ 0.0,
692
+ 0.0,
693
+ 0.0,
694
+ 0.0,
695
+ 0.0,
696
+ 0.0,
697
+ 0.0,
698
+ 0.0,
699
+ 0.0,
700
+ 0.0,
701
+ 0.0,
702
+ 0.0,
703
+ 0.0,
704
+ 0.0
705
+ ],
706
+ [
707
+ 0.0037247471045702696,
708
+ 0.016174985095858574,
709
+ 0.0034561112988740206,
710
+ 0.2548469305038452,
711
+ 0.011354505084455013,
712
+ 0.00969721656292677,
713
+ 0.0014788563130423427,
714
+ 0.0046325745061039925,
715
+ 0.005288855172693729,
716
+ 0.011365160346031189,
717
+ 0.018620217218995094,
718
+ 0.0024125350173562765,
719
+ 0.005841365549713373,
720
+ 0.0014116688398644328,
721
+ 0.2372138947248459,
722
+ 0.008001001551747322,
723
+ 0.006045340560376644,
724
+ 0.015324318781495094,
725
+ 0.20039218664169312,
726
+ 0.18271753191947937,
727
+ 0.0,
728
+ 0.0,
729
+ 0.0,
730
+ 0.0,
731
+ 0.0,
732
+ 0.0,
733
+ 0.0,
734
+ 0.0,
735
+ 0.0,
736
+ 0.0,
737
+ 0.0,
738
+ 0.0,
739
+ 0.0
740
+ ],
741
+ [
742
+ 0.01079960074275732,
743
+ 0.011146274395287037,
744
+ 0.0022937306202948093,
745
+ 0.0337345227599144,
746
+ 0.004673227667808533,
747
+ 0.006295668426901102,
748
+ 0.005560890305787325,
749
+ 0.021529361605644226,
750
+ 0.0016887009842321277,
751
+ 0.00373083702288568,
752
+ 0.42941388487815857,
753
+ 0.0041452618315815926,
754
+ 0.008445270359516144,
755
+ 0.0054857381619513035,
756
+ 0.03315233439207077,
757
+ 0.017140835523605347,
758
+ 0.006511059124022722,
759
+ 0.010242325253784657,
760
+ 0.03184131160378456,
761
+ 0.026989096775650978,
762
+ 0.3251800835132599,
763
+ 0.0,
764
+ 0.0,
765
+ 0.0,
766
+ 0.0,
767
+ 0.0,
768
+ 0.0,
769
+ 0.0,
770
+ 0.0,
771
+ 0.0,
772
+ 0.0,
773
+ 0.0,
774
+ 0.0
775
+ ],
776
+ [
777
+ 0.012195952236652374,
778
+ 0.01541457325220108,
779
+ 0.004169343970716,
780
+ 0.005737371277064085,
781
+ 0.02516764961183071,
782
+ 0.021368741989135742,
783
+ 0.0050116535276174545,
784
+ 0.004626625683158636,
785
+ 0.015415117144584656,
786
+ 0.009670281782746315,
787
+ 0.008729609660804272,
788
+ 0.016900869086384773,
789
+ 0.47324156761169434,
790
+ 0.004930323921144009,
791
+ 0.0050043268129229546,
792
+ 0.008104763925075531,
793
+ 0.008036436513066292,
794
+ 0.016462786123156548,
795
+ 0.005592823028564453,
796
+ 0.005777249112725258,
797
+ 0.00816836953163147,
798
+ 0.32027363777160645,
799
+ 0.0,
800
+ 0.0,
801
+ 0.0,
802
+ 0.0,
803
+ 0.0,
804
+ 0.0,
805
+ 0.0,
806
+ 0.0,
807
+ 0.0,
808
+ 0.0,
809
+ 0.0
810
+ ],
811
+ [
812
+ 0.03327929228544235,
813
+ 0.003866764483973384,
814
+ 0.5242980122566223,
815
+ 0.003138359636068344,
816
+ 0.005307186394929886,
817
+ 0.0022026854567229748,
818
+ 0.015319602563977242,
819
+ 0.014236025512218475,
820
+ 0.012412027455866337,
821
+ 0.007968375459313393,
822
+ 0.0025641906540840864,
823
+ 0.020395701751112938,
824
+ 0.002656317315995693,
825
+ 0.01604440249502659,
826
+ 0.003501161700114608,
827
+ 0.008572499267756939,
828
+ 0.004963376093655825,
829
+ 0.003821315709501505,
830
+ 0.0032031801529228687,
831
+ 0.003379451809450984,
832
+ 0.002596514532342553,
833
+ 0.0029120221734046936,
834
+ 0.3033614754676819,
835
+ 0.0,
836
+ 0.0,
837
+ 0.0,
838
+ 0.0,
839
+ 0.0,
840
+ 0.0,
841
+ 0.0,
842
+ 0.0,
843
+ 0.0,
844
+ 0.0
845
+ ],
846
+ [
847
+ 0.558424174785614,
848
+ 0.0032870760187506676,
849
+ 0.017560983076691628,
850
+ 0.002726122969761491,
851
+ 0.0015964414924383163,
852
+ 0.002364212181419134,
853
+ 0.0028389273211359978,
854
+ 0.016513871029019356,
855
+ 0.0018205271335318685,
856
+ 0.004956931807100773,
857
+ 0.0098385289311409,
858
+ 0.002564018592238426,
859
+ 0.0061355289071798325,
860
+ 0.0028477362357079983,
861
+ 0.00282105035148561,
862
+ 0.006238655187189579,
863
+ 0.0025214713532477617,
864
+ 0.003119264729321003,
865
+ 0.002728755120187998,
866
+ 0.002714051166549325,
867
+ 0.008698313497006893,
868
+ 0.006394281983375549,
869
+ 0.013485396280884743,
870
+ 0.31780368089675903,
871
+ 0.0,
872
+ 0.0,
873
+ 0.0,
874
+ 0.0,
875
+ 0.0,
876
+ 0.0,
877
+ 0.0,
878
+ 0.0,
879
+ 0.0
880
+ ],
881
+ [
882
+ 0.001357833738438785,
883
+ 0.0038780684117227793,
884
+ 0.005161534994840622,
885
+ 0.0018303628312423825,
886
+ 0.008151231333613396,
887
+ 0.01706889271736145,
888
+ 0.007357828784734011,
889
+ 0.0006390373455360532,
890
+ 0.5401540994644165,
891
+ 0.004446706734597683,
892
+ 0.0006896582781337202,
893
+ 0.003252560505643487,
894
+ 0.005721498746424913,
895
+ 0.007361728232353926,
896
+ 0.0018193587893620133,
897
+ 0.002798643661662936,
898
+ 0.0036879985127598047,
899
+ 0.004257186781615019,
900
+ 0.0017483910778537393,
901
+ 0.0018491669325158,
902
+ 0.0007520219660364091,
903
+ 0.005122872069478035,
904
+ 0.005116137210279703,
905
+ 0.0016101087676361203,
906
+ 0.36416706442832947,
907
+ 0.0,
908
+ 0.0,
909
+ 0.0,
910
+ 0.0,
911
+ 0.0,
912
+ 0.0,
913
+ 0.0,
914
+ 0.0
915
+ ],
916
+ [
917
+ 0.005138115491718054,
918
+ 0.006377323530614376,
919
+ 0.0021314136683940887,
920
+ 0.00988619402050972,
921
+ 0.007929463870823383,
922
+ 0.5124956965446472,
923
+ 0.011656934395432472,
924
+ 0.00480135390534997,
925
+ 0.01403276901692152,
926
+ 0.003970280289649963,
927
+ 0.0036652283743023872,
928
+ 0.013282215222716331,
929
+ 0.011281702667474747,
930
+ 0.010792362503707409,
931
+ 0.0080588823184371,
932
+ 0.010196010582149029,
933
+ 0.0035960422828793526,
934
+ 0.006226282566785812,
935
+ 0.008744598366320133,
936
+ 0.00802096351981163,
937
+ 0.0038902314845472574,
938
+ 0.010061136446893215,
939
+ 0.0024058816488832235,
940
+ 0.005201243795454502,
941
+ 0.014590348117053509,
942
+ 0.3015674352645874,
943
+ 0.0,
944
+ 0.0,
945
+ 0.0,
946
+ 0.0,
947
+ 0.0,
948
+ 0.0,
949
+ 0.0
950
+ ],
951
+ [
952
+ 0.003743792185559869,
953
+ 0.001907623023726046,
954
+ 0.008844357915222645,
955
+ 0.00082426454173401,
956
+ 0.001524312887340784,
957
+ 0.01078791357576847,
958
+ 0.3340506851673126,
959
+ 0.003355358261615038,
960
+ 0.006870136596262455,
961
+ 0.002977126743644476,
962
+ 0.0025026449002325535,
963
+ 0.021317970007658005,
964
+ 0.0018526981584727764,
965
+ 0.3373992145061493,
966
+ 0.0007903872756287456,
967
+ 0.0027546691708266735,
968
+ 0.003685057396069169,
969
+ 0.001800690544769168,
970
+ 0.0009049717918969691,
971
+ 0.0008664859342388809,
972
+ 0.0025867330841720104,
973
+ 0.0017302741762250662,
974
+ 0.008128976449370384,
975
+ 0.003793260781094432,
976
+ 0.006667313165962696,
977
+ 0.008749839849770069,
978
+ 0.21958324313163757,
979
+ 0.0,
980
+ 0.0,
981
+ 0.0,
982
+ 0.0,
983
+ 0.0,
984
+ 0.0
985
+ ],
986
+ [
987
+ 0.0031250757165253162,
988
+ 0.0014282881747931242,
989
+ 0.008465477265417576,
990
+ 0.0006142972852103412,
991
+ 0.0012052106903865933,
992
+ 0.007640608120709658,
993
+ 0.2727498710155487,
994
+ 0.0029759255703538656,
995
+ 0.005529684480279684,
996
+ 0.0025021929759532213,
997
+ 0.0020051158498972654,
998
+ 0.01989907957613468,
999
+ 0.0014877136563882232,
1000
+ 0.27548494935035706,
1001
+ 0.000595156685449183,
1002
+ 0.0023828695993870497,
1003
+ 0.003073555650189519,
1004
+ 0.0013416430447250605,
1005
+ 0.0006769761675968766,
1006
+ 0.0006438216078095138,
1007
+ 0.002058266429230571,
1008
+ 0.0013912650756537914,
1009
+ 0.0075393300503492355,
1010
+ 0.0031569055281579494,
1011
+ 0.005308047402650118,
1012
+ 0.006292236037552357,
1013
+ 0.17703121900558472,
1014
+ 0.18339522182941437,
1015
+ 0.0,
1016
+ 0.0,
1017
+ 0.0,
1018
+ 0.0,
1019
+ 0.0
1020
+ ],
1021
+ [
1022
+ 0.003358415560796857,
1023
+ 0.005256603006273508,
1024
+ 0.01290899608284235,
1025
+ 0.0015022206353023648,
1026
+ 0.0019044167129322886,
1027
+ 0.01627197302877903,
1028
+ 0.03569798171520233,
1029
+ 0.005330842919647694,
1030
+ 0.004044719040393829,
1031
+ 0.0015259361825883389,
1032
+ 0.002250880701467395,
1033
+ 0.45522555708885193,
1034
+ 0.0064188153482973576,
1035
+ 0.036091700196266174,
1036
+ 0.0014627263881266117,
1037
+ 0.0028452491387724876,
1038
+ 0.003286269959062338,
1039
+ 0.0049146125093102455,
1040
+ 0.0016189708840101957,
1041
+ 0.0016796275740489364,
1042
+ 0.0024636464659124613,
1043
+ 0.0057686069048941135,
1044
+ 0.011038044467568398,
1045
+ 0.0036903643049299717,
1046
+ 0.004285763017833233,
1047
+ 0.013809888623654842,
1048
+ 0.03151492029428482,
1049
+ 0.033759891986846924,
1050
+ 0.29007235169410706,
1051
+ 0.0,
1052
+ 0.0,
1053
+ 0.0,
1054
+ 0.0
1055
+ ],
1056
+ [
1057
+ 0.008303455077111721,
1058
+ 0.009926557540893555,
1059
+ 0.00761584984138608,
1060
+ 0.006386931054294109,
1061
+ 0.006312921643257141,
1062
+ 0.009911159984767437,
1063
+ 0.005230232607573271,
1064
+ 0.010570872575044632,
1065
+ 0.0035870082210749388,
1066
+ 0.006392893847078085,
1067
+ 0.008192788809537888,
1068
+ 0.004686950705945492,
1069
+ 0.003758795792236924,
1070
+ 0.005263189319521189,
1071
+ 0.006001153960824013,
1072
+ 0.4830405116081238,
1073
+ 0.00879328977316618,
1074
+ 0.008780372329056263,
1075
+ 0.006860379595309496,
1076
+ 0.005471652839332819,
1077
+ 0.007860271260142326,
1078
+ 0.003956700675189495,
1079
+ 0.007576397620141506,
1080
+ 0.007419833913445473,
1081
+ 0.004187455400824547,
1082
+ 0.00912896916270256,
1083
+ 0.0050418260507285595,
1084
+ 0.0052025155164301395,
1085
+ 0.004729395266622305,
1086
+ 0.3298097252845764,
1087
+ 0.0,
1088
+ 0.0,
1089
+ 0.0
1090
+ ],
1091
+ [
1092
+ 0.002535423031076789,
1093
+ 0.017202870920300484,
1094
+ 0.0040954481810331345,
1095
+ 0.011674498207867146,
1096
+ 0.48751765489578247,
1097
+ 0.008554724045097828,
1098
+ 0.0018213633447885513,
1099
+ 0.01984923705458641,
1100
+ 0.014986270107328892,
1101
+ 0.011790622025728226,
1102
+ 0.002596684033051133,
1103
+ 0.0014828713610768318,
1104
+ 0.011266556568443775,
1105
+ 0.0018179143080487847,
1106
+ 0.011096468195319176,
1107
+ 0.007654431741684675,
1108
+ 0.00985523872077465,
1109
+ 0.01614738442003727,
1110
+ 0.010381334461271763,
1111
+ 0.010833223350346088,
1112
+ 0.002641882048919797,
1113
+ 0.011476932093501091,
1114
+ 0.004257702734321356,
1115
+ 0.002431617584079504,
1116
+ 0.014161685481667519,
1117
+ 0.00837952084839344,
1118
+ 0.002142657758668065,
1119
+ 0.0019652554765343666,
1120
+ 0.001792567316442728,
1121
+ 0.006788833532482386,
1122
+ 0.2808011770248413,
1123
+ 0.0,
1124
+ 0.0
1125
+ ],
1126
+ [
1127
+ 0.02847876399755478,
1128
+ 0.002269891556352377,
1129
+ 0.01325911283493042,
1130
+ 0.0062932102009654045,
1131
+ 0.04032102972269058,
1132
+ 0.005264019127935171,
1133
+ 0.005873104091733694,
1134
+ 0.4263416528701782,
1135
+ 0.0018110958626493812,
1136
+ 0.011429733596742153,
1137
+ 0.013090435415506363,
1138
+ 0.005403920542448759,
1139
+ 0.00482679158449173,
1140
+ 0.005964090581983328,
1141
+ 0.0064321476966142654,
1142
+ 0.012229817919433117,
1143
+ 0.016589293256402016,
1144
+ 0.0027871341444551945,
1145
+ 0.006442916113883257,
1146
+ 0.00659376522526145,
1147
+ 0.012403004802763462,
1148
+ 0.005699170287698507,
1149
+ 0.012190915644168854,
1150
+ 0.023476582020521164,
1151
+ 0.0019740986172109842,
1152
+ 0.005311937537044287,
1153
+ 0.006386847700923681,
1154
+ 0.0067416587844491005,
1155
+ 0.005594385787844658,
1156
+ 0.012196172028779984,
1157
+ 0.03210291266441345,
1158
+ 0.2542203664779663,
1159
+ 0.0
1160
+ ],
1161
+ [
1162
+ 0.0193486325442791,
1163
+ 0.0027795261703431606,
1164
+ 0.012100462801754475,
1165
+ 0.023452261462807655,
1166
+ 0.012927545234560966,
1167
+ 0.006814536638557911,
1168
+ 0.006893753074109554,
1169
+ 0.005798648111522198,
1170
+ 0.020403049886226654,
1171
+ 0.35817620158195496,
1172
+ 0.0025705862790346146,
1173
+ 0.0016229369211941957,
1174
+ 0.008698305115103722,
1175
+ 0.006920742802321911,
1176
+ 0.025664983317255974,
1177
+ 0.010590286925435066,
1178
+ 0.02775728330016136,
1179
+ 0.0028971214778721333,
1180
+ 0.02300073578953743,
1181
+ 0.0217321515083313,
1182
+ 0.0028180184308439493,
1183
+ 0.00929475761950016,
1184
+ 0.011649117805063725,
1185
+ 0.016284432262182236,
1186
+ 0.018083419650793076,
1187
+ 0.008077215403318405,
1188
+ 0.007342285476624966,
1189
+ 0.007146804593503475,
1190
+ 0.0019610587041825056,
1191
+ 0.010746720246970654,
1192
+ 0.013193007558584213,
1193
+ 0.009442578069865704,
1194
+ 0.28381088376045227
1195
+ ]
1196
+ ],
1197
+ "expected": [
1198
+ [
1199
+ 17,
1200
+ 1
1201
+ ],
1202
+ [
1203
+ 18,
1204
+ 3
1205
+ ],
1206
+ [
1207
+ 18,
1208
+ 14
1209
+ ],
1210
+ [
1211
+ 19,
1212
+ 3
1213
+ ],
1214
+ [
1215
+ 19,
1216
+ 14
1217
+ ],
1218
+ [
1219
+ 20,
1220
+ 10
1221
+ ],
1222
+ [
1223
+ 21,
1224
+ 12
1225
+ ],
1226
+ [
1227
+ 22,
1228
+ 2
1229
+ ],
1230
+ [
1231
+ 23,
1232
+ 0
1233
+ ],
1234
+ [
1235
+ 24,
1236
+ 8
1237
+ ],
1238
+ [
1239
+ 25,
1240
+ 5
1241
+ ],
1242
+ [
1243
+ 26,
1244
+ 6
1245
+ ],
1246
+ [
1247
+ 26,
1248
+ 13
1249
+ ],
1250
+ [
1251
+ 27,
1252
+ 6
1253
+ ],
1254
+ [
1255
+ 27,
1256
+ 13
1257
+ ],
1258
+ [
1259
+ 28,
1260
+ 11
1261
+ ],
1262
+ [
1263
+ 29,
1264
+ 15
1265
+ ],
1266
+ [
1267
+ 30,
1268
+ 4
1269
+ ],
1270
+ [
1271
+ 31,
1272
+ 7
1273
+ ],
1274
+ [
1275
+ 32,
1276
+ 9
1277
+ ]
1278
+ ]
1279
+ }
default-seed1/analysis/figures/head_ablation.png ADDED
default-seed1/analysis/figures/lookup_current.png ADDED
default-seed1/analysis/figures/lookup_next.png ADDED
default-seed1/analysis/figures/lookup_next_pattern.png ADDED
default-seed1/analysis/figures/results.json ADDED
@@ -0,0 +1,219 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "runs/default-seed1",
3
+ "code": {
4
+ "commit": "42933bd7a4e2e13fd737d0d48d489f72fbeed8ac",
5
+ "dirty": true,
6
+ "snapshot": "source-snapshot.tar.gz"
7
+ },
8
+ "environment": {
9
+ "python": "3.12.3",
10
+ "torch": "2.15.0+cu132",
11
+ "device": "cuda",
12
+ "precision": "torch.float32",
13
+ "uv_lock_sha256": "60eba889e8dcd90d00fdcc282a51fc42df3869f3ef3807b1946aaa3cfd5f62e2",
14
+ "gpu": "NVIDIA GeForce RTX 3060",
15
+ "cuda": "13.2"
16
+ },
17
+ "test_size": 2048,
18
+ "chance_token_accuracy": 0.015625,
19
+ "trained": {
20
+ "loss": 1.1899957826244645e-05,
21
+ "token_accuracy": 1.0,
22
+ "exact_match": 1.0
23
+ },
24
+ "untrained": {
25
+ "loss": 4.231888294219971,
26
+ "token_accuracy": 0.0140380859375,
27
+ "exact_match": 0.0
28
+ },
29
+ "head_ablation": {
30
+ "clean_loss": 1.1899957826244645e-05,
31
+ "identity_patch_max_logit_change": 0.0,
32
+ "loss_rise": [
33
+ [
34
+ 0.009939101075360668,
35
+ 5.683765084540937e-05,
36
+ 0.22735828768236388,
37
+ 6.165035268773863,
38
+ 4.92795243189903e-05,
39
+ 0.07720187168342818,
40
+ 0.25589226793272246,
41
+ 0.04213370989782561
42
+ ],
43
+ [
44
+ 0.0015396755352412583,
45
+ 0.017370990028211963,
46
+ 0.00043754249236371834,
47
+ 0.000748932261558366,
48
+ 0.010392792073616874,
49
+ 0.003005323482284439,
50
+ 0.003752188773432863,
51
+ 0.0001520529494882794
52
+ ],
53
+ [
54
+ 0.0002516775730327936,
55
+ 8.41566961753415e-05,
56
+ 0.0002615795692690881,
57
+ 0.005035566182414186,
58
+ 0.0005736572366004111,
59
+ 0.00016033766041800845,
60
+ 0.0008616791619715514,
61
+ 0.0014845907153357984
62
+ ],
63
+ [
64
+ 8.289516699733213e-06,
65
+ 0.0002178412123612361,
66
+ 5.374424290494062e-06,
67
+ -1.4844681572867557e-07,
68
+ -1.2055061233695596e-06,
69
+ -4.305347829358652e-07,
70
+ 3.061305505980272e-06,
71
+ 3.428203854127787e-07
72
+ ]
73
+ ],
74
+ "token_accuracy_drop": [
75
+ [
76
+ 0.00244140625,
77
+ 3.0517578125e-05,
78
+ 0.02294921875,
79
+ 0.757965087890625,
80
+ 3.0517578125e-05,
81
+ 0.01043701171875,
82
+ 0.043701171875,
83
+ 0.006103515625
84
+ ],
85
+ [
86
+ 0.000518798828125,
87
+ 0.0048828125,
88
+ 9.1552734375e-05,
89
+ 0.000335693359375,
90
+ 0.00274658203125,
91
+ 0.001007080078125,
92
+ 0.001312255859375,
93
+ 3.0517578125e-05
94
+ ],
95
+ [
96
+ 0.0,
97
+ 3.0517578125e-05,
98
+ 9.1552734375e-05,
99
+ 0.002288818359375,
100
+ 0.00018310546875,
101
+ 3.0517578125e-05,
102
+ 0.000244140625,
103
+ 0.0006103515625
104
+ ],
105
+ [
106
+ 0.0,
107
+ 0.0,
108
+ 0.0,
109
+ 0.0,
110
+ 0.0,
111
+ 0.0,
112
+ 0.0,
113
+ 0.0
114
+ ]
115
+ ],
116
+ "layer_loss_rise": [
117
+ 7.488848230352232,
118
+ 0.20484483849031676,
119
+ 0.17635237287504424,
120
+ 0.004784376842508209
121
+ ],
122
+ "layer_token_accuracy_drop": [
123
+ 0.88482666015625,
124
+ 0.056793212890625,
125
+ 0.0625,
126
+ 0.001190185546875
127
+ ]
128
+ },
129
+ "lookup_next": {
130
+ "score": [
131
+ [
132
+ 0.325408935546875,
133
+ 0.19158935546875,
134
+ 0.24566650390625,
135
+ 0.129638671875,
136
+ 0.229583740234375,
137
+ 0.35150146484375,
138
+ 0.112030029296875,
139
+ 0.3319091796875
140
+ ],
141
+ [
142
+ 0.3260498046875,
143
+ 0.425262451171875,
144
+ 0.4627685546875,
145
+ 0.472564697265625,
146
+ 0.45611572265625,
147
+ 0.444305419921875,
148
+ 0.487762451171875,
149
+ 0.45831298828125
150
+ ],
151
+ [
152
+ 0.43133544921875,
153
+ 0.589935302734375,
154
+ 0.513824462890625,
155
+ 0.5626220703125,
156
+ 0.60723876953125,
157
+ 0.51190185546875,
158
+ 0.536346435546875,
159
+ 0.489349365234375
160
+ ],
161
+ [
162
+ 0.354034423828125,
163
+ 0.410614013671875,
164
+ 0.4019775390625,
165
+ 0.220245361328125,
166
+ 0.181854248046875,
167
+ 0.370697021484375,
168
+ 0.2957763671875,
169
+ 0.246856689453125
170
+ ]
171
+ ],
172
+ "chance": 0.07715225219726562
173
+ },
174
+ "lookup_current": {
175
+ "score": [
176
+ [
177
+ 0.274139404296875,
178
+ 0.13470458984375,
179
+ 0.109710693359375,
180
+ 0.994598388671875,
181
+ 0.22613525390625,
182
+ 0.29803466796875,
183
+ 0.48126220703125,
184
+ 0.26092529296875
185
+ ],
186
+ [
187
+ 0.665252685546875,
188
+ 0.162628173828125,
189
+ 0.197509765625,
190
+ 0.362091064453125,
191
+ 0.314239501953125,
192
+ 0.163330078125,
193
+ 0.19354248046875,
194
+ 0.289886474609375
195
+ ],
196
+ [
197
+ 0.279388427734375,
198
+ 0.2215576171875,
199
+ 0.33441162109375,
200
+ 0.267822265625,
201
+ 0.23614501953125,
202
+ 0.194366455078125,
203
+ 0.251312255859375,
204
+ 0.311798095703125
205
+ ],
206
+ [
207
+ 0.230010986328125,
208
+ 0.256103515625,
209
+ 0.21990966796875,
210
+ 0.166107177734375,
211
+ 0.149200439453125,
212
+ 0.243621826171875,
213
+ 0.219970703125,
214
+ 0.25
215
+ ]
216
+ ],
217
+ "chance": 0.07715225219726562
218
+ }
219
+ }
default-seed1/analysis/figures/source-snapshot.tar.gz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b33f10047810fe6d53b996f5128c29481cfa337cba04f007cfe8a49edce5bcf
3
+ size 347464
default-seed1/analysis/final/critical_pattern.png ADDED
default-seed1/analysis/final/head_ablation.png ADDED
default-seed1/analysis/final/lookup_current.png ADDED
default-seed1/analysis/final/lookup_next.png ADDED
default-seed1/analysis/final/lookup_next_pattern.png ADDED