kacperwikiel commited on
Commit
64b3931
·
verified ·
1 Parent(s): 6eed3c4

Add benchmark plots generated from EVALUATION.json

Browse files
.gitattributes CHANGED
@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ plots/additional-benchmarks.png filter=lfs diff=lfs merge=lfs -text
37
+ plots/arc-challenge.png filter=lfs diff=lfs merge=lfs -text
38
+ plots/belebele-polish.png filter=lfs diff=lfs merge=lfs -text
39
+ plots/benchmark-overview.png filter=lfs diff=lfs merge=lfs -text
40
+ plots/boolq.png filter=lfs diff=lfs merge=lfs -text
41
+ plots/hellaswag-pl.png filter=lfs diff=lfs merge=lfs -text
42
+ plots/hellaswag.png filter=lfs diff=lfs merge=lfs -text
43
+ plots/mmlu-pl.png filter=lfs diff=lfs merge=lfs -text
44
+ plots/openbookqa.png filter=lfs diff=lfs merge=lfs -text
45
+ plots/piqa.png filter=lfs diff=lfs merge=lfs -text
46
+ plots/winogrande.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -167,6 +167,16 @@ MMLU-PL and Belebele-PL score the answer letter A–D given listed options; ARC-
167
 
168
  There is no replay-only control for this continuation, and initialization, exposure and schedule differ between checkpoints. These measurements do not isolate a causal effect of the new datasets or quantify training-seed variance. Earlier internal diagnostics are separate historical records, not a criterion for naming or publishing this model.
169
 
 
 
 
 
 
 
 
 
 
 
170
  ### Auxiliary class-balanced diagnostic
171
 
172
  Balanced accuracy is the unweighted mean of recall across gold-answer classes. It was collected after the primary measurements. The values below are descriptive; their source does not provide confidence intervals. They retain a separate method and source digest in [EVALUATION.json](EVALUATION.json).
@@ -198,6 +208,10 @@ Scores are percentages with reported pointwise 95% bootstrap intervals (10,000 d
198
 
199
  **Descriptive results; training-lineage contamination scan pending.** GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.
200
 
 
 
 
 
201
  ## Cost, licensing and limitations
202
 
203
  Continued training is budget-limited and may pause with spot availability. The funding receipt above is not the current balance or a guarantee of a final update count. Metered attribution is not a settled invoice; hardware choice must account for effective retained progress, interruptions, storage and evaluation, not only an advertised GPU-hour price. This page does not allocate a new budget to gold-data training or change the existing cost guards.
 
167
 
168
  There is no replay-only control for this continuation, and initialization, exposure and schedule differ between checkpoints. These measurements do not isolate a causal effect of the new datasets or quantify training-seed variance. Earlier internal diagnostics are separate historical records, not a criterion for naming or publishing this model.
169
 
170
+ ### Benchmark plots
171
+
172
+ ![Benchmark scores across Slayer1B versions and 1.09 checkpoints](plots/benchmark-overview.png)
173
+
174
+ Each panel shows one benchmark from 1.07 and 1.08 through the measured 1.09 checkpoints at updates 5,000–45,000. Error bars and shading show the reported pointwise 95% bootstrap confidence intervals. The x-axis uses equally spaced checkpoint categories; y-axis ranges differ across panels. These intervals describe individual scores, not paired differences or training-seed variance. Training-lineage contamination checks remain incomplete.
175
+
176
+ The plots use the full-precision values in [EVALUATION.json](EVALUATION.json), selecting the primary metric declared for each task and converting fractions to percentages. Download the overview as [SVG](plots/benchmark-overview.svg) or [PDF](plots/benchmark-overview.pdf), the [plotted data as CSV](plots/benchmark-scores.csv), or the [reproducible plotting script](plots/plot_benchmarks.py) (Python with Matplotlib; run from a repository checkout).
177
+
178
+ Individual plots: [ARC-Challenge](plots/arc-challenge.png), [HellaSwag](plots/hellaswag.png), [PIQA](plots/piqa.png), [WinoGrande](plots/winogrande.png), [BoolQ](plots/boolq.png), [OpenBookQA](plots/openbookqa.png), [Belebele-PL](plots/belebele-polish.png), [HellaSwag-PL](plots/hellaswag-pl.png), [MMLU-PL](plots/mmlu-pl.png).
179
+
180
  ### Auxiliary class-balanced diagnostic
181
 
182
  Balanced accuracy is the unweighted mean of recall across gold-answer classes. It was collected after the primary measurements. The values below are descriptive; their source does not provide confidence intervals. They retain a separate method and source digest in [EVALUATION.json](EVALUATION.json).
 
208
 
209
  **Descriptive results; training-lineage contamination scan pending.** GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.
210
 
211
+ ![GSM8K and SST-2 scores for the three measured checkpoints](plots/additional-benchmarks.png)
212
+
213
+ The plot retains this separate evaluation protocol and shows only the three published measurements, with their reported pointwise 95% intervals. [Download SVG](plots/additional-benchmarks.svg).
214
+
215
  ## Cost, licensing and limitations
216
 
217
  Continued training is budget-limited and may pause with spot availability. The funding receipt above is not the current balance or a guarantee of a final update count. Metered attribution is not a settled invoice; hardware choice must account for effective retained progress, interruptions, storage and evaluation, not only an advertised GPU-hour price. This page does not allocate a new budget to gold-data training or change the existing cost guards.
plots/additional-benchmarks.png ADDED

Git LFS Details

  • SHA256: 92fb75feb6ec965d6025cf6fe2ffa331065daf4c98e81bebdc0ab50c0641c762
  • Pointer size: 131 Bytes
  • Size of remote file: 160 kB
plots/additional-benchmarks.svg ADDED
plots/arc-challenge.png ADDED

Git LFS Details

  • SHA256: 9339ff44b3be46033deb0e6995dcbb704c0d7f4799129e68f35519a16baa6e90
  • Pointer size: 131 Bytes
  • Size of remote file: 136 kB
plots/belebele-polish.png ADDED

Git LFS Details

  • SHA256: 6fd8b9052ac31e6d21a421a586f018e735990047aa52c6be49088f56b8d3e2af
  • Pointer size: 131 Bytes
  • Size of remote file: 135 kB
plots/benchmark-overview.pdf ADDED
Binary file (53.1 kB). View file
 
plots/benchmark-overview.png ADDED

Git LFS Details

  • SHA256: e5ab75c6f9d3481a2685d18e8e23cb00513fe3915bea8ded433d07c374122a05
  • Pointer size: 131 Bytes
  • Size of remote file: 570 kB
plots/benchmark-overview.svg ADDED
plots/benchmark-scores.csv ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ task,metric,n,model,score,ci95_low,ci95_high
2
+ ARC-Challenge,acc_norm,1172,1.07,27.3038,24.744,29.8635
3
+ ARC-Challenge,acc_norm,1172,1.08,30.034100000000002,27.3891,32.6792
4
+ ARC-Challenge,acc_norm,1172,"1.09, update 5,000",29.3515,26.7918,31.996599999999997
5
+ ARC-Challenge,acc_norm,1172,"1.09, update 10,000",30.4608,27.8157,33.1058
6
+ ARC-Challenge,acc_norm,1172,"1.09, update 15,000",32.5085,29.8635,35.2389
7
+ ARC-Challenge,acc_norm,1172,"1.09, update 20,000",33.1058,30.4608,35.8362
8
+ ARC-Challenge,acc_norm,1172,"1.09, update 25,000",33.7884,31.058000000000003,36.5188
9
+ ARC-Challenge,acc_norm,1172,"1.09, update 30,000",33.020500000000006,30.2901,35.6655
10
+ ARC-Challenge,acc_norm,1172,"1.09, update 35,000",35.153600000000004,32.4232,37.884
11
+ ARC-Challenge,acc_norm,1172,"1.09, update 40,000",34.9829,32.2526,37.7986
12
+ ARC-Challenge,acc_norm,1172,"1.09, update 45,000",34.8123,32.081900000000005,37.5427
13
+ HellaSwag,acc_norm,10042,1.07,43.3778,42.431799999999996,44.3438
14
+ HellaSwag,acc_norm,10042,1.08,43.4176,42.431799999999996,44.3935
15
+ HellaSwag,acc_norm,10042,"1.09, update 5,000",46.5346,45.5487,47.5204
16
+ HellaSwag,acc_norm,10042,"1.09, update 10,000",48.9345,47.9685,49.910399999999996
17
+ HellaSwag,acc_norm,10042,"1.09, update 15,000",49.771,48.795100000000005,50.7568
18
+ HellaSwag,acc_norm,10042,"1.09, update 20,000",50.3784,49.392599999999995,51.354299999999995
19
+ HellaSwag,acc_norm,10042,"1.09, update 25,000",51.165099999999995,50.1792,52.141000000000005
20
+ HellaSwag,acc_norm,10042,"1.09, update 30,000",52.0016,51.015699999999995,52.9775
21
+ HellaSwag,acc_norm,10042,"1.09, update 35,000",52.479600000000005,51.503699999999995,53.4555
22
+ HellaSwag,acc_norm,10042,"1.09, update 40,000",52.798199999999994,51.812400000000004,53.7841
23
+ HellaSwag,acc_norm,10042,"1.09, update 45,000",53.6547,52.6887,54.620599999999996
24
+ PIQA,acc_norm,1838,1.07,66.975,64.7987,69.1513
25
+ PIQA,acc_norm,1838,1.08,66.2133,64.1458,68.33510000000001
26
+ PIQA,acc_norm,1838,"1.09, update 5,000",68.28070000000001,66.2119,70.4026
27
+ PIQA,acc_norm,1838,"1.09, update 10,000",68.988,66.9206,71.10990000000001
28
+ PIQA,acc_norm,1838,"1.09, update 15,000",68.988,66.86619999999999,71.05550000000001
29
+ PIQA,acc_norm,1838,"1.09, update 20,000",70.7835,68.716,72.8509
30
+ PIQA,acc_norm,1838,"1.09, update 25,000",71.5452,69.47630000000001,73.6126
31
+ PIQA,acc_norm,1838,"1.09, update 30,000",70.6746,68.6072,72.7421
32
+ PIQA,acc_norm,1838,"1.09, update 35,000",71.2731,69.1513,73.28620000000001
33
+ PIQA,acc_norm,1838,"1.09, update 40,000",71.2187,69.20429999999999,73.28620000000001
34
+ PIQA,acc_norm,1838,"1.09, update 45,000",70.29379999999999,68.2263,72.3069
35
+ WinoGrande,acc,1267,1.07,53.59120000000001,50.828700000000005,56.3536
36
+ WinoGrande,acc,1267,1.08,53.6701,50.9077,56.4325
37
+ WinoGrande,acc,1267,"1.09, update 5,000",54.9329,52.1705,57.6953
38
+ WinoGrande,acc,1267,"1.09, update 10,000",54.9329,52.1705,57.6164
39
+ WinoGrande,acc,1267,"1.09, update 15,000",56.748200000000004,54.0647,59.4317
40
+ WinoGrande,acc,1267,"1.09, update 20,000",55.0118,52.2494,57.774300000000004
41
+ WinoGrande,acc,1267,"1.09, update 25,000",55.879999999999995,53.117599999999996,58.6425
42
+ WinoGrande,acc,1267,"1.09, update 30,000",57.1429,54.4574,59.8264
43
+ WinoGrande,acc,1267,"1.09, update 35,000",56.748200000000004,54.0647,59.4317
44
+ WinoGrande,acc,1267,"1.09, update 40,000",57.6953,55.0118,60.3788
45
+ WinoGrande,acc,1267,"1.09, update 45,000",59.037099999999995,56.2747,61.720600000000005
46
+ BoolQ,acc,3270,1.07,58.2875,56.6055,59.96939999999999
47
+ BoolQ,acc,3270,1.08,59.96939999999999,58.2569,61.590199999999996
48
+ BoolQ,acc,3270,"1.09, update 5,000",61.8043,60.1827,63.486200000000004
49
+ BoolQ,acc,3270,"1.09, update 10,000",61.1315,59.4495,62.7829
50
+ BoolQ,acc,3270,"1.09, update 15,000",59.8777,58.195699999999995,61.5291
51
+ BoolQ,acc,3270,"1.09, update 20,000",51.40670000000001,49.6942,53.1193
52
+ BoolQ,acc,3270,"1.09, update 25,000",52.2018,50.519099999999995,53.91439999999999
53
+ BoolQ,acc,3270,"1.09, update 30,000",60.6728,58.9908,62.354699999999994
54
+ BoolQ,acc,3270,"1.09, update 35,000",59.8777,58.195699999999995,61.5596
55
+ BoolQ,acc,3270,"1.09, update 40,000",59.1743,57.4618,60.825700000000005
56
+ BoolQ,acc,3270,"1.09, update 45,000",61.9266,60.244600000000005,63.6086
57
+ OpenBookQA,acc_norm,500,1.07,33.6,29.4,37.8
58
+ OpenBookQA,acc_norm,500,1.08,34.8,30.599999999999998,39.0
59
+ OpenBookQA,acc_norm,500,"1.09, update 5,000",35.6,31.4,40.0
60
+ OpenBookQA,acc_norm,500,"1.09, update 10,000",34.4,30.2,38.6
61
+ OpenBookQA,acc_norm,500,"1.09, update 15,000",36.0,31.8,40.2
62
+ OpenBookQA,acc_norm,500,"1.09, update 20,000",35.199999999999996,31.0,39.4
63
+ OpenBookQA,acc_norm,500,"1.09, update 25,000",37.2,32.800000000000004,41.4
64
+ OpenBookQA,acc_norm,500,"1.09, update 30,000",36.199999999999996,32.0,40.400000000000006
65
+ OpenBookQA,acc_norm,500,"1.09, update 35,000",34.4,30.4,38.6
66
+ OpenBookQA,acc_norm,500,"1.09, update 40,000",34.599999999999994,30.4,38.800000000000004
67
+ OpenBookQA,acc_norm,500,"1.09, update 45,000",36.0,31.8,40.2
68
+ "Belebele, Polish",acc,900,1.07,26.0,23.1111,28.8889
69
+ "Belebele, Polish",acc,900,1.08,24.0,21.2222,26.8889
70
+ "Belebele, Polish",acc,900,"1.09, update 5,000",24.1111,21.3333,26.8889
71
+ "Belebele, Polish",acc,900,"1.09, update 10,000",24.1111,21.3333,27.0
72
+ "Belebele, Polish",acc,900,"1.09, update 15,000",22.555600000000002,19.8889,25.333299999999998
73
+ "Belebele, Polish",acc,900,"1.09, update 20,000",24.6667,21.8889,27.555600000000002
74
+ "Belebele, Polish",acc,900,"1.09, update 25,000",25.333299999999998,22.4444,28.222199999999997
75
+ "Belebele, Polish",acc,900,"1.09, update 30,000",24.444399999999998,21.6667,27.333299999999998
76
+ "Belebele, Polish",acc,900,"1.09, update 35,000",24.333299999999998,21.5556,27.1139
77
+ "Belebele, Polish",acc,900,"1.09, update 40,000",25.555600000000002,22.7778,28.444399999999998
78
+ "Belebele, Polish",acc,900,"1.09, update 45,000",25.333299999999998,22.555600000000002,28.222199999999997
79
+ HellaSwag-PL,acc_norm,10042,1.07,43.825900000000004,42.8401,44.792100000000005
80
+ HellaSwag-PL,acc_norm,10042,1.08,43.8956,42.8998,44.8715
81
+ HellaSwag-PL,acc_norm,10042,"1.09, update 5,000",44.7321,43.7562,45.708
82
+ HellaSwag-PL,acc_norm,10042,"1.09, update 10,000",45.508900000000004,44.513000000000005,46.4848
83
+ HellaSwag-PL,acc_norm,10042,"1.09, update 15,000",44.184400000000004,43.2085,45.1603
84
+ HellaSwag-PL,acc_norm,10042,"1.09, update 20,000",44.2442,43.2583,45.2201
85
+ HellaSwag-PL,acc_norm,10042,"1.09, update 25,000",43.9355,42.939699999999995,44.9114
86
+ HellaSwag-PL,acc_norm,10042,"1.09, update 30,000",45.7479,44.771699999999996,46.7337
87
+ HellaSwag-PL,acc_norm,10042,"1.09, update 35,000",46.634100000000004,45.6779,47.63
88
+ HellaSwag-PL,acc_norm,10042,"1.09, update 40,000",46.6142,45.648300000000006,47.5702
89
+ HellaSwag-PL,acc_norm,10042,"1.09, update 45,000",46.783500000000004,45.7877,47.7495
90
+ MMLU-PL,acc,14042,1.07,24.0137,23.3015,24.7116
91
+ MMLU-PL,acc,14042,1.08,23.038,22.3401,23.743100000000002
92
+ MMLU-PL,acc,14042,"1.09, update 5,000",24.0991,23.387,24.8184
93
+ MMLU-PL,acc,14042,"1.09, update 10,000",23.3229,22.625,24.0279
94
+ MMLU-PL,acc,14042,"1.09, update 15,000",23.500899999999998,22.7959,24.206
95
+ MMLU-PL,acc,14042,"1.09, update 20,000",23.8641,23.1662,24.5763
96
+ MMLU-PL,acc,14042,"1.09, update 25,000",24.262900000000002,23.572100000000002,24.9608
97
+ MMLU-PL,acc,14042,"1.09, update 30,000",24.7116,23.985200000000003,25.4237
98
+ MMLU-PL,acc,14042,"1.09, update 35,000",23.2802,22.5823,23.9709
99
+ MMLU-PL,acc,14042,"1.09, update 40,000",24.2416,23.5437,24.968
100
+ MMLU-PL,acc,14042,"1.09, update 45,000",25.7371,25.0247,26.4635
101
+ GSM8K,"strict exact match, 5-shot",1319,1.07,1.5163002274450341,0.9097801364670205,2.1986353297952994
102
+ GSM8K,"strict exact match, 5-shot",1319,1.08,0.9097801364670205,0.45489006823351025,1.4404852160727823
103
+ GSM8K,"strict exact match, 5-shot",1319,"1.09, update 10,000",0.9855951478392722,0.45489006823351025,1.5163002274450341
104
+ SST-2,"accuracy, 0-shot",872,1.07,51.03211009174312,47.706422018348626,54.357798165137616
105
+ SST-2,"accuracy, 0-shot",872,1.08,50.45871559633027,47.13302752293578,53.669724770642205
106
+ SST-2,"accuracy, 0-shot",872,"1.09, update 10,000",58.256880733944946,54.93119266055045,61.353211009174316
plots/boolq.png ADDED

Git LFS Details

  • SHA256: 6bdaaeb1839066f13009541f295ac8e33197b9e73b29cc2a8b750b4237d4ec55
  • Pointer size: 131 Bytes
  • Size of remote file: 157 kB
plots/hellaswag-pl.png ADDED

Git LFS Details

  • SHA256: fbcaa8fbceccd8c9278df70da99d205afd4b02003a31bf59fb11a94c6ce860b5
  • Pointer size: 131 Bytes
  • Size of remote file: 144 kB
plots/hellaswag.png ADDED

Git LFS Details

  • SHA256: 3e0d407f089e25179a2f9cdae9067dc5cac4d21146a9a916b01d7b8680d81d93
  • Pointer size: 131 Bytes
  • Size of remote file: 132 kB
plots/mmlu-pl.png ADDED

Git LFS Details

  • SHA256: 8e7323ebe67781000e4736e65e0d97cf26dcb4e91d110eeeda7c4e1728921075
  • Pointer size: 131 Bytes
  • Size of remote file: 158 kB
plots/openbookqa.png ADDED

Git LFS Details

  • SHA256: 525bb714f61a2ccdac4bc0d5dd3478a646b5a27f356c749d8646d47835796da5
  • Pointer size: 131 Bytes
  • Size of remote file: 148 kB
plots/piqa.png ADDED

Git LFS Details

  • SHA256: ab8f39020b354917450905723e116cb741f7a38ee753a77e6bae593dd1dbd1a6
  • Pointer size: 131 Bytes
  • Size of remote file: 134 kB
plots/plot_benchmarks.py ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+ import csv
3
+ import re
4
+ import json
5
+ import matplotlib
6
+ matplotlib.use('Agg')
7
+ import matplotlib.pyplot as plt
8
+ from matplotlib.ticker import MaxNLocator
9
+
10
+ ROOT = Path(__file__).resolve().parent
11
+ card_path = ROOT / 'model-card.md'
12
+ if not card_path.exists():
13
+ card_path = ROOT.parent / 'README.md'
14
+ lines = card_path.read_text().splitlines()
15
+ records = []
16
+ groups = []
17
+ for i, line in enumerate(lines):
18
+ if not line.startswith('| Task | Metric'):
19
+ continue
20
+ headers = [s.strip() for s in line.strip('|').split('|')][3:]
21
+ tasks = []
22
+ for row in lines[i+2:]:
23
+ if not row.startswith('|'):
24
+ break
25
+ cells = [s.strip() for s in row.strip('|').split('|')]
26
+ task, metric, n = cells[:3]
27
+ values = []
28
+ for label, cell in zip(headers, cells[3:]):
29
+ match = re.fullmatch(r'([\d.]+) \[([\d.]+), ([\d.]+)\]', cell)
30
+ assert match, cell
31
+ score, low, high = map(float, match.groups())
32
+ assert low <= score <= high
33
+ values.append((score, low, high))
34
+ records.append(dict(task=task, metric=metric, n=int(n.replace(',', '')), model=label, score=score, ci95_low=low, ci95_high=high))
35
+ assert len(values) == len(headers)
36
+ tasks.append((task, metric, n, values))
37
+ groups.append((headers, tasks))
38
+
39
+ json_path = ROOT / 'EVALUATION.json'
40
+ if not json_path.exists():
41
+ json_path = ROOT.parent / 'EVALUATION.json'
42
+ aggregate = json.loads(json_path.read_text())
43
+ task_ids = list(aggregate['protocol']['primary_metrics'])
44
+ model_ids = list(aggregate['models'])
45
+ records = []
46
+ for group_index, (headers, tasks) in enumerate(groups):
47
+ for task_index, (name, metric, n, rounded) in enumerate(tasks):
48
+ exact = []
49
+ for label, (score, low, high) in zip(headers, rounded):
50
+ model_id = label.replace('1.09, update ', '1.09-k').replace(',', '')
51
+ if group_index == 0:
52
+ task_id = task_ids[task_index]
53
+ item = aggregate['models'][model_id]['tasks'][task_id]
54
+ values = (item[metric], *item[metric + '_ci95'])
55
+ count = item['n']
56
+ else:
57
+ task_id = ['gsm8k', 'sst2'][task_index]
58
+ item = next(iter(aggregate['additional_measurements'].values()))['models'][model_id]['tasks'][task_id]
59
+ values = (item['value'], item['ci95']['lower'], item['ci95']['upper'])
60
+ count = item['n_scored']
61
+ values = tuple(v * 100 for v in values)
62
+ assert all(abs(a-b) <= .0051 for a,b in zip(values, (score,low,high))), (name,label,values)
63
+ assert count == int(n.replace(',', ''))
64
+ exact.append(values)
65
+ records.append(dict(task=name, metric=metric, n=count, model=label, score=values[0], ci95_low=values[1], ci95_high=values[2]))
66
+ tasks[task_index] = (name, metric, n, exact)
67
+ print('Verified all 105 table results and intervals against EVALUATION.json; plotting JSON precision.')
68
+
69
+ plt.rcParams.update({'font.family': 'DejaVu Sans', 'font.size': 11, 'axes.spines.top': False, 'axes.spines.right': False, 'axes.titleweight': 'bold', 'savefig.facecolor': 'white'})
70
+ colors = ['#2563eb', '#0891b2', '#7c3aed', '#d97706', '#db2777', '#059669', '#9333ea', '#0284c7', '#475569']
71
+
72
+ def draw(ax, headers, task, color, compact=False):
73
+ name, metric, n, values = task
74
+ x = list(range(len(headers)))
75
+ scores, lows, highs = zip(*values)
76
+ ax.fill_between(x, lows, highs, color=color, alpha=.12, linewidth=0)
77
+ ax.errorbar(x, scores, yerr=[[s-l for s,l in zip(scores,lows)], [h-s for s,h in zip(scores,highs)]], fmt='o-', color=color, lw=2, ms=4.5, capsize=3, elinewidth=1)
78
+ ax.set_title(f'{name} · n={n}', loc='left', fontsize=12 if compact else 15)
79
+ ax.set_ylabel(f'{metric} (%)')
80
+ labels = [h.replace('1.09, update ', '1.09\n') for h in headers]
81
+ ax.set_xticks(x, labels, rotation=50 if compact else 35, ha='right', fontsize=8 if compact else 10)
82
+ ax.grid(axis='y', alpha=.2)
83
+ ax.yaxis.set_major_locator(MaxNLocator(nbins=5))
84
+ ax.set_xlim(-.4, len(x)-.35)
85
+ ax.annotate(f'{scores[-1]:.2f}', (x[-1], scores[-1]), xytext=(4, 7), textcoords='offset points', color=color, fontsize=9, weight='bold')
86
+
87
+ headers, tasks = groups[0]
88
+ assert len(tasks) == 9 and len(headers) == 11
89
+ fig, axes = plt.subplots(3, 3, figsize=(18, 13))
90
+ for ax, task, color in zip(axes.flat, tasks, colors):
91
+ draw(ax, headers, task, color, True)
92
+ fig.suptitle('Slayer1B · benchmark scores across model versions', fontsize=23, weight='bold', x=.06, ha='left', y=.98)
93
+ fig.text(.06, .945, '1.07 → 1.08 → 1.09 checkpoints at 5,000–45,000 updates · zero-shot · float32', fontsize=12, color='#475569')
94
+ fig.text(.06, .025, 'Bars and shading: reported pointwise 95% bootstrap CIs. Equally spaced model/checkpoint categories; panel y-scales differ.\nSource: SlayerLab/Slayer1B-1.09 model card · retrieved 16 Nov 2026 · descriptive results; lineage contamination checks incomplete.', fontsize=10, color='#475569')
95
+ fig.subplots_adjust(top=.90, bottom=.12, hspace=.58, wspace=.28)
96
+ for ext in ('png', 'svg', 'pdf'):
97
+ fig.savefig(ROOT / f'benchmark-overview.{ext}', dpi=200)
98
+ plt.close(fig)
99
+
100
+ for idx, task in enumerate(tasks):
101
+ fig, ax = plt.subplots(figsize=(11, 5.8))
102
+ draw(ax, headers, task, colors[idx])
103
+ ax.set_xlabel('Model version / 1.09 continuation update (equally spaced categories)')
104
+ fig.text(.08, .02, 'Reported pointwise 95% bootstrap CIs · source: SlayerLab/Slayer1B-1.09 · 16 Nov 2026', fontsize=9, color='#475569')
105
+ fig.tight_layout(rect=(0,.05,1,1))
106
+ slug = re.sub(r'[^a-z0-9]+', '-', task[0].lower()).strip('-')
107
+ fig.savefig(ROOT / f'{slug}.png', dpi=200)
108
+ plt.close(fig)
109
+
110
+ headers2, tasks2 = groups[1]
111
+ fig, axes = plt.subplots(1, 2, figsize=(12, 5.5))
112
+ for ax, task, color in zip(axes, tasks2, colors):
113
+ draw(ax, headers2, task, color)
114
+ fig.suptitle('Slayer1B · additional benchmarks (separate evaluation protocol)', fontsize=16, weight='bold')
115
+ fig.text(.06, .025, 'Reported pointwise 95% CIs · GSM8K: 5-shot strict exact match; SST-2: zero-shot accuracy.\nOnly the three published checkpoints are shown. Source: model card, 16 Nov 2026.', fontsize=10, color='#475569')
116
+ fig.tight_layout(rect=(0,.12,1,.92))
117
+ fig.savefig(ROOT / 'additional-benchmarks.png', dpi=200)
118
+ fig.savefig(ROOT / 'additional-benchmarks.svg')
119
+ plt.close(fig)
120
+ with (ROOT / 'benchmark-scores.csv').open('w') as f:
121
+ writer = csv.DictWriter(f, fieldnames=list(records[0]))
122
+ writer.writeheader()
123
+ writer.writerows(records)
124
+ print(f'Created overview, 9 individual plots, 2-task supplemental plot, and CSV ({len(records)} results).')
plots/winogrande.png ADDED

Git LFS Details

  • SHA256: c4a2333e5aab25e9c9b1a1f846c9b85112afae16ec9a523f9459a1ee4a47d1ea
  • Pointer size: 131 Bytes
  • Size of remote file: 137 kB