Add benchmark plots generated from EVALUATION.json
Browse files- .gitattributes +11 -0
- README.md +14 -0
- plots/additional-benchmarks.png +3 -0
- plots/additional-benchmarks.svg +2730 -0
- plots/arc-challenge.png +3 -0
- plots/belebele-polish.png +3 -0
- plots/benchmark-overview.pdf +0 -0
- plots/benchmark-overview.png +3 -0
- plots/benchmark-overview.svg +0 -0
- plots/benchmark-scores.csv +106 -0
- plots/boolq.png +3 -0
- plots/hellaswag-pl.png +3 -0
- plots/hellaswag.png +3 -0
- plots/mmlu-pl.png +3 -0
- plots/openbookqa.png +3 -0
- plots/piqa.png +3 -0
- plots/plot_benchmarks.py +124 -0
- plots/winogrande.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,14 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
plots/additional-benchmarks.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
plots/arc-challenge.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
plots/belebele-polish.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
plots/benchmark-overview.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
plots/boolq.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
plots/hellaswag-pl.png filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
plots/hellaswag.png filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
plots/mmlu-pl.png filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
plots/openbookqa.png filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
plots/piqa.png filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
plots/winogrande.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -167,6 +167,16 @@ MMLU-PL and Belebele-PL score the answer letter A–D given listed options; ARC-
|
|
| 167 |
|
| 168 |
There is no replay-only control for this continuation, and initialization, exposure and schedule differ between checkpoints. These measurements do not isolate a causal effect of the new datasets or quantify training-seed variance. Earlier internal diagnostics are separate historical records, not a criterion for naming or publishing this model.
|
| 169 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
### Auxiliary class-balanced diagnostic
|
| 171 |
|
| 172 |
Balanced accuracy is the unweighted mean of recall across gold-answer classes. It was collected after the primary measurements. The values below are descriptive; their source does not provide confidence intervals. They retain a separate method and source digest in [EVALUATION.json](EVALUATION.json).
|
|
@@ -198,6 +208,10 @@ Scores are percentages with reported pointwise 95% bootstrap intervals (10,000 d
|
|
| 198 |
|
| 199 |
**Descriptive results; training-lineage contamination scan pending.** GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.
|
| 200 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
## Cost, licensing and limitations
|
| 202 |
|
| 203 |
Continued training is budget-limited and may pause with spot availability. The funding receipt above is not the current balance or a guarantee of a final update count. Metered attribution is not a settled invoice; hardware choice must account for effective retained progress, interruptions, storage and evaluation, not only an advertised GPU-hour price. This page does not allocate a new budget to gold-data training or change the existing cost guards.
|
|
|
|
| 167 |
|
| 168 |
There is no replay-only control for this continuation, and initialization, exposure and schedule differ between checkpoints. These measurements do not isolate a causal effect of the new datasets or quantify training-seed variance. Earlier internal diagnostics are separate historical records, not a criterion for naming or publishing this model.
|
| 169 |
|
| 170 |
+
### Benchmark plots
|
| 171 |
+
|
| 172 |
+

|
| 173 |
+
|
| 174 |
+
Each panel shows one benchmark from 1.07 and 1.08 through the measured 1.09 checkpoints at updates 5,000–45,000. Error bars and shading show the reported pointwise 95% bootstrap confidence intervals. The x-axis uses equally spaced checkpoint categories; y-axis ranges differ across panels. These intervals describe individual scores, not paired differences or training-seed variance. Training-lineage contamination checks remain incomplete.
|
| 175 |
+
|
| 176 |
+
The plots use the full-precision values in [EVALUATION.json](EVALUATION.json), selecting the primary metric declared for each task and converting fractions to percentages. Download the overview as [SVG](plots/benchmark-overview.svg) or [PDF](plots/benchmark-overview.pdf), the [plotted data as CSV](plots/benchmark-scores.csv), or the [reproducible plotting script](plots/plot_benchmarks.py) (Python with Matplotlib; run from a repository checkout).
|
| 177 |
+
|
| 178 |
+
Individual plots: [ARC-Challenge](plots/arc-challenge.png), [HellaSwag](plots/hellaswag.png), [PIQA](plots/piqa.png), [WinoGrande](plots/winogrande.png), [BoolQ](plots/boolq.png), [OpenBookQA](plots/openbookqa.png), [Belebele-PL](plots/belebele-polish.png), [HellaSwag-PL](plots/hellaswag-pl.png), [MMLU-PL](plots/mmlu-pl.png).
|
| 179 |
+
|
| 180 |
### Auxiliary class-balanced diagnostic
|
| 181 |
|
| 182 |
Balanced accuracy is the unweighted mean of recall across gold-answer classes. It was collected after the primary measurements. The values below are descriptive; their source does not provide confidence intervals. They retain a separate method and source digest in [EVALUATION.json](EVALUATION.json).
|
|
|
|
| 208 |
|
| 209 |
**Descriptive results; training-lineage contamination scan pending.** GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.
|
| 210 |
|
| 211 |
+

|
| 212 |
+
|
| 213 |
+
The plot retains this separate evaluation protocol and shows only the three published measurements, with their reported pointwise 95% intervals. [Download SVG](plots/additional-benchmarks.svg).
|
| 214 |
+
|
| 215 |
## Cost, licensing and limitations
|
| 216 |
|
| 217 |
Continued training is budget-limited and may pause with spot availability. The funding receipt above is not the current balance or a guarantee of a final update count. Metered attribution is not a settled invoice; hardware choice must account for effective retained progress, interruptions, storage and evaluation, not only an advertised GPU-hour price. This page does not allocate a new budget to gold-data training or change the existing cost guards.
|
plots/additional-benchmarks.png
ADDED
|
Git LFS Details
|
plots/additional-benchmarks.svg
ADDED
|
|
plots/arc-challenge.png
ADDED
|
Git LFS Details
|
plots/belebele-polish.png
ADDED
|
Git LFS Details
|
plots/benchmark-overview.pdf
ADDED
|
Binary file (53.1 kB). View file
|
|
|
plots/benchmark-overview.png
ADDED
|
Git LFS Details
|
plots/benchmark-overview.svg
ADDED
|
|
plots/benchmark-scores.csv
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
task,metric,n,model,score,ci95_low,ci95_high
|
| 2 |
+
ARC-Challenge,acc_norm,1172,1.07,27.3038,24.744,29.8635
|
| 3 |
+
ARC-Challenge,acc_norm,1172,1.08,30.034100000000002,27.3891,32.6792
|
| 4 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 5,000",29.3515,26.7918,31.996599999999997
|
| 5 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 10,000",30.4608,27.8157,33.1058
|
| 6 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 15,000",32.5085,29.8635,35.2389
|
| 7 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 20,000",33.1058,30.4608,35.8362
|
| 8 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 25,000",33.7884,31.058000000000003,36.5188
|
| 9 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 30,000",33.020500000000006,30.2901,35.6655
|
| 10 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 35,000",35.153600000000004,32.4232,37.884
|
| 11 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 40,000",34.9829,32.2526,37.7986
|
| 12 |
+
ARC-Challenge,acc_norm,1172,"1.09, update 45,000",34.8123,32.081900000000005,37.5427
|
| 13 |
+
HellaSwag,acc_norm,10042,1.07,43.3778,42.431799999999996,44.3438
|
| 14 |
+
HellaSwag,acc_norm,10042,1.08,43.4176,42.431799999999996,44.3935
|
| 15 |
+
HellaSwag,acc_norm,10042,"1.09, update 5,000",46.5346,45.5487,47.5204
|
| 16 |
+
HellaSwag,acc_norm,10042,"1.09, update 10,000",48.9345,47.9685,49.910399999999996
|
| 17 |
+
HellaSwag,acc_norm,10042,"1.09, update 15,000",49.771,48.795100000000005,50.7568
|
| 18 |
+
HellaSwag,acc_norm,10042,"1.09, update 20,000",50.3784,49.392599999999995,51.354299999999995
|
| 19 |
+
HellaSwag,acc_norm,10042,"1.09, update 25,000",51.165099999999995,50.1792,52.141000000000005
|
| 20 |
+
HellaSwag,acc_norm,10042,"1.09, update 30,000",52.0016,51.015699999999995,52.9775
|
| 21 |
+
HellaSwag,acc_norm,10042,"1.09, update 35,000",52.479600000000005,51.503699999999995,53.4555
|
| 22 |
+
HellaSwag,acc_norm,10042,"1.09, update 40,000",52.798199999999994,51.812400000000004,53.7841
|
| 23 |
+
HellaSwag,acc_norm,10042,"1.09, update 45,000",53.6547,52.6887,54.620599999999996
|
| 24 |
+
PIQA,acc_norm,1838,1.07,66.975,64.7987,69.1513
|
| 25 |
+
PIQA,acc_norm,1838,1.08,66.2133,64.1458,68.33510000000001
|
| 26 |
+
PIQA,acc_norm,1838,"1.09, update 5,000",68.28070000000001,66.2119,70.4026
|
| 27 |
+
PIQA,acc_norm,1838,"1.09, update 10,000",68.988,66.9206,71.10990000000001
|
| 28 |
+
PIQA,acc_norm,1838,"1.09, update 15,000",68.988,66.86619999999999,71.05550000000001
|
| 29 |
+
PIQA,acc_norm,1838,"1.09, update 20,000",70.7835,68.716,72.8509
|
| 30 |
+
PIQA,acc_norm,1838,"1.09, update 25,000",71.5452,69.47630000000001,73.6126
|
| 31 |
+
PIQA,acc_norm,1838,"1.09, update 30,000",70.6746,68.6072,72.7421
|
| 32 |
+
PIQA,acc_norm,1838,"1.09, update 35,000",71.2731,69.1513,73.28620000000001
|
| 33 |
+
PIQA,acc_norm,1838,"1.09, update 40,000",71.2187,69.20429999999999,73.28620000000001
|
| 34 |
+
PIQA,acc_norm,1838,"1.09, update 45,000",70.29379999999999,68.2263,72.3069
|
| 35 |
+
WinoGrande,acc,1267,1.07,53.59120000000001,50.828700000000005,56.3536
|
| 36 |
+
WinoGrande,acc,1267,1.08,53.6701,50.9077,56.4325
|
| 37 |
+
WinoGrande,acc,1267,"1.09, update 5,000",54.9329,52.1705,57.6953
|
| 38 |
+
WinoGrande,acc,1267,"1.09, update 10,000",54.9329,52.1705,57.6164
|
| 39 |
+
WinoGrande,acc,1267,"1.09, update 15,000",56.748200000000004,54.0647,59.4317
|
| 40 |
+
WinoGrande,acc,1267,"1.09, update 20,000",55.0118,52.2494,57.774300000000004
|
| 41 |
+
WinoGrande,acc,1267,"1.09, update 25,000",55.879999999999995,53.117599999999996,58.6425
|
| 42 |
+
WinoGrande,acc,1267,"1.09, update 30,000",57.1429,54.4574,59.8264
|
| 43 |
+
WinoGrande,acc,1267,"1.09, update 35,000",56.748200000000004,54.0647,59.4317
|
| 44 |
+
WinoGrande,acc,1267,"1.09, update 40,000",57.6953,55.0118,60.3788
|
| 45 |
+
WinoGrande,acc,1267,"1.09, update 45,000",59.037099999999995,56.2747,61.720600000000005
|
| 46 |
+
BoolQ,acc,3270,1.07,58.2875,56.6055,59.96939999999999
|
| 47 |
+
BoolQ,acc,3270,1.08,59.96939999999999,58.2569,61.590199999999996
|
| 48 |
+
BoolQ,acc,3270,"1.09, update 5,000",61.8043,60.1827,63.486200000000004
|
| 49 |
+
BoolQ,acc,3270,"1.09, update 10,000",61.1315,59.4495,62.7829
|
| 50 |
+
BoolQ,acc,3270,"1.09, update 15,000",59.8777,58.195699999999995,61.5291
|
| 51 |
+
BoolQ,acc,3270,"1.09, update 20,000",51.40670000000001,49.6942,53.1193
|
| 52 |
+
BoolQ,acc,3270,"1.09, update 25,000",52.2018,50.519099999999995,53.91439999999999
|
| 53 |
+
BoolQ,acc,3270,"1.09, update 30,000",60.6728,58.9908,62.354699999999994
|
| 54 |
+
BoolQ,acc,3270,"1.09, update 35,000",59.8777,58.195699999999995,61.5596
|
| 55 |
+
BoolQ,acc,3270,"1.09, update 40,000",59.1743,57.4618,60.825700000000005
|
| 56 |
+
BoolQ,acc,3270,"1.09, update 45,000",61.9266,60.244600000000005,63.6086
|
| 57 |
+
OpenBookQA,acc_norm,500,1.07,33.6,29.4,37.8
|
| 58 |
+
OpenBookQA,acc_norm,500,1.08,34.8,30.599999999999998,39.0
|
| 59 |
+
OpenBookQA,acc_norm,500,"1.09, update 5,000",35.6,31.4,40.0
|
| 60 |
+
OpenBookQA,acc_norm,500,"1.09, update 10,000",34.4,30.2,38.6
|
| 61 |
+
OpenBookQA,acc_norm,500,"1.09, update 15,000",36.0,31.8,40.2
|
| 62 |
+
OpenBookQA,acc_norm,500,"1.09, update 20,000",35.199999999999996,31.0,39.4
|
| 63 |
+
OpenBookQA,acc_norm,500,"1.09, update 25,000",37.2,32.800000000000004,41.4
|
| 64 |
+
OpenBookQA,acc_norm,500,"1.09, update 30,000",36.199999999999996,32.0,40.400000000000006
|
| 65 |
+
OpenBookQA,acc_norm,500,"1.09, update 35,000",34.4,30.4,38.6
|
| 66 |
+
OpenBookQA,acc_norm,500,"1.09, update 40,000",34.599999999999994,30.4,38.800000000000004
|
| 67 |
+
OpenBookQA,acc_norm,500,"1.09, update 45,000",36.0,31.8,40.2
|
| 68 |
+
"Belebele, Polish",acc,900,1.07,26.0,23.1111,28.8889
|
| 69 |
+
"Belebele, Polish",acc,900,1.08,24.0,21.2222,26.8889
|
| 70 |
+
"Belebele, Polish",acc,900,"1.09, update 5,000",24.1111,21.3333,26.8889
|
| 71 |
+
"Belebele, Polish",acc,900,"1.09, update 10,000",24.1111,21.3333,27.0
|
| 72 |
+
"Belebele, Polish",acc,900,"1.09, update 15,000",22.555600000000002,19.8889,25.333299999999998
|
| 73 |
+
"Belebele, Polish",acc,900,"1.09, update 20,000",24.6667,21.8889,27.555600000000002
|
| 74 |
+
"Belebele, Polish",acc,900,"1.09, update 25,000",25.333299999999998,22.4444,28.222199999999997
|
| 75 |
+
"Belebele, Polish",acc,900,"1.09, update 30,000",24.444399999999998,21.6667,27.333299999999998
|
| 76 |
+
"Belebele, Polish",acc,900,"1.09, update 35,000",24.333299999999998,21.5556,27.1139
|
| 77 |
+
"Belebele, Polish",acc,900,"1.09, update 40,000",25.555600000000002,22.7778,28.444399999999998
|
| 78 |
+
"Belebele, Polish",acc,900,"1.09, update 45,000",25.333299999999998,22.555600000000002,28.222199999999997
|
| 79 |
+
HellaSwag-PL,acc_norm,10042,1.07,43.825900000000004,42.8401,44.792100000000005
|
| 80 |
+
HellaSwag-PL,acc_norm,10042,1.08,43.8956,42.8998,44.8715
|
| 81 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 5,000",44.7321,43.7562,45.708
|
| 82 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 10,000",45.508900000000004,44.513000000000005,46.4848
|
| 83 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 15,000",44.184400000000004,43.2085,45.1603
|
| 84 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 20,000",44.2442,43.2583,45.2201
|
| 85 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 25,000",43.9355,42.939699999999995,44.9114
|
| 86 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 30,000",45.7479,44.771699999999996,46.7337
|
| 87 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 35,000",46.634100000000004,45.6779,47.63
|
| 88 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 40,000",46.6142,45.648300000000006,47.5702
|
| 89 |
+
HellaSwag-PL,acc_norm,10042,"1.09, update 45,000",46.783500000000004,45.7877,47.7495
|
| 90 |
+
MMLU-PL,acc,14042,1.07,24.0137,23.3015,24.7116
|
| 91 |
+
MMLU-PL,acc,14042,1.08,23.038,22.3401,23.743100000000002
|
| 92 |
+
MMLU-PL,acc,14042,"1.09, update 5,000",24.0991,23.387,24.8184
|
| 93 |
+
MMLU-PL,acc,14042,"1.09, update 10,000",23.3229,22.625,24.0279
|
| 94 |
+
MMLU-PL,acc,14042,"1.09, update 15,000",23.500899999999998,22.7959,24.206
|
| 95 |
+
MMLU-PL,acc,14042,"1.09, update 20,000",23.8641,23.1662,24.5763
|
| 96 |
+
MMLU-PL,acc,14042,"1.09, update 25,000",24.262900000000002,23.572100000000002,24.9608
|
| 97 |
+
MMLU-PL,acc,14042,"1.09, update 30,000",24.7116,23.985200000000003,25.4237
|
| 98 |
+
MMLU-PL,acc,14042,"1.09, update 35,000",23.2802,22.5823,23.9709
|
| 99 |
+
MMLU-PL,acc,14042,"1.09, update 40,000",24.2416,23.5437,24.968
|
| 100 |
+
MMLU-PL,acc,14042,"1.09, update 45,000",25.7371,25.0247,26.4635
|
| 101 |
+
GSM8K,"strict exact match, 5-shot",1319,1.07,1.5163002274450341,0.9097801364670205,2.1986353297952994
|
| 102 |
+
GSM8K,"strict exact match, 5-shot",1319,1.08,0.9097801364670205,0.45489006823351025,1.4404852160727823
|
| 103 |
+
GSM8K,"strict exact match, 5-shot",1319,"1.09, update 10,000",0.9855951478392722,0.45489006823351025,1.5163002274450341
|
| 104 |
+
SST-2,"accuracy, 0-shot",872,1.07,51.03211009174312,47.706422018348626,54.357798165137616
|
| 105 |
+
SST-2,"accuracy, 0-shot",872,1.08,50.45871559633027,47.13302752293578,53.669724770642205
|
| 106 |
+
SST-2,"accuracy, 0-shot",872,"1.09, update 10,000",58.256880733944946,54.93119266055045,61.353211009174316
|
plots/boolq.png
ADDED
|
Git LFS Details
|
plots/hellaswag-pl.png
ADDED
|
Git LFS Details
|
plots/hellaswag.png
ADDED
|
Git LFS Details
|
plots/mmlu-pl.png
ADDED
|
Git LFS Details
|
plots/openbookqa.png
ADDED
|
Git LFS Details
|
plots/piqa.png
ADDED
|
Git LFS Details
|
plots/plot_benchmarks.py
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from pathlib import Path
|
| 2 |
+
import csv
|
| 3 |
+
import re
|
| 4 |
+
import json
|
| 5 |
+
import matplotlib
|
| 6 |
+
matplotlib.use('Agg')
|
| 7 |
+
import matplotlib.pyplot as plt
|
| 8 |
+
from matplotlib.ticker import MaxNLocator
|
| 9 |
+
|
| 10 |
+
ROOT = Path(__file__).resolve().parent
|
| 11 |
+
card_path = ROOT / 'model-card.md'
|
| 12 |
+
if not card_path.exists():
|
| 13 |
+
card_path = ROOT.parent / 'README.md'
|
| 14 |
+
lines = card_path.read_text().splitlines()
|
| 15 |
+
records = []
|
| 16 |
+
groups = []
|
| 17 |
+
for i, line in enumerate(lines):
|
| 18 |
+
if not line.startswith('| Task | Metric'):
|
| 19 |
+
continue
|
| 20 |
+
headers = [s.strip() for s in line.strip('|').split('|')][3:]
|
| 21 |
+
tasks = []
|
| 22 |
+
for row in lines[i+2:]:
|
| 23 |
+
if not row.startswith('|'):
|
| 24 |
+
break
|
| 25 |
+
cells = [s.strip() for s in row.strip('|').split('|')]
|
| 26 |
+
task, metric, n = cells[:3]
|
| 27 |
+
values = []
|
| 28 |
+
for label, cell in zip(headers, cells[3:]):
|
| 29 |
+
match = re.fullmatch(r'([\d.]+) \[([\d.]+), ([\d.]+)\]', cell)
|
| 30 |
+
assert match, cell
|
| 31 |
+
score, low, high = map(float, match.groups())
|
| 32 |
+
assert low <= score <= high
|
| 33 |
+
values.append((score, low, high))
|
| 34 |
+
records.append(dict(task=task, metric=metric, n=int(n.replace(',', '')), model=label, score=score, ci95_low=low, ci95_high=high))
|
| 35 |
+
assert len(values) == len(headers)
|
| 36 |
+
tasks.append((task, metric, n, values))
|
| 37 |
+
groups.append((headers, tasks))
|
| 38 |
+
|
| 39 |
+
json_path = ROOT / 'EVALUATION.json'
|
| 40 |
+
if not json_path.exists():
|
| 41 |
+
json_path = ROOT.parent / 'EVALUATION.json'
|
| 42 |
+
aggregate = json.loads(json_path.read_text())
|
| 43 |
+
task_ids = list(aggregate['protocol']['primary_metrics'])
|
| 44 |
+
model_ids = list(aggregate['models'])
|
| 45 |
+
records = []
|
| 46 |
+
for group_index, (headers, tasks) in enumerate(groups):
|
| 47 |
+
for task_index, (name, metric, n, rounded) in enumerate(tasks):
|
| 48 |
+
exact = []
|
| 49 |
+
for label, (score, low, high) in zip(headers, rounded):
|
| 50 |
+
model_id = label.replace('1.09, update ', '1.09-k').replace(',', '')
|
| 51 |
+
if group_index == 0:
|
| 52 |
+
task_id = task_ids[task_index]
|
| 53 |
+
item = aggregate['models'][model_id]['tasks'][task_id]
|
| 54 |
+
values = (item[metric], *item[metric + '_ci95'])
|
| 55 |
+
count = item['n']
|
| 56 |
+
else:
|
| 57 |
+
task_id = ['gsm8k', 'sst2'][task_index]
|
| 58 |
+
item = next(iter(aggregate['additional_measurements'].values()))['models'][model_id]['tasks'][task_id]
|
| 59 |
+
values = (item['value'], item['ci95']['lower'], item['ci95']['upper'])
|
| 60 |
+
count = item['n_scored']
|
| 61 |
+
values = tuple(v * 100 for v in values)
|
| 62 |
+
assert all(abs(a-b) <= .0051 for a,b in zip(values, (score,low,high))), (name,label,values)
|
| 63 |
+
assert count == int(n.replace(',', ''))
|
| 64 |
+
exact.append(values)
|
| 65 |
+
records.append(dict(task=name, metric=metric, n=count, model=label, score=values[0], ci95_low=values[1], ci95_high=values[2]))
|
| 66 |
+
tasks[task_index] = (name, metric, n, exact)
|
| 67 |
+
print('Verified all 105 table results and intervals against EVALUATION.json; plotting JSON precision.')
|
| 68 |
+
|
| 69 |
+
plt.rcParams.update({'font.family': 'DejaVu Sans', 'font.size': 11, 'axes.spines.top': False, 'axes.spines.right': False, 'axes.titleweight': 'bold', 'savefig.facecolor': 'white'})
|
| 70 |
+
colors = ['#2563eb', '#0891b2', '#7c3aed', '#d97706', '#db2777', '#059669', '#9333ea', '#0284c7', '#475569']
|
| 71 |
+
|
| 72 |
+
def draw(ax, headers, task, color, compact=False):
|
| 73 |
+
name, metric, n, values = task
|
| 74 |
+
x = list(range(len(headers)))
|
| 75 |
+
scores, lows, highs = zip(*values)
|
| 76 |
+
ax.fill_between(x, lows, highs, color=color, alpha=.12, linewidth=0)
|
| 77 |
+
ax.errorbar(x, scores, yerr=[[s-l for s,l in zip(scores,lows)], [h-s for s,h in zip(scores,highs)]], fmt='o-', color=color, lw=2, ms=4.5, capsize=3, elinewidth=1)
|
| 78 |
+
ax.set_title(f'{name} · n={n}', loc='left', fontsize=12 if compact else 15)
|
| 79 |
+
ax.set_ylabel(f'{metric} (%)')
|
| 80 |
+
labels = [h.replace('1.09, update ', '1.09\n') for h in headers]
|
| 81 |
+
ax.set_xticks(x, labels, rotation=50 if compact else 35, ha='right', fontsize=8 if compact else 10)
|
| 82 |
+
ax.grid(axis='y', alpha=.2)
|
| 83 |
+
ax.yaxis.set_major_locator(MaxNLocator(nbins=5))
|
| 84 |
+
ax.set_xlim(-.4, len(x)-.35)
|
| 85 |
+
ax.annotate(f'{scores[-1]:.2f}', (x[-1], scores[-1]), xytext=(4, 7), textcoords='offset points', color=color, fontsize=9, weight='bold')
|
| 86 |
+
|
| 87 |
+
headers, tasks = groups[0]
|
| 88 |
+
assert len(tasks) == 9 and len(headers) == 11
|
| 89 |
+
fig, axes = plt.subplots(3, 3, figsize=(18, 13))
|
| 90 |
+
for ax, task, color in zip(axes.flat, tasks, colors):
|
| 91 |
+
draw(ax, headers, task, color, True)
|
| 92 |
+
fig.suptitle('Slayer1B · benchmark scores across model versions', fontsize=23, weight='bold', x=.06, ha='left', y=.98)
|
| 93 |
+
fig.text(.06, .945, '1.07 → 1.08 → 1.09 checkpoints at 5,000–45,000 updates · zero-shot · float32', fontsize=12, color='#475569')
|
| 94 |
+
fig.text(.06, .025, 'Bars and shading: reported pointwise 95% bootstrap CIs. Equally spaced model/checkpoint categories; panel y-scales differ.\nSource: SlayerLab/Slayer1B-1.09 model card · retrieved 16 Nov 2026 · descriptive results; lineage contamination checks incomplete.', fontsize=10, color='#475569')
|
| 95 |
+
fig.subplots_adjust(top=.90, bottom=.12, hspace=.58, wspace=.28)
|
| 96 |
+
for ext in ('png', 'svg', 'pdf'):
|
| 97 |
+
fig.savefig(ROOT / f'benchmark-overview.{ext}', dpi=200)
|
| 98 |
+
plt.close(fig)
|
| 99 |
+
|
| 100 |
+
for idx, task in enumerate(tasks):
|
| 101 |
+
fig, ax = plt.subplots(figsize=(11, 5.8))
|
| 102 |
+
draw(ax, headers, task, colors[idx])
|
| 103 |
+
ax.set_xlabel('Model version / 1.09 continuation update (equally spaced categories)')
|
| 104 |
+
fig.text(.08, .02, 'Reported pointwise 95% bootstrap CIs · source: SlayerLab/Slayer1B-1.09 · 16 Nov 2026', fontsize=9, color='#475569')
|
| 105 |
+
fig.tight_layout(rect=(0,.05,1,1))
|
| 106 |
+
slug = re.sub(r'[^a-z0-9]+', '-', task[0].lower()).strip('-')
|
| 107 |
+
fig.savefig(ROOT / f'{slug}.png', dpi=200)
|
| 108 |
+
plt.close(fig)
|
| 109 |
+
|
| 110 |
+
headers2, tasks2 = groups[1]
|
| 111 |
+
fig, axes = plt.subplots(1, 2, figsize=(12, 5.5))
|
| 112 |
+
for ax, task, color in zip(axes, tasks2, colors):
|
| 113 |
+
draw(ax, headers2, task, color)
|
| 114 |
+
fig.suptitle('Slayer1B · additional benchmarks (separate evaluation protocol)', fontsize=16, weight='bold')
|
| 115 |
+
fig.text(.06, .025, 'Reported pointwise 95% CIs · GSM8K: 5-shot strict exact match; SST-2: zero-shot accuracy.\nOnly the three published checkpoints are shown. Source: model card, 16 Nov 2026.', fontsize=10, color='#475569')
|
| 116 |
+
fig.tight_layout(rect=(0,.12,1,.92))
|
| 117 |
+
fig.savefig(ROOT / 'additional-benchmarks.png', dpi=200)
|
| 118 |
+
fig.savefig(ROOT / 'additional-benchmarks.svg')
|
| 119 |
+
plt.close(fig)
|
| 120 |
+
with (ROOT / 'benchmark-scores.csv').open('w') as f:
|
| 121 |
+
writer = csv.DictWriter(f, fieldnames=list(records[0]))
|
| 122 |
+
writer.writeheader()
|
| 123 |
+
writer.writerows(records)
|
| 124 |
+
print(f'Created overview, 9 individual plots, 2-task supplemental plot, and CSV ({len(records)} results).')
|
plots/winogrande.png
ADDED
|
Git LFS Details
|