BlockArtica commited on
Commit
5c5797d
·
verified ·
1 Parent(s): b079e9d

model card: add MMLU/GSM8K general benchmarks (base vs adapter) from the candle harness

Browse files
Files changed (1) hide show
  1. README.md +23 -3
README.md CHANGED
@@ -64,9 +64,29 @@ The adapter **improves every active domain with zero regressions.**
64
  Domains helped: 9 / 9. Domains hurt: 0. A held-out CE regression guard (ceiling = base + 0.15)
65
  was active for the whole run and never tripped, so the base capability is provably intact.
66
 
67
- > Honesty note: these are domain-CE deltas on the Aether holdout, not general-benchmark
68
- > (MMLU/HumanEval) numbers. The claim is narrow and measured: on the Aether knowledge domains the
69
- > 10-Sephirot adapter is a consistent, regression-free improvement over the raw 7B base.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
 
71
  ## Training
72
 
 
64
  Domains helped: 9 / 9. Domains hurt: 0. A held-out CE regression guard (ceiling = base + 0.15)
65
  was active for the whole run and never tripped, so the base capability is provably intact.
66
 
67
+ > The numbers above are domain-CE deltas on the Aether holdout. General-benchmark numbers
68
+ > (MMLU, GSM8K) are below.
69
+
70
+ ## General benchmarks (base vs adapter)
71
+
72
+ Off-the-shelf lm-eval cannot load the native candle build, so these were produced by a
73
+ purpose-built candle harness (`aether-v7-eval`) that scores the SAME frozen Q4 weights twice,
74
+ once with the Sephirot adapter active and once with it off. MMLU is multiple-choice
75
+ loglikelihood over the A/B/C/D answer tokens; GSM8K is greedy chain-of-thought generation with
76
+ final-number extraction.
77
+
78
+ | benchmark | n | base | v7.1 (adapter) | change |
79
+ |---|---|---|---|---|
80
+ | MMLU (all subjects) | 14,042 | 71.28% | 71.17% | -0.11 |
81
+ | GSM8K | 625 | 67.8% | 77.8% | +10.0 |
82
+
83
+ Read this the way it reads: **general knowledge is held** (MMLU is flat across the full 57-subject
84
+ set, the regression guard never tripped), and **multi-step reasoning improves** (GSM8K up ~10
85
+ points on a 625-question sample, partly from the adapter following the chain-of-thought and
86
+ final-answer format more reliably). The adapter does not trade away breadth for the domain gains.
87
+
88
+ (GSM8K is a 625-of-1319 sample: the full run is generation-bound on a single 12 GB card and the
89
+ sample is already statistically tight. MMLU is the complete set.)
90
 
91
  ## Training
92