anurag051194 commited on
Commit
1f8ff43
·
verified ·
1 Parent(s): dba8c25

Update TwIL-LM3: weights, tokenizer and model card

Browse files
Files changed (1) hide show
  1. README.md +21 -3
README.md CHANGED
@@ -70,9 +70,27 @@ yet reported for it.
70
  for the Δ table above, hence 0.3466 against 0.3356 — run-to-run variation of the same
71
  checkpoint. The paired run is the correct basis for the improvement claim.
72
 
73
- `strict-7` is the mean of seven lanes scored under strict metrics only (`fol_translation`,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
74
  `entailment_label`, `mcq_answer`, `semantic_parse` and `lean_formalize` exact match,
75
- `lean_critic` and `procedural` accuracy), with no loose-match credit anywhere.
 
 
76
 
77
  Among the released models, TwIL-LM3 leads both headline metrics and wins every lane the pipeline
78
  targets directly: `lean_formalize` token-F1 0.5869 against 0.4655 for the nearest arm,
@@ -91,7 +109,7 @@ better.
91
 
92
  It is also the most efficient arm in the table by a wide margin — 28.1 answers/sec, from
93
  generations averaging 564 tokens where every other arm except Llama runs past 690. The Liquid
94
- models decode faster in raw tokens per second, 25000 and 22000 against 15880, but their length
95
  more than cancels it.
96
 
97
  #### Track B — held-out benchmarks
 
70
  for the Δ table above, hence 0.3466 against 0.3356 — run-to-run variation of the same
71
  checkpoint. The paired run is the correct basis for the improvement claim.
72
 
73
+ The three summary rows aggregate the objective lanes in different ways. None of them include the
74
+ perplexity lanes or the token-F1 scorings, which are not on a common 0–1 accuracy scale.
75
+
76
+ **`macro gate`** is the headline metric and the one the training pipeline gates on. It is the
77
+ equal-weight mean of five objectives: the four bounded classification lanes (`entailment_label`,
78
+ `mcq_answer`, `procedural`, `lean_critic`) plus `rule_induction`, scored by its continuous
79
+ derivation score. Rule induction is included specifically so a fine-tune cannot pass the gate
80
+ while quietly regressing inductive reasoning. In the gate, `mcq_answer` and `procedural` are
81
+ credited as `max(exact_match, loose_match)`: for free-text answer lanes, a response that is
82
+ correct but differently formatted is a formatting artefact rather than a reasoning failure. This
83
+ affects the aggregate only — the per-lane rows above stay strict.
84
+
85
+ **`macro_primary`** is the same mean over the four classification lanes alone, without
86
+ `rule_induction`. It is the narrower "bounded classification" view, kept for comparability with
87
+ earlier reports; the gate is the metric to read for overall in-domain capability.
88
+
89
+ **`strict-7`** is the mean of seven lanes scored under strict metrics only (`fol_translation`,
90
  `entailment_label`, `mcq_answer`, `semantic_parse` and `lean_formalize` exact match,
91
+ `lean_critic` and `procedural` accuracy), with no loose-match credit anywhere. It is deliberately
92
+ harsh — exact match on generative lanes is near zero for every arm — so it is useful for ranking
93
+ models against each other but not as an absolute capability measure.
94
 
95
  Among the released models, TwIL-LM3 leads both headline metrics and wins every lane the pipeline
96
  targets directly: `lean_formalize` token-F1 0.5869 against 0.4655 for the nearest arm,
 
109
 
110
  It is also the most efficient arm in the table by a wide margin — 28.1 answers/sec, from
111
  generations averaging 564 tokens where every other arm except Llama runs past 690. The Liquid
112
+ models decode faster in raw tokens per second, 25230 and 22480 against 15880, but their length
113
  more than cancels it.
114
 
115
  #### Track B — held-out benchmarks