codybum commited on
Commit
5021e67
Β·
verified Β·
1 Parent(s): 9534c44

Card: add representation-flow/spectral wins + disclose T11 demographic bias, T4 rank, Med-HALT FCT negatives; correct 'no bias eval' claim

Browse files
Files changed (1) hide show
  1. README.md +47 -13
README.md CHANGED
@@ -136,18 +136,38 @@ data** (0.3–36 T tokens). Pool of 17 models, 95 ranked metrics. BPB (bits-per-
136
  as a ranked measure.
137
 
138
  **Tally:** **20 outright rank-1 wins Β· 6 best-tied (frontier parity) Β· 9 top-3 Β· 60 trailing.** Wins concentrate on
139
- held-out medical BPB (rank 1/17 mean), attention health, medical entity extraction, and reasoning discrimination.
 
140
 
141
  **Where it wins (rank-1 of 17):** held-out medical BPB (T1 rad / cxr / clin / 5-domain mean **0.4309**), long-context
142
  BPB (T10 L/2, L/4), attention health (T2 collapsed-head frac **0.0506**, entropy-min, BOS-sink), medical entity
143
  extraction (RadGraph DR.1/2/3 macro-F1 **0.7665 / 0.7990 / 0.8150**; BLURB T7 BIOSSES / HoC), reasoning discrimination
144
- (T6b MedThink rank-1 **0.9450**). Calibration T9 mean ECE **0.1366**.
145
-
146
- **Where it is weak (token-volume-bound):** closed-book medical MCQ (T6 β€” near-chance, expected for a 180 B-token base),
147
- long-context needle retrieval (T10), Med-HALT NOTA / FQT, BLURB linear probes at low token count. These scale with raw
148
- pretraining-token volume + instruction tuning β€” a **token-budget** gap, not an architecture flaw; the format-bound ones
149
- (NOTA, FQT, IFEval) are the job of the SFT/RL line
150
- ([KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct)).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
 
152
  ### Apples-to-apples vs BioMedLM (the only non-commercial, biomedical-only peer; ~300 B PubMed tokens)
153
  **KOS-V4-Base wins 28 of 38 comparable cells.** Selected:
@@ -160,12 +180,21 @@ pretraining-token volume + instruction tuning β€” a **token-budget** gap, not an
160
  | T6b MedThink rank-1 | **0.9450** | 0.8250 | +0.120 βœ… |
161
  | T7 BLURB mean / 4 | **0.7465** | 0.6829 | +0.064 βœ… |
162
  | pubmedqa | **0.6980** | 0.4440 | +0.254 βœ… |
 
 
163
  | T9 mean ECE (lower = better) | **0.1366** | 0.2231 | βˆ’0.086 βœ… |
164
- | T8 FQT (Med-HALT) | 0.3868 | **0.7054** | βˆ’0.319 βœ— |
165
  | T8 NOTA (Med-HALT) | 0.6600 | **0.8420** | βˆ’0.182 βœ— |
 
 
 
166
 
167
- Losses concentrate on corpus-volume- / format-bound axes (Med-HALT NOTA / FQT, long-form MCQ). BioMedLM is also
168
- architecturally capped at 1024 context (GPT-2 learned positions); KOS-V4-Base (RoPE, 24,576) holds the long-context axis.
 
 
 
 
169
 
170
  ## Compute footprint
171
  Pre-training: 24Γ— H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours.
@@ -185,5 +214,10 @@ Pre-training: 24Γ— H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours.
185
  - **Not a knowledge-QA model.** As a base LM it does not reliably recall parametric medical facts (closed-book MCQ is
186
  near-chance) β€” fine-tune and/or ground it with retrieval.
187
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
188
- - **No safety or bias evaluation.** This model has not been red-teamed, nor evaluated for toxicity, clinical bias, or
189
- hallucination rates. It may produce harmful, biased, or medically inaccurate content.
 
 
 
 
 
 
136
  as a ranked measure.
137
 
138
  **Tally:** **20 outright rank-1 wins Β· 6 best-tied (frontier parity) Β· 9 top-3 Β· 60 trailing.** Wins concentrate on
139
+ held-out medical BPB (rank 1/17 mean), attention health, internal representation / spectral geometry, medical entity
140
+ extraction, and reasoning discrimination.
141
 
142
  **Where it wins (rank-1 of 17):** held-out medical BPB (T1 rad / cxr / clin / 5-domain mean **0.4309**), long-context
143
  BPB (T10 L/2, L/4), attention health (T2 collapsed-head frac **0.0506**, entropy-min, BOS-sink), medical entity
144
  extraction (RadGraph DR.1/2/3 macro-F1 **0.7665 / 0.7990 / 0.8150**; BLURB T7 BIOSSES / HoC), reasoning discrimination
145
+ (T6b MedThink rank-1 **0.9450**), calibration (T9 mean ECE **0.1366**). Notably, the **clean-tokenizer + clean-corpus
146
+ recipe produces healthy internals with no geometric regularizer**: logit-lens decodability rises through depth (T3
147
+ final-lens accuracy **0.5715** vs BioMedLM 0.2903 β€” a **top-3** placement) and the weight-matrix spectra are mature
148
+ (T5 WeightWatcher median power-law Ξ± wins; safe / under-trained layer ratios place **top-3**). Beyond the 20 outright
149
+ wins, KOS-V4-Base reaches **frontier parity (best-tied) on 6 metrics** (near-zero dead-neuron rate; saturated needle
150
+ retrieval at several depths) and **top-3 on 9 more**.
151
+
152
+ **Where it is weak:**
153
+ - **Closed-book medical MCQ** (T6 β€” near-chance): as a 180 B-token base it does not reliably recall parametric medical
154
+ facts; it is worst on the hardest professional/college splits (mmlu_professional_medicine **0.2096**,
155
+ mmlu_college_medicine **0.2312**, both below BioMedLM). Token-volume + instruction-tuning bound.
156
+ - **Long-context needle retrieval** (T10): pool-relative trailing except at the saturated depths (it does beat BioMedLM
157
+ on the 3-depth mean, **0.8667 vs 0.3667**). Token-volume bound.
158
+ - **Hallucination discipline** (Med-HALT T8 **FCT 0.0400 / NOTA 0.6600 / FQT 0.3868**): all three trail the specialist
159
+ BioMedLM (0.162 / 0.842 / 0.705) β€” a real-PubMed-format, instruction-format gap the SFT/RL line addresses.
160
+ - **Effective-rank geometry** (T4 RankMe **170.8** vs pool leaders' ~187) and the **byte-level tokenizer single-token
161
+ rate** (T1b **0.0448** vs 0.0572): the model trades a little representational rank / single-token coverage for a
162
+ smaller, byte-efficient vocabulary. (Anisotropy / isotropy also run high β€” a known BBPE base characteristic.)
163
+ - **Demographic bias** (T11 CrowS-Pairs |disparity| **1.0791** β€” **rank 9/17, mid-pack, and notably *more* biased than
164
+ the biomedical specialists** BioMedLM 0.3705 / MedGemma 0.2978): the clean-corpus recipe did **not** remove
165
+ social-stereotype bias. Treat generations as **unaudited for fairness**.
166
+
167
+ The token- and format-bound axes (T6 knowledge, T10 needle, Med-HALT NOTA / FQT, IFEval) scale with raw pretraining-token
168
+ volume and instruction tuning β€” a **token-budget** gap rather than an architecture flaw; the format-bound ones are the
169
+ job of the SFT/RL line ([KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct)). Bias and
170
+ effective-rank are **not** simply token-budget artifacts and remain open limitations.
171
 
172
  ### Apples-to-apples vs BioMedLM (the only non-commercial, biomedical-only peer; ~300 B PubMed tokens)
173
  **KOS-V4-Base wins 28 of 38 comparable cells.** Selected:
 
180
  | T6b MedThink rank-1 | **0.9450** | 0.8250 | +0.120 βœ… |
181
  | T7 BLURB mean / 4 | **0.7465** | 0.6829 | +0.064 βœ… |
182
  | pubmedqa | **0.6980** | 0.4440 | +0.254 βœ… |
183
+ | T3 final-lens accuracy (higher = better) | **0.5715** | 0.2903 | +0.281 βœ… |
184
+ | T10 needle 3-depth mean (higher = better) | **0.8667** | 0.3667 | +0.500 βœ… |
185
  | T9 mean ECE (lower = better) | **0.1366** | 0.2231 | βˆ’0.086 βœ… |
186
+ | T8 FCT (Med-HALT) | 0.0400 | **0.1620** | βˆ’0.122 βœ— |
187
  | T8 NOTA (Med-HALT) | 0.6600 | **0.8420** | βˆ’0.182 βœ— |
188
+ | T8 FQT (Med-HALT) | 0.3868 | **0.7054** | βˆ’0.319 βœ— |
189
+ | T4 RankMe effective rank (higher = better) | 170.8 | **186.8** | βˆ’16.0 βœ— |
190
+ | T11 CrowS-Pairs \|disparity\| (lower = better) | 1.0791 | **0.3705** | +0.709 βœ— |
191
 
192
+ **KOS-V4-Base wins 28 of 38 comparable cells.** Wins concentrate on attention pathology (T2), representation flow (T3),
193
+ spectral health (T5), radiology entity extraction (Block C), and chemical/disease NER. **Losses** concentrate on
194
+ corpus-volume- / instruction-format-bound axes (Med-HALT FCT / NOTA / FQT, long-form closed-book MCQ, BLURB linear
195
+ probes at low token count) plus two that are **not** token-budget artifacts: lower effective rank (T4) and **higher
196
+ demographic-bias disparity (T11)**. BioMedLM is architecturally capped at 1024 context (GPT-2 learned positions);
197
+ KOS-V4-Base (RoPE, 24,576) holds the long-context axis.
198
 
199
  ## Compute footprint
200
  Pre-training: 24Γ— H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours.
 
214
  - **Not a knowledge-QA model.** As a base LM it does not reliably recall parametric medical facts (closed-book MCQ is
215
  near-chance) β€” fine-tune and/or ground it with retrieval.
216
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
217
+ - **Measured demographic bias.** On CrowS-Pairs (T11) the model is **mid-pack (rank 9/17) and more biased than the
218
+ biomedical specialists** (|disparity| 1.0791 vs BioMedLM 0.37 / MedGemma 0.30). The clean-corpus recipe removes
219
+ structural sink tokens but does **not** remove social-stereotype bias β€” outputs are **unaudited for fairness**.
220
+ - **Measured hallucination weakness.** Med-HALT (T8 FCT / NOTA / FQT) trails the specialist BioMedLM; the model may
221
+ fabricate confidently.
222
+ - **Not otherwise safety-tested.** Beyond CrowS-Pairs and Med-HALT above, this model has **not** been red-teamed or
223
+ evaluated for toxicity or clinical safety. It may produce harmful, biased, or medically inaccurate content.