codybum commited on
Commit
84feacf
Β·
verified Β·
1 Parent(s): 68ad787

Card: add radiology/clinical home-turf win (rank 1/17 per-domain BPB vs trillion-token models) + tokenizer byte-efficiency win (rank 4/17); remove forward reference

Browse files
Files changed (1) hide show
  1. README.md +18 -1
README.md CHANGED
@@ -87,6 +87,14 @@ other special tokens** β€” no chat / ChatML tokens; the base was not trained on
87
  residual byte-fragment "sink" merges present at pretraining have been **removed (corrected tokenizer)**; the
88
  vocabulary stays 32,000. BPB is per-byte, so this does not confound cross-model comparison.
89
 
 
 
 
 
 
 
 
 
90
  ## Pre-training
91
  Trained **from scratch, not distilled or continued**. Pure next-token cross-entropy (**no auxiliary losses**),
92
  full-parameter on the LlamaFactory trainer, AdamW, peak LR 3.0e-4 cosine (warmup 1%), grad-clip 1.0,
@@ -125,7 +133,7 @@ kentucky_health web).
125
  license verification; they will be added once resolved.)*
126
 
127
  **Disclosed issue:** ~35% of tokens are duplicates (a FineWeb-Edu sharding build bug + PMC repetition); a deduped
128
- corpus (`v4_dedup`, 79.4 M docs, 0% dup) is ready but was **not** trained (it seeded the next base, V5).
129
 
130
  **Governance:** includes PhysioNet-credentialed, de-identified, redistribution-restricted clinical sources (MIMIC-IV
131
  discharge / radiology, MIMIC-CXR) β€” any downstream use must confirm PhysioNet DUA compliance.
@@ -149,6 +157,15 @@ final-lens accuracy **0.5715** vs BioMedLM 0.2903 β€” a **top-3** placement) and
149
  wins, KOS-V4-Base reaches **frontier parity (best-tied) on 6 metrics** (near-zero dead-neuron rate; saturated needle
150
  retrieval at several depths) and **top-3 on 9 more**.
151
 
 
 
 
 
 
 
 
 
 
152
  **Where it is weak:**
153
  - **Closed-book medical MCQ** (T6 β€” near-chance): as a 180 B-token base it does not reliably recall parametric medical
154
  facts; it is worst on the hardest professional/college splits (mmlu_professional_medicine **0.2096**,
 
87
  residual byte-fragment "sink" merges present at pretraining have been **removed (corrected tokenizer)**; the
88
  vocabulary stays 32,000. BPB is per-byte, so this does not confound cross-model comparison.
89
 
90
+ **Where the tokenizer does well.** On raw byte-compression efficiency (tokens-per-byte **0.2591**) it ranks **4th of
91
+ 17**, essentially tied with the most byte-efficient general tokenizers (Llama-3 0.2573) β€” fewer tokens per document, a
92
+ multiplicative saving on both training and inference. It deliberately does *not* optimize for whole-word medical
93
+ vocabulary, so its medical single-token rate (T1b STRR) is the lowest in the pool β€” but that tradeoff is **vindicated**:
94
+ despite the lowest single-token rate, the model still ranks **1st of 17 on held-out medical BPB and leads the pool on
95
+ medical entity extraction**. A high single-token rate is neither necessary for nor sufficient for strong downstream
96
+ medical modeling.
97
+
98
  ## Pre-training
99
  Trained **from scratch, not distilled or continued**. Pure next-token cross-entropy (**no auxiliary losses**),
100
  full-parameter on the LlamaFactory trainer, AdamW, peak LR 3.0e-4 cosine (warmup 1%), grad-clip 1.0,
 
133
  license verification; they will be added once resolved.)*
134
 
135
  **Disclosed issue:** ~35% of tokens are duplicates (a FineWeb-Edu sharding build bug + PMC repetition); a deduped
136
+ corpus (`v4_dedup`, 79.4 M docs, 0% dup) is ready but was **not** trained β€” this release is the original single-epoch corpus.
137
 
138
  **Governance:** includes PhysioNet-credentialed, de-identified, redistribution-restricted clinical sources (MIMIC-IV
139
  discharge / radiology, MIMIC-CXR) β€” any downstream use must confirm PhysioNet DUA compliance.
 
157
  wins, KOS-V4-Base reaches **frontier parity (best-tied) on 6 metrics** (near-zero dead-neuron rate; saturated needle
158
  retrieval at several depths) and **top-3 on 9 more**.
159
 
160
+ **Radiology and clinical text is its home turf β€” and there it beats every trillion-token model in the pool.** On
161
+ per-domain held-out BPB (lower = better) it ranks **1st of 17** on radiology reports (**0.4761** vs Qwen3-4B/36 T
162
+ 0.7995, Llama-3-8B 0.7444, Gemma-2-9B 0.7760), chest-X-ray reports (**0.5887**), and clinical narratives (**0.3221**);
163
+ and **1st of 17** on radiology entity/relation extraction (RadGraph DR.1/2/3 macro-F1 **0.7665 / 0.7990 / 0.8150**). It
164
+ *trails* on the general axes the trillion-token models saturate β€” biomedical literature (rank 17/17) and textbooks
165
+ (rank 12/17) β€” but the clinical/radiology wins are decisive enough to carry the **5-domain BPB mean to rank 1/17**. A
166
+ 3 B model on 180 B tokens out-modeling 4–9 B models trained on 8–36 T tokens, specifically on the clinical text it was
167
+ built for, is the payoff of a from-scratch clinical corpus.
168
+
169
  **Where it is weak:**
170
  - **Closed-book medical MCQ** (T6 β€” near-chance): as a 180 B-token base it does not reliably recall parametric medical
171
  facts; it is worst on the hardest professional/college splits (mmlu_professional_medicine **0.2096**,