codybum commited on
Commit
e884e9f
·
verified ·
1 Parent(s): 1f84a3a

Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

Browse files
Files changed (1) hide show
  1. README.md +85 -239
README.md CHANGED
@@ -18,70 +18,48 @@ tags:
18
  <img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
19
  </div>
20
 
21
- > **Code name: Scratch.** The KOS-V4 series is nicknamed **Scratch LLM**. It is not a frontier model, and building a
22
- > from-scratch clinical LLM on this budget was arguably not the *sane* thing to attempt. That is exactly the point:
23
- > it was trained **completely from scratch** by a small team, on a fraction of the data and compute, and it serves a
24
- > **real, narrow purpose** (grounded clinical RAG), not a leaderboard.
25
 
26
  > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
27
  > clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
28
 
29
  ---
30
 
31
- ## KOS-V4: a from-scratch 3B clinical suite that reaches original-ChatGPT-level instruction-following
32
-
33
- KOS-V4 is a **first native LLM release** from a University of Kentucky College of Medicine
34
- [Office for Research](https://medicine.uky.edu/sites/research) and the
35
- [Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/) research team. It is a family of
36
- **3B models built completely from scratch from publicly curated data without the use of a commercially provided
37
- base model**. The suite is built from a custom 32k tokenizer, a from-scratch pre-trained base, and task-specialized
38
- post-training, assembled into a **grounded instruction and RAG system** (a retriever, grounded readers that abstain
39
- when the answer is absent, and an instruction and tool model). The goal is not to memorize medicine; it is to
40
- reliably follow the standard of care retrieved for it.
41
-
42
- **David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs** over about six days. The
43
- models it is measured against were built at an enormously larger scale, on two fronts. Compute: OpenAI trained the
44
- original ChatGPT (**GPT-3.5**) on a Microsoft supercomputer of roughly **10,000 GPUs** (285,000 CPU cores), one of
45
- the largest machines in the world at the time, on the order of **400 times more GPUs** than we used. Data: today's
46
- common open models are trained on **trillions of tokens**, for example Qwen2.5 on about **18 trillion**, Qwen3 on
47
- about **36 trillion**, and Llama 3 on about **15 trillion**. KOS-V4's **180 billion tokens** is **100 to 200 times
48
- less data**, two orders of magnitude smaller. It reaches the original-ChatGPT bar from that small fraction of the
49
- data and compute.
50
-
51
- **And it reaches that bar.** On **IFEval**, the standard instruction-following benchmark, KOS-V4-Instruct scores
52
- **AVG4 62.7**, above the proprietary **ChatGPT (gpt-3.5-turbo-1106, 58.0)** and within three points of the
53
- commercially trained 3B, **Qwen2.5-3B-Instruct (65.4)**.
54
-
55
- | IFEval (AVG4 of the four strict and loose metrics) | score | who built it, and how |
56
- | :-- | :-- | :-- |
57
- | **KOS-V4-Instruct (ours)** | **62.7** | a small team, from scratch, 180B tokens, 24 GPUs |
58
- | GPT-3.5-turbo, the original ChatGPT (gpt-3.5-turbo-1106, verified) | 58.0 | OpenAI, about 10,000-GPU supercomputer |
59
- | Qwen2.5-3B-Instruct | 65.4 | Alibaba, about 18 trillion tokens |
60
-
61
- **Where it genuinely leads.** On the RAG tasks it was built for, KOS-V4 is competitive or best in class:
62
- - **Guideline grounding** (RGB negative-rejection: refusing when the guidance does not cover the question) is
63
- **45.0**, the best in our comparison set, above even the newer Qwen3-4B (40.7).
64
- - **Retrieval** (BEIR SciFact, zero-shot) is **NDCG@10 = 0.7327**, inside the strong-dense band (0.70 to 0.76) and
65
- above BM25 (0.665).
66
-
67
- **Read honestly.** This is a first release from a small team on a small budget, not a frontier model. Our numbers
68
- are measured on public benchmarks; the GPT-3.5-turbo figure is the verified gpt-3.5-turbo-1106 snapshot from the
69
- InternLM2 technical report (50.5, 61.2, 54.9, 65.5 across the four metrics, AVG4 58.0). Each model card states its own
70
- results, including where it trails.
71
 
72
- ---
 
 
 
 
 
 
 
 
 
 
 
73
 
74
- # KOS-V4-Instruct (3B, from scratch)
 
 
75
 
76
- **KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a small team. It is
77
- a decoder-only transformer (Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and
78
- tool / function calling** over clinical and general English text. Its instruction ability comes from **GRPO
79
- reinforcement learning** on a from-scratch clinical base, reaching **IFEval AVG4 = 62.7**. That is on par with the
80
- verified score for the original ChatGPT (**GPT-3.5-turbo-1106, 58.0**) and within about three points of
81
- **Qwen2.5-3B-Instruct (65.4)**, from a fraction of the data and compute. It also adds reliable **tool and function
82
- calling** (official BFCL 73/73/61), a capability the original ChatGPT lacked at launch, though modern small models
83
- score higher here. It is the flagship of the
84
- **KOS-V4** family (a retriever and two grounded readers ship alongside it).
 
 
 
 
 
 
85
 
86
  ## Core specifications
87
  | Attribute | Detail |
@@ -92,13 +70,12 @@ score higher here. It is the flagship of the
92
  | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
93
  | **Feed-forward** | SwiGLU, intermediate size 8192 |
94
  | **Embeddings** | Untied input/output |
95
- | **Vocabulary** | 32,000, custom medical **byte-level BPE** (8% fewer tokens/medical-char than GPT-4/Llama/Qwen) |
96
  | **Context length** | 24,576 (`max_position_embeddings` 65,536) |
97
  | **Position encoding** | RoPE, θ = 25,000 |
98
  | **Precision** | bfloat16 |
99
  | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
100
  | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
101
- | **License** | *to be set by Kentucky Open Science* |
102
 
103
  ## Quickstart (Hugging Face Transformers)
104
  ```python
@@ -129,7 +106,7 @@ KOS-V4-Instruct uses **ChatML**:
129
  <|im_start|>assistant
130
  {response}<|im_end|>
131
  ```
132
- `<|im_end|>` is the true end-of-turn / eos token. Tool calling uses dedicated `<|tool_call|>` / `<|tool_response|>`
133
  tokens; pass tools via the tokenizer's chat template `tools=` argument.
134
 
135
  ## Pre-training (the KOS-V4 base)
@@ -139,209 +116,78 @@ Trained from scratch, not distilled or continued from another model.
139
  | Parameter | Value |
140
  | :--- | :--- |
141
  | Objective | Pure next-token cross-entropy (**no auxiliary losses**) |
142
- | Optimizer | AdamW (β1 = 0.9, β2 = 0.999, eps 1e-8), weight decay 0.0 |
143
- | Peak learning rate | 3.0e-4 |
144
- | LR schedule | Cosine, warmup ratio 0.01, 1 epoch |
145
  | Gradient clipping | 1.0 max-norm |
146
  | Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
147
- | Micro-batch / grad-accum | 1 / 1, giving a **global batch of 589,824 tokens/step** |
148
  | Precision / kernels | bfloat16, FlashAttention-2 |
149
  | Total steps / tokens | 305,613 / 180.3 B token-positions |
150
 
151
  ### Training data mixture
152
  English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
153
- (approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B
154
- BlueScrubs 4.6 B, Hindawi 2.9 B, MedAL 2.5 B, S2ORC 1.2 B, clinical trials 0.7 B, Wikipedia 0.6 B
155
- plus 30 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora (e.g. MIMIC-IV
156
- MIMIC-CXR). Register mix (chunks): formal-academic 100 M, formal-clinical 14.5 M, patient-facing 7.8 M
157
- technical-reference 5.9 M, conversational 0.5 M.
158
 
159
  ### Known pre-training issues (disclosed)
160
- - **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/
161
- filter build bug re-emitted byte-identical docs, 2.2× copies; plus 24 % in PMC). A fully deduplicated corpus was
162
- built but **not** trained, so this base carries the duplication; dedup is a ready improvement for the next base.
163
- - **38 "sink" tokens to surgical tokenizer.** 38 orphan BPE merges (markdown/table residue) behaved as a localized
164
- attention sink; the 38 merge rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a
165
- fail-closed gate enforced at SFT/RL/serve.
166
- - **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000**, pin it on export.
167
 
168
  ## Post-training (this model)
169
  ### 1. Instruction SFT
170
- **Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch
171
- NEFTune α=5**, whole-document packing on 8×H200. Corpus (0.9 M+ rows): a stratified medical+tool+general majority
172
- (615 k), a **23-family instruction-following taxonomy** (200 k, generated with a **verifier-in-the-loop** loop
173
- constraints sampled from the empirical IFEval distribution, teacher rejection-sampling kept only responses passing
174
- the official checker), generic function-calling (xLAM, 20 k), and grounded cite/abstain RAG (79 k). A forgetting
175
- gate (broad perplexity ≤ 1.5× base) guards retention.
176
 
177
  ### 2. Reinforcement learning (GRPO)
178
- **GRPO** via **verl** on 8×H200, with **deterministic verifiable rewards, no reward model, no LLM judge.** The
179
- primary reward runs the *official* IFEval constraint checker (dense shaping `0.7*frac + 0.3*all_pass`); other data
180
- sources reward grounded RAG (abstain-when-unanswerable) and tool-call recall. Settings: lr 1e-6, KL 0.001
181
- (low-variance), group size n=16, 128 prompts/step, checkpoint every 40 steps (**ship the argmax**). Official IFEval
182
- AVG4 rose across GRPO steps from **47.7 (SFT base)** to **62.7 at step 1120 (shipped)**, a gain of 15.0 points
183
- (intermediate checkpoints scored 58.6, 59.4, 59.7, 61.5, then the 62.7 peak, then 61.8).
184
 
185
  ## Evaluation & benchmarks
186
- Public benchmarks only. **IFEval** is the EleutherAI lm-evaluation-harness with `--apply_chat_template`, AVG4 of the
187
- four strict and loose metrics. **BFCL** is the official Berkeley Function-Calling Leaderboard package (`bfcl_eval`),
188
- run with its **official per-model native handlers** in function-calling mode: each model is prompted in its own tool
189
- format and scored by the official AST checker. The GPT-3.5-turbo IFEval column is the **verified** gpt-3.5-turbo-1106
190
- snapshot from the InternLM2 technical report, included as the original-ChatGPT reference point rather than run by us.
191
-
192
- | IFEval | KOS-V4-Instruct (ours) | GPT-3.5-turbo-1106 (original ChatGPT, verified) | Qwen2.5-3B-Instruct |
193
- | :--- | :--- | :--- | :--- |
194
- | **AVG4** | **62.7** | 58.0 | 65.4 |
195
- | prompt-strict / inst-strict / prompt-loose / inst-loose | 55.8 / 67.0 / 58.4 / 69.7 | 50.5 / 61.2 / 54.9 / 65.5 | 59.5 / 68.5 / 62.1 / 71.3 |
196
- | *(reference)* KOS-V4 SFT base, pre-RL AVG4 | 47.7 | | |
197
-
198
- Tool-calling, **official BFCL** (function-calling mode; simple / multiple / parallel accuracy, higher is better):
199
-
200
- | BFCL (official FC) | KOS-V4-Instruct (ours) | Qwen2.5-3B-Instruct | Llama-3.2-3B-Instruct |
201
- | :--- | :--- | :--- | :--- |
202
  | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
203
 
204
- **Reading.** From a small team's from-scratch 3B, IFEval **62.7** clears the original ChatGPT
205
- (GPT-3.5-turbo-1106, a verified 58.0) and lands within about three points of Qwen2.5-3B-Instruct (65.4), while being
206
- **15.0 points above our own SFT base** (47.7). On tool-calling, KOS-V4 works (official BFCL 72.75 / 73.00 / 60.50)
207
- but clearly **trails the modern small instruct models**: both Qwen2.5-3B and Llama-3.2-3B score around 90 or above on
208
- simple and multiple, and far higher on parallel. Function-calling is a real capability here, one the original ChatGPT
209
- did not have at launch, but it is **not a strength relative to today's small models**, which were trained on far more
210
- tool-use data. The same holds on IFEval, where newer, more heavily distilled small models score higher; KOS-V4's
211
- results are notable for its data and compute budget, not a claim to lead the current frontier.
212
-
213
- ## IFEval in context (models our size or larger)
214
- Every model is scored on the **same standard metric, IFEval AVG4** (mean of the four strict + loose sub-metrics from
215
- the official IFEval implementation, `google-research/instruction_following_eval`). For open models, AVG4 is computed
216
- identically from the archived **Open LLM Leaderboard** results (`open-llm-leaderboard/results`), and **company,
217
- parameters, and release date are taken from `open-llm-leaderboard/contents`**. The **weights** column marks each model
218
- **Open** (open weights, downloadable, whoever built it) or **Proprietary** (closed, API only); proprietary models are
219
- not on the leaderboard. A **\*** on a score means the AVG4 is an estimate, and a **+** on a size means the parameter
220
- count is unofficial; both are explained under the table. Models smaller than KOS-V4 are excluded. Calibration: our
221
- harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
222
-
223
- | model | weights | company | released | params | IFEval AVG4 |
224
- | :-- | :-- | :-- | :-- | :-- | :-- |
225
- | GPT-4o-mini | Proprietary | OpenAI | Jul 2024 | 8B + | 82 \* |
226
- | Llama-3.2-3B-Instruct | Open | Meta | Sep 2024 | 3.2B | 76.6 |
227
- | Qwen2.5-3B-Instruct | Open | Alibaba | Sep 2024 | 3.0B | 66.0 |
228
- | Mistral-Large | Proprietary | Mistral AI | Feb 2024 | 46.7B + | 66 \* |
229
- | Phi-3-medium-4k-instruct | Open | Microsoft | May 2024 | 14.0B | 65.7 |
230
- | **KOS-V4-Instruct (ours)** | **Open** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **62.7** |
231
- | Yi-1.5-9B-Chat | Open | 01.AI | May 2024 | 8.8B | 62.7 |
232
- | GPT-3.5-turbo-0613 | Proprietary | OpenAI | Jun 2023 | 20B + | 61 \* |
233
- | Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 60.5 |
234
- | Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
235
- | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
236
- | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 58 \* |
237
- | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 58 \* |
238
- | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
239
- | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
240
- | Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 56.6 |
241
- | gemma-1.1-7b-it | Open | Google | Mar 2024 | 8.5B | 52.9 |
242
- | GPT-3.5-turbo-0125 | Proprietary | OpenAI | Jan 2024 | 20B + | 52.1 |
243
- | Zephyr-7B-beta | Open | Hugging Face | Oct 2023 | 7.2B | 51.2 |
244
- | Llama-3.1-8B-Instruct | Open | Meta | Jul 2024 | 8.0B | 51.0 |
245
- | Llama-2-13b-chat | Open | Meta | Jul 2023 | 13.0B | 46.5 |
246
- | Qwen1.5-7B-Chat | Open | Alibaba | Jan 2024 | 7.7B | 46.1 |
247
- | Qwen1.5-4B-Chat | Open | Alibaba | Jan 2024 | 4.0B | 33.2 |
248
-
249
- **\* AVG4 is an estimate.** No official four-metric IFEval AVG4 is published for this model; the value is estimated
250
- from adjacent evidence (a prompt-strict-only score, or the model's capability class). Point value with range, method,
251
- and confidence are in the baseline block below and in `COMMERCIAL_IFEVAL_BASELINES.md`. Rows with no marker are
252
- measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
253
-
254
- **\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
255
- GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 100B, Gemini-1.0-Pro about 30B
256
- (public estimates span 13B to over 500B; 30B is a low-end estimate).
257
-
258
- **Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**
259
- (1106 = 58.0, 0125 = 52.1, and the estimated 0613 = 61), ties the 8.8B Yi-1.5, and clears the estimated Command-R,
260
- Gemini-1.0-Pro and Mistral-Medium. It outscores every 7B-14B open instruct model below it, including both Mistral-7B-Instruct
261
- releases, **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and Qwen1.5-7B. **The newest major
262
- model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Ahead of it sit three open models
263
- (Llama-3.2-3B, Qwen2.5-3B, Phi-3-medium-14B) and two proprietary estimates (GPT-4o-mini, Mistral-Large).
264
- (Llama-3.1-8B's leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived
265
- value.)
266
-
267
- **Commercial baselines (IFEval AVG4, same metric as the table above).** KOS-V4's 62.7 sits at the top of the original
268
- commercial-API tier and below the 2024 frontier. Rows are grouped by evidence quality. **Verified** = AVG4 computed
269
- here from four published sub-metrics. **Reported** = a four-metric AVG4 published by a primary source (sub-metrics not
270
- broken out). **Estimate (EST)** = no IFEval AVG4 is published; estimated from adjacent evidence and clearly labeled,
271
- not to be cited as measured.
272
-
273
- Verified (four sub-metrics, AVG4 computed here):
274
-
275
- | commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
276
- | :-- | :-- | :-- | :-- | :-- |
277
- | GPT-4 | OpenAI | gpt-4-0613 | 82.1 | InternLM2 report (arXiv:2403.17297) |
278
- | Command-R+ | Cohere | c4ai-command-r-plus | 77.9 | Open LLM Leaderboard raw results |
279
- | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 58.0 | InternLM2 report (arXiv:2403.17297) |
280
-
281
- Reported (four-metric AVG4 published, sub-metrics not broken out):
282
-
283
- | commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
284
- | :-- | :-- | :-- | :-- | :-- |
285
- | Claude-3.5-Sonnet | Anthropic | claude-3-5-sonnet-20240620 | 86.2 | Llama 3.1 report (arXiv:2407.21783) |
286
- | GPT-4o | OpenAI | gpt-4o-2024-05-13 | 84.3 | Llama 3.1 report |
287
- | Gemini-1.5-Pro | Google | May 2024 | 82.3 | Llama 3.1 report |
288
- | GPT-4-turbo | OpenAI | gpt-4-1106-preview | 81.4 | Llama 3.1 report |
289
- | Gemini-1.5-Flash | Google | May 2024 | 77.2 | Llama 3.1 report |
290
- | Claude-3-Sonnet | Anthropic | Mar 2024 | 72.3 | Llama 3.1 report |
291
- | Claude-3-Haiku | Anthropic | Mar 2024 | 68.1 | Llama 3.1 report |
292
- | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | 52.1 | Llama 3.1 report |
293
-
294
- Estimate (no published IFEval AVG4; EST values, not measured):
295
-
296
- | commercial model | company | endpoint / snapshot | IFEval AVG4 | basis |
297
- | :-- | :-- | :-- | :-- | :-- |
298
- | GPT-4o-mini | OpenAI | gpt-4o-mini-2024-07-18 | 82 EST (79-84) | class anchor to GPT-4o |
299
- | Mistral-Large | Mistral AI | mistral-large-2402 | 66 EST (60-72) | class anchor |
300
- | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0613 | 61 EST (60-63) | uplift from prompt-strict 54.4 (Conifer) |
301
- | Gemini-1.0-Pro | Google | gemini-1.0-pro-001 | 58 EST (55-61) | class anchor (report uses InFoBench, not IFEval) |
302
- | Command-R | Cohere | command-r (Mar 2024) | 60 EST (54-68) | anchor to Command-R+ |
303
- | Mistral-Medium | Mistral AI | mistral-medium-2312 | 58 EST (54-63) | class anchor |
304
-
305
- The GPT-3.5-turbo family spans AVG4 52.1 (0125) to 58.0 (1106) to 61 est. (0613) across snapshots. KOS-V4's 62.7 sits
306
- at or above the top of that band on every snapshot with data.
307
-
308
- **Takeaway.** KOS-V4 (62.7) clears the entire original ChatGPT / GPT-3.5-turbo generation and the estimated
309
- Gemini-1.0-Pro and Command-R tier. The 2024 frontier (GPT-4, GPT-4o, Claude-3.5-Sonnet, Gemini-1.5, Command-R+) stays
310
- well ahead. A from-scratch 3B on 180B tokens and 24 GPUs, at the instruction-following level of the first commercial
311
- ChatGPT generation.
312
-
313
- **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
314
- Open-model AVG4: computed from `open-llm-leaderboard/results`, with company, parameters and release date from
315
- `open-llm-leaderboard/contents`. Commercial verified sub-metrics: InternLM2 technical report (arXiv:2403.17297) for
316
- GPT-4-0613 and GPT-3.5-turbo-1106; Open LLM Leaderboard raw results for Command-R+. Commercial reported AVG4: Llama 3.1
317
- technical report (arXiv:2407.21783), Table 2. GPT-3.5-turbo-0613 prompt-strict: Conifer (arXiv:2404.02823). Estimate
318
- rows are derived, not measured, and are labeled EST. Full dataset with methods and confidence:
319
- `COMMERCIAL_IFEVAL_BASELINES.md`.
320
-
321
- ## Compute footprint & requirements
322
- ### Training
323
- - **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, **about 3,300 H200-GPU-hours**.
324
- - **SFT:** 1× 8×H200 node, 1 epoch. **RL (GRPO):** 1× 8×H200 node.
325
-
326
- ### Deployment (inference)
327
  | Precision | Approx. VRAM | Notes |
328
  | :--- | :--- | :--- |
329
- | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; 1× 16 GB GPU is comfortable |
330
- | INT8 (approx.) | 4 GB | not shipped; typical for a 3 B model |
331
- | INT4 (approx.) | 2.5 GB | not shipped; runs on consumer/edge hardware |
332
 
333
  ## Intended use & limitations
334
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
335
- workflows. For grounded record/guideline reading with abstention use **KOS-V4-Grounded-Record** / **KOS-V4-Grounded-Guideline**; for retrieval use **KOS-V4-Retriever**.
336
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
337
  parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
338
- - This instruction-tuned checkpoint trades away grounded abstention (its RGB negative-rejection is near zero), so for
339
- non-fabrication on unanswerable questions use **KOS-V4-Grounded-Record** or **KOS-V4-Grounded-Guideline**.
340
- - IFEval 62.7 is on the original-ChatGPT (GPT-3.5-turbo) level and near Qwen2.5-3B (65.4); newer small models trained
341
- on far more data score higher. English only. Strong public-benchmark numbers are not validation on real clinical data.
342
-
343
- ## The KOS-V4 family
344
- - **KOS-V4-Instruct**, this model (general instruction + tool calling).
345
- - **[KOS-V4-Retriever](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Retriever)**, clinical dense retriever (BEIR SciFact 0.7327 zero-shot).
346
- - **[KOS-V4-Grounded-Record](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Grounded-Record)**, patient-record grounding specialist (RGB neg-rejection 35.0).
347
- - **[KOS-V4-Grounded-Guideline](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Grounded-Guideline)**, clinical-guideline grounding specialist (RGB neg-rejection 45.0).
 
18
  <img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
19
  </div>
20
 
21
+ > **Code name: Scratch.** KOS-V4 is nicknamed **Scratch LLM** because it was trained **completely from scratch** by a
22
+ > small team on a fraction of the data and compute of commercial models. It is not a frontier model.
 
 
23
 
24
  > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
25
  > clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
26
 
27
  ---
28
 
29
+ # KOS-V4-Instruct — a from-scratch 3B instruction & tool model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
 
31
+ **KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a University of
32
+ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research),
33
+ [Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer
34
+ (Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and tool / function calling**
35
+ over clinical and general English text. It was pre-trained on **180 billion tokens using 24 GPUs**, then
36
+ post-trained with full-parameter SFT followed by **GRPO reinforcement learning** against the official IFEval checker.
37
+
38
+ ## Headline result (official benchmarks only)
39
+
40
+ Every number in this card comes from an **official benchmark suite** — **IFEval** via the EleutherAI
41
+ lm-evaluation-harness (`--apply_chat_template`) and **BFCL** via the official `bfcl_eval` package. No in-house
42
+ scorers are used.
43
 
44
+ **IFEval is reported here as the strict-average: `(prompt-level strict accuracy + instruction-level strict accuracy) / 2`.**
45
+ This is the same "IFEval" metric published on the Hugging Face Open LLM Leaderboard. We do **not** headline the looser
46
+ four-metric average (which reads a few points higher).
47
 
48
+ | IFEval **strict-avg** | model | notes |
49
+ | --: | :-- | :-- |
50
+ | **61.6** | **KOS-V4-Instruct (ours)** | from scratch, 180 B tokens, 24 GPUs |
51
+ | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI |
52
+ | 64.0 | Qwen2.5-3B-Instruct | Alibaba |
53
+ | 75.2 | Llama-3.2-3B-Instruct | Meta |
54
+ | 85.0 | Qwen3-4B-Instruct | Alibaba |
55
+
56
+ **Honest standing.** KOS-V4-Instruct (61.6) is **above the original GPT-3.5-turbo generation (55.9)** and **below
57
+ current small instruct models** — Qwen2.5-3B (64.0), Llama-3.2-3B (75.2) and Qwen3-4B (85.0), all of which were
58
+ trained on far more data and instruction-tuning. This is a strong result **for its data and compute budget**, not a
59
+ claim to lead the current field.
60
+
61
+ Peer numbers are from the **same official lm-eval harness**; GPT-3.5-turbo-1106 is the verified snapshot from the
62
+ InternLM2 technical report (strict sub-metrics 50.5 / 61.2).
63
 
64
  ## Core specifications
65
  | Attribute | Detail |
 
70
  | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
71
  | **Feed-forward** | SwiGLU, intermediate size 8192 |
72
  | **Embeddings** | Untied input/output |
73
+ | **Vocabulary** | 32,000, custom medical **byte-level BPE** |
74
  | **Context length** | 24,576 (`max_position_embeddings` 65,536) |
75
  | **Position encoding** | RoPE, θ = 25,000 |
76
  | **Precision** | bfloat16 |
77
  | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
78
  | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
 
79
 
80
  ## Quickstart (Hugging Face Transformers)
81
  ```python
 
106
  <|im_start|>assistant
107
  {response}<|im_end|>
108
  ```
109
+ `<|im_end|>` is the true end-of-turn / eos token. Tool calling uses dedicated `<tool_call>` / `<tool_response>`
110
  tokens; pass tools via the tokenizer's chat template `tools=` argument.
111
 
112
  ## Pre-training (the KOS-V4 base)
 
116
  | Parameter | Value |
117
  | :--- | :--- |
118
  | Objective | Pure next-token cross-entropy (**no auxiliary losses**) |
119
+ | Optimizer | AdamW (β1 = 0.9, β2 = 0.999), weight decay 0.0 |
120
+ | Peak learning rate | 3.0e-4, cosine, warmup ratio 0.01, 1 epoch |
 
121
  | Gradient clipping | 1.0 max-norm |
122
  | Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
123
+ | Global batch | 589,824 tokens/step |
124
  | Precision / kernels | bfloat16, FlashAttention-2 |
125
  | Total steps / tokens | 305,613 / 180.3 B token-positions |
126
 
127
  ### Training data mixture
128
  English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
129
+ (approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B,
130
+ BlueScrubs 4.6 B, plus ~40 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora.
 
 
 
131
 
132
  ### Known pre-training issues (disclosed)
133
+ - **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/filter
134
+ build bug plus PMC repetition). A fully deduplicated corpus was built but **not** trained; this base carries the
135
+ duplication, and dedup is a ready improvement for the next base.
136
+ - **38 "sink" tokens → surgical tokenizer.** 38 orphan BPE merges behaved as a localized attention sink; the merge
137
+ rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a fail-closed gate enforced.
138
+ - **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000** — pin it on export.
 
139
 
140
  ## Post-training (this model)
141
  ### 1. Instruction SFT
142
+ **Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch, NEFTune
143
+ α=5**, whole-document packing. Corpus (~0.9 M rows): a stratified medical + tool + general majority, a **23-family
144
+ instruction-following taxonomy** (generated with a verifier-in-the-loop loop, keeping only responses that pass the
145
+ official IFEval checker), generic function-calling (xLAM), and grounded cite/abstain RAG. A forgetting gate (broad
146
+ perplexity ≤ 1.5× base) guards retention.
 
147
 
148
  ### 2. Reinforcement learning (GRPO)
149
+ **GRPO** via **verl**, with **deterministic verifiable rewards — no reward model, no LLM judge.** The primary reward
150
+ runs the *official* IFEval constraint checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to
151
+ 61.6 at step 1120 (shipped)**. The shipped weights are checkpoint **step 1120**, selected as the argmax of the
152
+ official strict-avg over all RL checkpoints.
 
 
153
 
154
  ## Evaluation & benchmarks
155
+
156
+ ### IFEval (official EleutherAI lm-eval-harness, `--apply_chat_template`)
157
+ Strict sub-metrics and the strict-average (the headline metric):
158
+
159
+ | IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
160
+ | :-- | --: | --: | --: | --: | --: |
161
+ | prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
162
+ | instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
163
+ | **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
164
+
165
+ ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel, higher is better)
166
+
167
+ | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
168
+ | :-- | :-- | :-- | :-- |
 
 
169
  | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
170
 
171
+ **Reading.** On instruction following, KOS-V4-Instruct (strict-avg 61.6) clears the original GPT-3.5-turbo generation
172
+ but trails today's small instruct models. On tool calling it works, but again trails the modern small models, which
173
+ were trained on far more tool-use data. The result is notable for a from-scratch 3B on 180 B tokens and 24 GPUs — not
174
+ a claim to lead the current frontier.
175
+
176
+ ## Compute footprint
177
+ - **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, ~3,300 H200-GPU-hours.
178
+ - **SFT + RL (GRPO):** single 8-GPU node each.
179
+
180
+ ## Deployment (inference)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
181
  | Precision | Approx. VRAM | Notes |
182
  | :--- | :--- | :--- |
183
+ | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
184
+ | INT8 / INT4 (approx.) | 4 / 2.5 GB | not shipped; typical for a 3 B model |
 
185
 
186
  ## Intended use & limitations
187
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
188
+ workflows.
189
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
190
  parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
191
+ - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this
192
+ model's results are notable for its data/compute budget, not for leading the field.
193
+ - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.