codybum commited on
Commit
42d1f43
·
verified ·
1 Parent(s): 46d3c7b

Prepend shared kos-v4 family headline (data-fraction positioning)

Browse files
Files changed (1) hide show
  1. README.md +97 -90
README.md CHANGED
@@ -14,41 +14,39 @@ tags:
14
  - from-scratch
15
  ---
16
 
17
- ## kos-v4 family — a from-scratch clinical suite trained on ~1 % of the data
18
-
19
- **Why this suite exists.** kos-v4 is a small, fully open family of **3B models built completely from scratch** by a
20
- small team — custom 32k medical tokenizer → from-scratch pre-trained base → task-specialized post-training —
21
- assembled as the working parts of a **grounded clinical RAG system**: a **retriever** to find the right passage,
22
- **grounded readers** that answer *only* from it and **abstain** when the answer isn't there, and an
23
- **instruction / tool executor** that follows directions and calls tools. The aim is not to memorize medicine but to
24
- safely **follow the standard of care retrieved for it**.
25
-
26
- **The headline — a fraction of the data.** The base was pre-trained on **~180 B tokens**. The models it is compared
27
- against saw **50–200× more** — Llama-3.2-3B ≈ 9 T (~50×), Qwen2.5-3B ≈ 18 T (~100×), Qwen3-4B ≈ 36 T (~200×).
28
- kos-v4 is also *smaller* (3.015 B params) and was trained on 24 GPUs in under 6 days (≈ 3,300 GPU-hours).
29
-
30
- **Where that lands us.** Even at **~1 % of the training data**, kos-v4 performs at the level of the
31
- **2024-generation open 3B instruct models** (Qwen2.5-3B, Llama-3.2-3B) — and on the two tasks it was purpose-built
32
- for, it **matches or beats even the newest peer, Qwen3-4B**:
33
-
34
- | axis | kos-v4 | listed peers | verdict |
35
- | :--- | :--- | :--- | :--- |
36
- | Instruction following · IFEval AVG4 | **62.7** | Qwen2.5-3B 65.4 · Qwen3-4B 86.6 | within ~3 pts of 2024-gen |
37
- | Retrieval · BEIR SciFact NDCG@10 (zero-shot) | **0.7327** | BM25 0.665 · strong-dense 0.70–0.76 | in the strong-dense band |
38
- | Guideline grounding · RGB neg-rejection | **45.0** | Qwen3-4B 40.7 · Qwen2.5-3B 32 | **above every 3B–4B peer** |
39
- | Record grounding · RGB neg-rejection | **35.0** | Llama-3.2-3B 35.3 · Qwen2.5-3B 32 | at/above the 3B peers |
40
-
41
- **Bottom line:** with **~1 % of the data** and a smaller model, kos-v4 is at **2024-generation parity in general**,
42
- and **competitive with current-generation models on the retrieval and guideline-grounding tasks it targets** — a
43
- deliberately efficient, honestly-measured result from a small team.
44
-
45
- **Reading the comparison tables.** Every number for *our* models is measured on a public benchmark
46
- (EleutherAI lm-eval IFEval with `--apply_chat_template`; official BEIR NDCG@10; RGB) and reported honestly,
47
- including where we trail. Where a **peer** cell is blank (—), that figure is simply **not publicly reported for
48
- that exact benchmark/setting** (or we have not reproduced it under identical conditions) — it does **not** mean the
49
- peer scores zero.
50
-
51
- *Shared kos-v4 family headline — the model-specific card follows below.*
52
 
53
  ---
54
 
@@ -57,9 +55,11 @@ peer scores zero.
57
  **kos-v4-instruct** is an open-weights **3B language model trained completely from scratch** by a small team. It is
58
  a decoder-only transformer (Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and
59
  tool / function calling** over clinical and general English text. Its instruction ability comes from **GRPO
60
- reinforcement learning** on a from-scratch clinical base, reaching **IFEval AVG4 = 62.7** — within ~3 points of
61
- Qwen2.5-3B-Instruct (65.4) — while holding strong tool-calling (BFCL 81/73/62). It is the flagship of the
62
- **kos-v4** family (a retriever and a grounded reader ship alongside it).
 
 
63
 
64
  ## Core specifications
65
  | Attribute | Detail |
@@ -70,12 +70,12 @@ Qwen2.5-3B-Instruct (65.4) — while holding strong tool-calling (BFCL 81/73/62)
70
  | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
71
  | **Feed-forward** | SwiGLU, intermediate size 8192 |
72
  | **Embeddings** | Untied input/output |
73
- | **Vocabulary** | 32,000 — custom medical **byte-level BPE** (~8% fewer tokens/medical-char than GPT-4/Llama/Qwen) |
74
  | **Context length** | 24,576 (`max_position_embeddings` 65,536) |
75
  | **Position encoding** | RoPE, θ = 25,000 |
76
  | **Precision** | bfloat16 |
77
  | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
78
- | **Pretraining tokens** | ~180.3 B (English medical/biomedical + web) |
79
  | **License** | *to be set by Kentucky Open Science* |
80
 
81
  ## Quickstart (Hugging Face Transformers)
@@ -88,11 +88,11 @@ tok = AutoTokenizer.from_pretrained(model_id)
88
  model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
89
 
90
  messages = [
91
- {"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}
92
  ]
93
  inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
94
  with torch.no_grad():
95
- out = model.generate(inputs, max_new_tokens=256, do_sample=False)
96
  print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
97
  ```
98
  Serve with vLLM / TGI as a standard `Qwen3`-architecture causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export.
@@ -111,7 +111,7 @@ kos-v4-instruct uses **ChatML**:
111
  tokens; pass tools via the tokenizer's chat template `tools=` argument.
112
 
113
  ## Pre-training (the kos-v4 base)
114
- Trained from scratch — not distilled or continued from another model.
115
 
116
  ### Hyperparameters & schedule
117
  | Parameter | Value |
@@ -122,81 +122,88 @@ Trained from scratch — not distilled or continued from another model.
122
  | LR schedule | Cosine, warmup ratio 0.01, 1 epoch |
123
  | Gradient clipping | 1.0 max-norm |
124
  | Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
125
- | Micro-batch / grad-accum | 1 / 1 → **global batch 589,824 tokens/step** |
126
  | Precision / kernels | bfloat16, FlashAttention-2 |
127
- | Total steps / tokens | 305,613 / ~180.3 B token-positions |
128
 
129
  ### Training data mixture
130
- English-only, **medical/biomedical-first**, **49 sources / ~130 M document chunks**. Largest contributors
131
- (approx. tokens): PubMed Central full text ~77 B · FineWeb-Edu (readability-filtered web) ~51 B · mMedC-en ~6.3 B ·
132
- BlueScrubs ~4.6 B · Hindawi ~2.9 B · MedAL ~2.5 B · S2ORC ~1.2 B · clinical trials ~0.7 B · Wikipedia ~0.6 B ·
133
- plus ~30 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora (e.g. MIMIC-IV,
134
- MIMIC-CXR). Register mix (chunks): formal-academic 100 M, formal-clinical 14.5 M, patient-facing 7.8 M,
135
  technical-reference 5.9 M, conversational 0.5 M.
136
 
137
  ### Known pre-training issues (disclosed)
138
- - **~35 % duplicate tokens.** An audit found ~35 % of training tokens were duplicate copies (a FineWeb-Edu shard/
139
- filter build bug re-emitted byte-identical docs, ~2.2× copies; plus ~24 % in PMC). A fully deduplicated corpus was
140
- built but **not** trained — so this base carries the duplication; dedup is a ready improvement for the next base.
141
- - **38 "sink" tokens → surgical tokenizer.** 38 orphan BPE merges (markdown/table residue) behaved as a localized
142
- attention sink; the 38 merge rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a
143
- fail-closed gate enforced at SFT/RL/serve.
144
- - **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000** — pin it on export.
145
 
146
  ## Post-training (this model)
147
  ### 1. Instruction SFT
148
- **Full-parameter** SFT (not LoRA) from the surgical-tokenizer base — ChatML template, **LR 7e-5, 1 epoch,
149
- NEFTune α=5**, whole-document packing on 8×H200. Corpus (~0.9 M+ rows): a stratified medical+tool+general majority
150
- (~615 k), a **23-family instruction-following taxonomy** (~200 k, generated with a **verifier-in-the-loop** loop —
151
  constraints sampled from the empirical IFEval distribution, teacher rejection-sampling kept only responses passing
152
- the official checker), generic function-calling (xLAM, 20 k), and grounded cite/abstain RAG (~79 k). A forgetting
153
  gate (broad perplexity ≤ 1.5× base) guards retention.
154
 
155
  ### 2. Reinforcement learning (GRPO)
156
- **GRPO** via **verl** on 8×H200, with **deterministic verifiable rewards — no reward model, no LLM judge.** The
157
- primary reward runs the *official* IFEval constraint checker (dense shaping `0.7·frac + 0.3·all_pass`); other data
158
  sources reward grounded RAG (abstain-when-unanswerable) and tool-call recall. Settings: lr 1e-6, KL 0.001
159
  (low-variance), group size n=16, 128 prompts/step, checkpoint every 40 steps (**ship the argmax**). Official IFEval
160
- AVG4 by step: **47.7 (SFT base) → 58.6 → 59.4 → 59.7 → 61.5 → 62.7 (step 1120, shipped) → 61.8**.
 
161
 
162
  ## Evaluation & benchmarks
163
- Public benchmarks only. **IFEval** = EleutherAI lm-evaluation-harness with `--apply_chat_template`, AVG4 of the four
164
- strict/loose metrics. **BFCL** = Berkeley Function-Calling Leaderboard.
 
 
165
 
166
- | Benchmark | kos-v4-instruct | Llama-3.2-3B-Instruct | Qwen2.5-3B-Instruct | Qwen3-4B-Instruct |
167
- | :--- | :--- | :--- | :--- | :--- |
168
- | **IFEval** AVG4 | **62.7** | — | 65.4 | 86.6 |
169
- | IFEval prompt-strict / inst-strict / prompt-loose / inst-loose | 55.8 / 67.0 / 58.4 / 69.7 | — | — | — |
170
- | **BFCL** simple / multiple / parallel | **0.81 / 0.73 / 0.62** | — | — | — |
171
- | *(reference)* kos-v4 SFT base, pre-RL, IFEval AVG4 | 47.7 | — | — | — |
172
-
173
- **Reading.** From a small team's from-scratch 3B, IFEval **62.7** sits ~3 points under Qwen2.5-3B (65.4) and is
174
- **+15.0 over our own SFT base** (47.7); tool-calling holds at 81/73/62 (base 84/76/65 — ~2 pts of tax for the
175
- instruction gain).
 
 
 
176
 
177
  ## Compute footprint & requirements
178
  ### Training
179
- - **Pre-training:** 24× H200 (3 nodes, pure DDP), ~5.73 days, **≈ 3,300 H200-GPU-hours**.
180
  - **SFT:** 1× 8×H200 node, 1 epoch. **RL (GRPO):** 1× 8×H200 node.
181
 
182
  ### Deployment (inference)
183
  | Precision | Approx. VRAM | Notes |
184
  | :--- | :--- | :--- |
185
- | **bfloat16** | ~7 GB | native weights (6.03 GB) + activations; 1× 16 GB GPU is comfortable |
186
- | INT8 (approx.) | ~4 GB | not shipped; typical for a 3 B model |
187
- | INT4 (approx.) | ~2.5 GB | not shipped; runs on consumer/edge hardware |
188
 
189
  ## Intended use & limitations
190
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
191
- workflows. For grounded record/guideline reading with abstention use **kos-v4-grounded-record** / **kos-v4-grounded-guideline**; for retrieval use **kos-v4-retriever**.
192
- - **NOT a medical-knowledge QA model.** Medical MCQ (MedQA/MedMCQA/MMLU-Med) ≈ 30 (≈ chance; Llama-3.2-3B ~51). It
193
- follows instructions and calls tools; it does not reliably recall parametric medical facts — **ground it with retrieval.**
194
- - This RL checkpoint's grounding/abstention regressed under IF-RL (RGB negative-rejection ~0.7) — use **kos-v4-grounded-record** / **kos-v4-grounded-guideline**
195
- when non-fabrication on unanswerable questions is required.
196
- - IFEval 62.7 trails Qwen2.5-3B (65.4) and Qwen3-4B (86.6). English only. Public-benchmark performance ≠ validation on real clinical data.
 
197
 
198
  ## The kos-v4 family
199
- - **kos-v4-instruct** — this model (general instruction + tool calling).
200
- - **[kos-v4-retriever](https://huggingface.co/Kentucky-Open-Science/kos-v4-retriever)** — clinical dense retriever (BEIR SciFact 0.7327 zero-shot).
201
- - **[kos-v4-grounded-record](https://huggingface.co/Kentucky-Open-Science/kos-v4-grounded-record)** — patient-record grounding specialist (RGB neg-rejection 35.0).
202
- - **[kos-v4-grounded-guideline](https://huggingface.co/Kentucky-Open-Science/kos-v4-grounded-guideline)** — clinical-guideline grounding specialist (RGB neg-rejection 45.0).
 
14
  - from-scratch
15
  ---
16
 
17
+ ## kos-v4: a from-scratch 3B clinical suite that reaches original-ChatGPT-level instruction-following
18
+
19
+ kos-v4 is a **first release from a small team**, not a large lab. It is a family of **3B clinical models built
20
+ completely from scratch**: a custom 32k medical tokenizer, a from-scratch pretrained base, and task-specialized
21
+ post-training, assembled into a **grounded clinical RAG system** (a retriever, grounded readers that abstain when
22
+ the answer is absent, and an instruction and tool model). The goal is not to memorize medicine; it is to reliably
23
+ follow the standard of care retrieved for it.
24
+
25
+ **David and Goliath.** The base was trained on **180 billion tokens using 24 GPUs** over about six days. For
26
+ contrast, OpenAI built the original ChatGPT (**GPT-3.5**) on a Microsoft supercomputer of roughly **10,000 GPUs**
27
+ and 285,000 CPU cores, one of the largest machines in the world at the time. kos-v4 used on the order of **400 times
28
+ fewer GPUs**, a small fraction of the data, and trained from scratch as a small group.
29
+
30
+ **And it reaches that bar.** On **IFEval**, the standard instruction-following benchmark, kos-v4-instruct scores
31
+ **AVG4 62.7**, on par with and a touch above the reported **GPT-3.5-turbo (60.7)**, and within about three points of
32
+ a current big-company 3B, **Qwen2.5-3B-Instruct (65.4)**.
33
+
34
+ | IFEval (AVG4 of the four strict and loose metrics) | score | who built it, and how |
35
+ | :-- | :-- | :-- |
36
+ | **kos-v4-instruct (ours)** | **62.7** | a small team, from scratch, 180B tokens, 24 GPUs |
37
+ | GPT-3.5-turbo, the original ChatGPT (reported) | 60.7 | OpenAI, about 10,000-GPU supercomputer |
38
+ | Qwen2.5-3B-Instruct | 65.4 | Alibaba, about 18 trillion tokens |
39
+
40
+ **Where it genuinely leads.** On the RAG tasks it was built for, kos-v4 is competitive or best in class:
41
+ - **Guideline grounding** (RGB negative-rejection: refusing when the guidance does not cover the question) is
42
+ **45.0**, the best in our comparison set, above even the newer Qwen3-4B (40.7).
43
+ - **Retrieval** (BEIR SciFact, zero-shot) is **NDCG@10 = 0.7327**, inside the strong-dense band (0.70 to 0.76) and
44
+ above BM25 (0.665).
45
+
46
+ **Read honestly.** This is a first release from a small team on a small budget, not a frontier model. Our numbers
47
+ are measured on public benchmarks; the GPT-3.5-turbo figure is OpenAI's model as reported in published
48
+ instruction-following evaluations (53.8, 64.7, 56.6, 67.5 across the four metrics). Each model card states its own
49
+ results, including where it trails.
 
 
50
 
51
  ---
52
 
 
55
  **kos-v4-instruct** is an open-weights **3B language model trained completely from scratch** by a small team. It is
56
  a decoder-only transformer (Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and
57
  tool / function calling** over clinical and general English text. Its instruction ability comes from **GRPO
58
+ reinforcement learning** on a from-scratch clinical base, reaching **IFEval AVG4 = 62.7**. That is on par with the
59
+ reported score for the original ChatGPT (**GPT-3.5-turbo, 60.7**) and within about three points of
60
+ **Qwen2.5-3B-Instruct (65.4)**, from a fraction of the data and compute. It also adds reliable **tool and function
61
+ calling** (BFCL 81/73/62), a capability the original ChatGPT did not have at launch. It is the flagship of the
62
+ **kos-v4** family (a retriever and two grounded readers ship alongside it).
63
 
64
  ## Core specifications
65
  | Attribute | Detail |
 
70
  | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
71
  | **Feed-forward** | SwiGLU, intermediate size 8192 |
72
  | **Embeddings** | Untied input/output |
73
+ | **Vocabulary** | 32,000, custom medical **byte-level BPE** (8% fewer tokens/medical-char than GPT-4/Llama/Qwen) |
74
  | **Context length** | 24,576 (`max_position_embeddings` 65,536) |
75
  | **Position encoding** | RoPE, θ = 25,000 |
76
  | **Precision** | bfloat16 |
77
  | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
78
+ | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
79
  | **License** | *to be set by Kentucky Open Science* |
80
 
81
  ## Quickstart (Hugging Face Transformers)
 
88
  model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
89
 
90
  messages = [
91
+ {"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}
92
  ]
93
  inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
94
  with torch.no_grad():
95
+ out = model.generate(inputs, max_new_tokens=256, do_sample=False)
96
  print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
97
  ```
98
  Serve with vLLM / TGI as a standard `Qwen3`-architecture causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export.
 
111
  tokens; pass tools via the tokenizer's chat template `tools=` argument.
112
 
113
  ## Pre-training (the kos-v4 base)
114
+ Trained from scratch, not distilled or continued from another model.
115
 
116
  ### Hyperparameters & schedule
117
  | Parameter | Value |
 
122
  | LR schedule | Cosine, warmup ratio 0.01, 1 epoch |
123
  | Gradient clipping | 1.0 max-norm |
124
  | Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
125
+ | Micro-batch / grad-accum | 1 / 1, giving a **global batch of 589,824 tokens/step** |
126
  | Precision / kernels | bfloat16, FlashAttention-2 |
127
+ | Total steps / tokens | 305,613 / 180.3 B token-positions |
128
 
129
  ### Training data mixture
130
+ English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
131
+ (approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B
132
+ BlueScrubs 4.6 B, Hindawi 2.9 B, MedAL 2.5 B, S2ORC 1.2 B, clinical trials 0.7 B, Wikipedia 0.6 B
133
+ plus 30 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora (e.g. MIMIC-IV
134
+ MIMIC-CXR). Register mix (chunks): formal-academic 100 M, formal-clinical 14.5 M, patient-facing 7.8 M
135
  technical-reference 5.9 M, conversational 0.5 M.
136
 
137
  ### Known pre-training issues (disclosed)
138
+ - **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/
139
+ filter build bug re-emitted byte-identical docs, 2.2× copies; plus 24 % in PMC). A fully deduplicated corpus was
140
+ built but **not** trained, so this base carries the duplication; dedup is a ready improvement for the next base.
141
+ - **38 "sink" tokens to surgical tokenizer.** 38 orphan BPE merges (markdown/table residue) behaved as a localized
142
+ attention sink; the 38 merge rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a
143
+ fail-closed gate enforced at SFT/RL/serve.
144
+ - **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000**, pin it on export.
145
 
146
  ## Post-training (this model)
147
  ### 1. Instruction SFT
148
+ **Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch
149
+ NEFTune α=5**, whole-document packing on 8×H200. Corpus (0.9 M+ rows): a stratified medical+tool+general majority
150
+ (615 k), a **23-family instruction-following taxonomy** (200 k, generated with a **verifier-in-the-loop** loop
151
  constraints sampled from the empirical IFEval distribution, teacher rejection-sampling kept only responses passing
152
+ the official checker), generic function-calling (xLAM, 20 k), and grounded cite/abstain RAG (79 k). A forgetting
153
  gate (broad perplexity ≤ 1.5× base) guards retention.
154
 
155
  ### 2. Reinforcement learning (GRPO)
156
+ **GRPO** via **verl** on 8×H200, with **deterministic verifiable rewards, no reward model, no LLM judge.** The
157
+ primary reward runs the *official* IFEval constraint checker (dense shaping `0.7*frac + 0.3*all_pass`); other data
158
  sources reward grounded RAG (abstain-when-unanswerable) and tool-call recall. Settings: lr 1e-6, KL 0.001
159
  (low-variance), group size n=16, 128 prompts/step, checkpoint every 40 steps (**ship the argmax**). Official IFEval
160
+ AVG4 rose across GRPO steps from **47.7 (SFT base)** to **62.7 at step 1120 (shipped)**, a gain of 15.0 points
161
+ (intermediate checkpoints scored 58.6, 59.4, 59.7, 61.5, then the 62.7 peak, then 61.8).
162
 
163
  ## Evaluation & benchmarks
164
+ Public benchmarks only. **IFEval** is the EleutherAI lm-evaluation-harness with `--apply_chat_template`, AVG4 of the
165
+ four strict and loose metrics. **BFCL** is the Berkeley Function-Calling AST checker. We measured kos-v4 and
166
+ Qwen2.5-3B ourselves on this harness. The GPT-3.5-turbo column is OpenAI's model **as reported** in published
167
+ instruction-following evaluations, included as the original-ChatGPT reference point rather than run by us.
168
 
169
+ | Benchmark | kos-v4-instruct (ours) | GPT-3.5-turbo (original ChatGPT, reported) | Qwen2.5-3B-Instruct |
170
+ | :--- | :--- | :--- | :--- |
171
+ | **IFEval** AVG4 | **62.7** | 60.7 | 65.4 |
172
+ | IFEval prompt-strict / inst-strict / prompt-loose / inst-loose | 55.8 / 67.0 / 58.4 / 69.7 | 53.8 / 64.7 / 56.6 / 67.5 | 59.5 / 68.5 / 62.1 / 71.3 |
173
+ | **BFCL** simple / multiple / parallel | **81 / 73 / 62** | none at launch | 95 / 89 / 75 |
174
+ | *(reference)* kos-v4 SFT base, pre-RL, IFEval AVG4 | 47.7 | | |
175
+
176
+ **Reading.** From a small team's from-scratch 3B, IFEval **62.7** lands right on the original ChatGPT
177
+ (GPT-3.5-turbo, a reported 60.7) and within about three points of Qwen2.5-3B-Instruct (65.4), while being
178
+ **15.0 points above our own SFT base** (47.7). On tool-calling, kos-v4 is functional (81/73/62), a capability the
179
+ original ChatGPT lacked at launch; general instruct models trained on far more tool data score higher (Qwen2.5-3B
180
+ at 95/89/75). Newer, more heavily distilled small models trained on 50 to 200 times more data also score higher on
181
+ IFEval; kos-v4's result is notable for its data and compute budget, not a claim to lead the current frontier.
182
 
183
  ## Compute footprint & requirements
184
  ### Training
185
+ - **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, **about 3,300 H200-GPU-hours**.
186
  - **SFT:** 1× 8×H200 node, 1 epoch. **RL (GRPO):** 1× 8×H200 node.
187
 
188
  ### Deployment (inference)
189
  | Precision | Approx. VRAM | Notes |
190
  | :--- | :--- | :--- |
191
+ | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; 1× 16 GB GPU is comfortable |
192
+ | INT8 (approx.) | 4 GB | not shipped; typical for a 3 B model |
193
+ | INT4 (approx.) | 2.5 GB | not shipped; runs on consumer/edge hardware |
194
 
195
  ## Intended use & limitations
196
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
197
+ workflows. For grounded record/guideline reading with abstention use **kos-v4-grounded-record** / **kos-v4-grounded-guideline**; for retrieval use **kos-v4-retriever**.
198
+ - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
199
+ parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
200
+ - This instruction-tuned checkpoint trades away grounded abstention (its RGB negative-rejection is near zero), so for
201
+ non-fabrication on unanswerable questions use **kos-v4-grounded-record** or **kos-v4-grounded-guideline**.
202
+ - IFEval 62.7 is on the original-ChatGPT (GPT-3.5-turbo) level and near Qwen2.5-3B (65.4); newer small models trained
203
+ on far more data score higher. English only. Strong public-benchmark numbers are not validation on real clinical data.
204
 
205
  ## The kos-v4 family
206
+ - **kos-v4-instruct**, this model (general instruction + tool calling).
207
+ - **[kos-v4-retriever](https://huggingface.co/Kentucky-Open-Science/kos-v4-retriever)**, clinical dense retriever (BEIR SciFact 0.7327 zero-shot).
208
+ - **[kos-v4-grounded-record](https://huggingface.co/Kentucky-Open-Science/kos-v4-grounded-record)**, patient-record grounding specialist (RGB neg-rejection 35.0).
209
+ - **[kos-v4-grounded-guideline](https://huggingface.co/Kentucky-Open-Science/kos-v4-grounded-guideline)**, clinical-guideline grounding specialist (RGB neg-rejection 45.0).