Text Generation
Transformers
Safetensors
English
qwen3
clinical
medical
instruction-following
tool-calling
function-calling
KOS-V4
from-scratch
conversational
text-generation-inference
Instructions to use Kentucky-Open-Science/KOS-V4-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kentucky-Open-Science/KOS-V4-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Kentucky-Open-Science/KOS-V4-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct") model = AutoModelForCausalLM.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kentucky-Open-Science/KOS-V4-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kentucky-Open-Science/KOS-V4-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
- SGLang
How to use Kentucky-Open-Science/KOS-V4-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Kentucky-Open-Science/KOS-V4-Instruct with Docker Model Runner:
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations
Browse files
README.md
CHANGED
|
@@ -18,70 +18,48 @@ tags:
|
|
| 18 |
<img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
|
| 19 |
</div>
|
| 20 |
|
| 21 |
-
> **Code name: Scratch.**
|
| 22 |
-
>
|
| 23 |
-
> it was trained **completely from scratch** by a small team, on a fraction of the data and compute, and it serves a
|
| 24 |
-
> **real, narrow purpose** (grounded clinical RAG), not a leaderboard.
|
| 25 |
|
| 26 |
> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
|
| 27 |
> clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
|
| 28 |
|
| 29 |
---
|
| 30 |
|
| 31 |
-
#
|
| 32 |
-
|
| 33 |
-
KOS-V4 is a **first native LLM release** from a University of Kentucky College of Medicine
|
| 34 |
-
[Office for Research](https://medicine.uky.edu/sites/research) and the
|
| 35 |
-
[Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/) research team. It is a family of
|
| 36 |
-
**3B models built completely from scratch from publicly curated data without the use of a commercially provided
|
| 37 |
-
base model**. The suite is built from a custom 32k tokenizer, a from-scratch pre-trained base, and task-specialized
|
| 38 |
-
post-training, assembled into a **grounded instruction and RAG system** (a retriever, grounded readers that abstain
|
| 39 |
-
when the answer is absent, and an instruction and tool model). The goal is not to memorize medicine; it is to
|
| 40 |
-
reliably follow the standard of care retrieved for it.
|
| 41 |
-
|
| 42 |
-
**David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs** over about six days. The
|
| 43 |
-
models it is measured against were built at an enormously larger scale, on two fronts. Compute: OpenAI trained the
|
| 44 |
-
original ChatGPT (**GPT-3.5**) on a Microsoft supercomputer of roughly **10,000 GPUs** (285,000 CPU cores), one of
|
| 45 |
-
the largest machines in the world at the time, on the order of **400 times more GPUs** than we used. Data: today's
|
| 46 |
-
common open models are trained on **trillions of tokens**, for example Qwen2.5 on about **18 trillion**, Qwen3 on
|
| 47 |
-
about **36 trillion**, and Llama 3 on about **15 trillion**. KOS-V4's **180 billion tokens** is **100 to 200 times
|
| 48 |
-
less data**, two orders of magnitude smaller. It reaches the original-ChatGPT bar from that small fraction of the
|
| 49 |
-
data and compute.
|
| 50 |
-
|
| 51 |
-
**And it reaches that bar.** On **IFEval**, the standard instruction-following benchmark, KOS-V4-Instruct scores
|
| 52 |
-
**AVG4 62.7**, above the proprietary **ChatGPT (gpt-3.5-turbo-1106, 58.0)** and within three points of the
|
| 53 |
-
commercially trained 3B, **Qwen2.5-3B-Instruct (65.4)**.
|
| 54 |
-
|
| 55 |
-
| IFEval (AVG4 of the four strict and loose metrics) | score | who built it, and how |
|
| 56 |
-
| :-- | :-- | :-- |
|
| 57 |
-
| **KOS-V4-Instruct (ours)** | **62.7** | a small team, from scratch, 180B tokens, 24 GPUs |
|
| 58 |
-
| GPT-3.5-turbo, the original ChatGPT (gpt-3.5-turbo-1106, verified) | 58.0 | OpenAI, about 10,000-GPU supercomputer |
|
| 59 |
-
| Qwen2.5-3B-Instruct | 65.4 | Alibaba, about 18 trillion tokens |
|
| 60 |
-
|
| 61 |
-
**Where it genuinely leads.** On the RAG tasks it was built for, KOS-V4 is competitive or best in class:
|
| 62 |
-
- **Guideline grounding** (RGB negative-rejection: refusing when the guidance does not cover the question) is
|
| 63 |
-
**45.0**, the best in our comparison set, above even the newer Qwen3-4B (40.7).
|
| 64 |
-
- **Retrieval** (BEIR SciFact, zero-shot) is **NDCG@10 = 0.7327**, inside the strong-dense band (0.70 to 0.76) and
|
| 65 |
-
above BM25 (0.665).
|
| 66 |
-
|
| 67 |
-
**Read honestly.** This is a first release from a small team on a small budget, not a frontier model. Our numbers
|
| 68 |
-
are measured on public benchmarks; the GPT-3.5-turbo figure is the verified gpt-3.5-turbo-1106 snapshot from the
|
| 69 |
-
InternLM2 technical report (50.5, 61.2, 54.9, 65.5 across the four metrics, AVG4 58.0). Each model card states its own
|
| 70 |
-
results, including where it trails.
|
| 71 |
|
| 72 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
-
|
|
|
|
|
|
|
| 75 |
|
| 76 |
-
**
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
**KOS-V4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
## Core specifications
|
| 87 |
| Attribute | Detail |
|
|
@@ -92,13 +70,12 @@ score higher here. It is the flagship of the
|
|
| 92 |
| **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
|
| 93 |
| **Feed-forward** | SwiGLU, intermediate size 8192 |
|
| 94 |
| **Embeddings** | Untied input/output |
|
| 95 |
-
| **Vocabulary** | 32,000, custom medical **byte-level BPE**
|
| 96 |
| **Context length** | 24,576 (`max_position_embeddings` 65,536) |
|
| 97 |
| **Position encoding** | RoPE, θ = 25,000 |
|
| 98 |
| **Precision** | bfloat16 |
|
| 99 |
| **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
|
| 100 |
| **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
|
| 101 |
-
| **License** | *to be set by Kentucky Open Science* |
|
| 102 |
|
| 103 |
## Quickstart (Hugging Face Transformers)
|
| 104 |
```python
|
|
@@ -129,7 +106,7 @@ KOS-V4-Instruct uses **ChatML**:
|
|
| 129 |
<|im_start|>assistant
|
| 130 |
{response}<|im_end|>
|
| 131 |
```
|
| 132 |
-
`<|im_end|>` is the true end-of-turn / eos token. Tool calling uses dedicated `<
|
| 133 |
tokens; pass tools via the tokenizer's chat template `tools=` argument.
|
| 134 |
|
| 135 |
## Pre-training (the KOS-V4 base)
|
|
@@ -139,209 +116,78 @@ Trained from scratch, not distilled or continued from another model.
|
|
| 139 |
| Parameter | Value |
|
| 140 |
| :--- | :--- |
|
| 141 |
| Objective | Pure next-token cross-entropy (**no auxiliary losses**) |
|
| 142 |
-
| Optimizer | AdamW (β1 = 0.9, β2 = 0.999
|
| 143 |
-
| Peak learning rate | 3.0e-4 |
|
| 144 |
-
| LR schedule | Cosine, warmup ratio 0.01, 1 epoch |
|
| 145 |
| Gradient clipping | 1.0 max-norm |
|
| 146 |
| Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
|
| 147 |
-
|
|
| 148 |
| Precision / kernels | bfloat16, FlashAttention-2 |
|
| 149 |
| Total steps / tokens | 305,613 / 180.3 B token-positions |
|
| 150 |
|
| 151 |
### Training data mixture
|
| 152 |
English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
|
| 153 |
-
(approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B
|
| 154 |
-
BlueScrubs 4.6 B,
|
| 155 |
-
plus 30 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora (e.g. MIMIC-IV
|
| 156 |
-
MIMIC-CXR). Register mix (chunks): formal-academic 100 M, formal-clinical 14.5 M, patient-facing 7.8 M
|
| 157 |
-
technical-reference 5.9 M, conversational 0.5 M.
|
| 158 |
|
| 159 |
### Known pre-training issues (disclosed)
|
| 160 |
-
- **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
- **38 "sink" tokens
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
- **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000**, pin it on export.
|
| 167 |
|
| 168 |
## Post-training (this model)
|
| 169 |
### 1. Instruction SFT
|
| 170 |
-
**Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
gate (broad perplexity ≤ 1.5× base) guards retention.
|
| 176 |
|
| 177 |
### 2. Reinforcement learning (GRPO)
|
| 178 |
-
**GRPO** via **verl**
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
AVG4 rose across GRPO steps from **47.7 (SFT base)** to **62.7 at step 1120 (shipped)**, a gain of 15.0 points
|
| 183 |
-
(intermediate checkpoints scored 58.6, 59.4, 59.7, 61.5, then the 62.7 peak, then 61.8).
|
| 184 |
|
| 185 |
## Evaluation & benchmarks
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
|
| 193 |
-
|
|
| 194 |
-
| **
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
| BFCL (official FC) | KOS-V4-Instruct (ours) | Qwen2.5-3B-Instruct | Llama-3.2-3B-Instruct |
|
| 201 |
-
| :--- | :--- | :--- | :--- |
|
| 202 |
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 203 |
|
| 204 |
-
**Reading.**
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
##
|
| 214 |
-
Every model is scored on the **same standard metric, IFEval AVG4** (mean of the four strict + loose sub-metrics from
|
| 215 |
-
the official IFEval implementation, `google-research/instruction_following_eval`). For open models, AVG4 is computed
|
| 216 |
-
identically from the archived **Open LLM Leaderboard** results (`open-llm-leaderboard/results`), and **company,
|
| 217 |
-
parameters, and release date are taken from `open-llm-leaderboard/contents`**. The **weights** column marks each model
|
| 218 |
-
**Open** (open weights, downloadable, whoever built it) or **Proprietary** (closed, API only); proprietary models are
|
| 219 |
-
not on the leaderboard. A **\*** on a score means the AVG4 is an estimate, and a **+** on a size means the parameter
|
| 220 |
-
count is unofficial; both are explained under the table. Models smaller than KOS-V4 are excluded. Calibration: our
|
| 221 |
-
harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
|
| 222 |
-
|
| 223 |
-
| model | weights | company | released | params | IFEval AVG4 |
|
| 224 |
-
| :-- | :-- | :-- | :-- | :-- | :-- |
|
| 225 |
-
| GPT-4o-mini | Proprietary | OpenAI | Jul 2024 | 8B + | 82 \* |
|
| 226 |
-
| Llama-3.2-3B-Instruct | Open | Meta | Sep 2024 | 3.2B | 76.6 |
|
| 227 |
-
| Qwen2.5-3B-Instruct | Open | Alibaba | Sep 2024 | 3.0B | 66.0 |
|
| 228 |
-
| Mistral-Large | Proprietary | Mistral AI | Feb 2024 | 46.7B + | 66 \* |
|
| 229 |
-
| Phi-3-medium-4k-instruct | Open | Microsoft | May 2024 | 14.0B | 65.7 |
|
| 230 |
-
| **KOS-V4-Instruct (ours)** | **Open** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **62.7** |
|
| 231 |
-
| Yi-1.5-9B-Chat | Open | 01.AI | May 2024 | 8.8B | 62.7 |
|
| 232 |
-
| GPT-3.5-turbo-0613 | Proprietary | OpenAI | Jun 2023 | 20B + | 61 \* |
|
| 233 |
-
| Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 60.5 |
|
| 234 |
-
| Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
|
| 235 |
-
| GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
|
| 236 |
-
| Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 58 \* |
|
| 237 |
-
| Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 58 \* |
|
| 238 |
-
| Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
|
| 239 |
-
| Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
|
| 240 |
-
| Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 56.6 |
|
| 241 |
-
| gemma-1.1-7b-it | Open | Google | Mar 2024 | 8.5B | 52.9 |
|
| 242 |
-
| GPT-3.5-turbo-0125 | Proprietary | OpenAI | Jan 2024 | 20B + | 52.1 |
|
| 243 |
-
| Zephyr-7B-beta | Open | Hugging Face | Oct 2023 | 7.2B | 51.2 |
|
| 244 |
-
| Llama-3.1-8B-Instruct | Open | Meta | Jul 2024 | 8.0B | 51.0 |
|
| 245 |
-
| Llama-2-13b-chat | Open | Meta | Jul 2023 | 13.0B | 46.5 |
|
| 246 |
-
| Qwen1.5-7B-Chat | Open | Alibaba | Jan 2024 | 7.7B | 46.1 |
|
| 247 |
-
| Qwen1.5-4B-Chat | Open | Alibaba | Jan 2024 | 4.0B | 33.2 |
|
| 248 |
-
|
| 249 |
-
**\* AVG4 is an estimate.** No official four-metric IFEval AVG4 is published for this model; the value is estimated
|
| 250 |
-
from adjacent evidence (a prompt-strict-only score, or the model's capability class). Point value with range, method,
|
| 251 |
-
and confidence are in the baseline block below and in `COMMERCIAL_IFEVAL_BASELINES.md`. Rows with no marker are
|
| 252 |
-
measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
|
| 253 |
-
|
| 254 |
-
**\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
|
| 255 |
-
GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 100B, Gemini-1.0-Pro about 30B
|
| 256 |
-
(public estimates span 13B to over 500B; 30B is a low-end estimate).
|
| 257 |
-
|
| 258 |
-
**Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**
|
| 259 |
-
(1106 = 58.0, 0125 = 52.1, and the estimated 0613 = 61), ties the 8.8B Yi-1.5, and clears the estimated Command-R,
|
| 260 |
-
Gemini-1.0-Pro and Mistral-Medium. It outscores every 7B-14B open instruct model below it, including both Mistral-7B-Instruct
|
| 261 |
-
releases, **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and Qwen1.5-7B. **The newest major
|
| 262 |
-
model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Ahead of it sit three open models
|
| 263 |
-
(Llama-3.2-3B, Qwen2.5-3B, Phi-3-medium-14B) and two proprietary estimates (GPT-4o-mini, Mistral-Large).
|
| 264 |
-
(Llama-3.1-8B's leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived
|
| 265 |
-
value.)
|
| 266 |
-
|
| 267 |
-
**Commercial baselines (IFEval AVG4, same metric as the table above).** KOS-V4's 62.7 sits at the top of the original
|
| 268 |
-
commercial-API tier and below the 2024 frontier. Rows are grouped by evidence quality. **Verified** = AVG4 computed
|
| 269 |
-
here from four published sub-metrics. **Reported** = a four-metric AVG4 published by a primary source (sub-metrics not
|
| 270 |
-
broken out). **Estimate (EST)** = no IFEval AVG4 is published; estimated from adjacent evidence and clearly labeled,
|
| 271 |
-
not to be cited as measured.
|
| 272 |
-
|
| 273 |
-
Verified (four sub-metrics, AVG4 computed here):
|
| 274 |
-
|
| 275 |
-
| commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
|
| 276 |
-
| :-- | :-- | :-- | :-- | :-- |
|
| 277 |
-
| GPT-4 | OpenAI | gpt-4-0613 | 82.1 | InternLM2 report (arXiv:2403.17297) |
|
| 278 |
-
| Command-R+ | Cohere | c4ai-command-r-plus | 77.9 | Open LLM Leaderboard raw results |
|
| 279 |
-
| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 58.0 | InternLM2 report (arXiv:2403.17297) |
|
| 280 |
-
|
| 281 |
-
Reported (four-metric AVG4 published, sub-metrics not broken out):
|
| 282 |
-
|
| 283 |
-
| commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
|
| 284 |
-
| :-- | :-- | :-- | :-- | :-- |
|
| 285 |
-
| Claude-3.5-Sonnet | Anthropic | claude-3-5-sonnet-20240620 | 86.2 | Llama 3.1 report (arXiv:2407.21783) |
|
| 286 |
-
| GPT-4o | OpenAI | gpt-4o-2024-05-13 | 84.3 | Llama 3.1 report |
|
| 287 |
-
| Gemini-1.5-Pro | Google | May 2024 | 82.3 | Llama 3.1 report |
|
| 288 |
-
| GPT-4-turbo | OpenAI | gpt-4-1106-preview | 81.4 | Llama 3.1 report |
|
| 289 |
-
| Gemini-1.5-Flash | Google | May 2024 | 77.2 | Llama 3.1 report |
|
| 290 |
-
| Claude-3-Sonnet | Anthropic | Mar 2024 | 72.3 | Llama 3.1 report |
|
| 291 |
-
| Claude-3-Haiku | Anthropic | Mar 2024 | 68.1 | Llama 3.1 report |
|
| 292 |
-
| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | 52.1 | Llama 3.1 report |
|
| 293 |
-
|
| 294 |
-
Estimate (no published IFEval AVG4; EST values, not measured):
|
| 295 |
-
|
| 296 |
-
| commercial model | company | endpoint / snapshot | IFEval AVG4 | basis |
|
| 297 |
-
| :-- | :-- | :-- | :-- | :-- |
|
| 298 |
-
| GPT-4o-mini | OpenAI | gpt-4o-mini-2024-07-18 | 82 EST (79-84) | class anchor to GPT-4o |
|
| 299 |
-
| Mistral-Large | Mistral AI | mistral-large-2402 | 66 EST (60-72) | class anchor |
|
| 300 |
-
| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0613 | 61 EST (60-63) | uplift from prompt-strict 54.4 (Conifer) |
|
| 301 |
-
| Gemini-1.0-Pro | Google | gemini-1.0-pro-001 | 58 EST (55-61) | class anchor (report uses InFoBench, not IFEval) |
|
| 302 |
-
| Command-R | Cohere | command-r (Mar 2024) | 60 EST (54-68) | anchor to Command-R+ |
|
| 303 |
-
| Mistral-Medium | Mistral AI | mistral-medium-2312 | 58 EST (54-63) | class anchor |
|
| 304 |
-
|
| 305 |
-
The GPT-3.5-turbo family spans AVG4 52.1 (0125) to 58.0 (1106) to 61 est. (0613) across snapshots. KOS-V4's 62.7 sits
|
| 306 |
-
at or above the top of that band on every snapshot with data.
|
| 307 |
-
|
| 308 |
-
**Takeaway.** KOS-V4 (62.7) clears the entire original ChatGPT / GPT-3.5-turbo generation and the estimated
|
| 309 |
-
Gemini-1.0-Pro and Command-R tier. The 2024 frontier (GPT-4, GPT-4o, Claude-3.5-Sonnet, Gemini-1.5, Command-R+) stays
|
| 310 |
-
well ahead. A from-scratch 3B on 180B tokens and 24 GPUs, at the instruction-following level of the first commercial
|
| 311 |
-
ChatGPT generation.
|
| 312 |
-
|
| 313 |
-
**Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
|
| 314 |
-
Open-model AVG4: computed from `open-llm-leaderboard/results`, with company, parameters and release date from
|
| 315 |
-
`open-llm-leaderboard/contents`. Commercial verified sub-metrics: InternLM2 technical report (arXiv:2403.17297) for
|
| 316 |
-
GPT-4-0613 and GPT-3.5-turbo-1106; Open LLM Leaderboard raw results for Command-R+. Commercial reported AVG4: Llama 3.1
|
| 317 |
-
technical report (arXiv:2407.21783), Table 2. GPT-3.5-turbo-0613 prompt-strict: Conifer (arXiv:2404.02823). Estimate
|
| 318 |
-
rows are derived, not measured, and are labeled EST. Full dataset with methods and confidence:
|
| 319 |
-
`COMMERCIAL_IFEVAL_BASELINES.md`.
|
| 320 |
-
|
| 321 |
-
## Compute footprint & requirements
|
| 322 |
-
### Training
|
| 323 |
-
- **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, **about 3,300 H200-GPU-hours**.
|
| 324 |
-
- **SFT:** 1× 8×H200 node, 1 epoch. **RL (GRPO):** 1× 8×H200 node.
|
| 325 |
-
|
| 326 |
-
### Deployment (inference)
|
| 327 |
| Precision | Approx. VRAM | Notes |
|
| 328 |
| :--- | :--- | :--- |
|
| 329 |
-
| **bfloat16** | 7 GB | native weights (6.03 GB) + activations;
|
| 330 |
-
| INT8 (approx.) | 4 GB | not shipped; typical for a 3 B model |
|
| 331 |
-
| INT4 (approx.) | 2.5 GB | not shipped; runs on consumer/edge hardware |
|
| 332 |
|
| 333 |
## Intended use & limitations
|
| 334 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
|
| 335 |
-
workflows.
|
| 336 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
|
| 337 |
parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
|
| 338 |
-
-
|
| 339 |
-
|
| 340 |
-
-
|
| 341 |
-
on far more data score higher. English only. Strong public-benchmark numbers are not validation on real clinical data.
|
| 342 |
-
|
| 343 |
-
## The KOS-V4 family
|
| 344 |
-
- **KOS-V4-Instruct**, this model (general instruction + tool calling).
|
| 345 |
-
- **[KOS-V4-Retriever](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Retriever)**, clinical dense retriever (BEIR SciFact 0.7327 zero-shot).
|
| 346 |
-
- **[KOS-V4-Grounded-Record](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Grounded-Record)**, patient-record grounding specialist (RGB neg-rejection 35.0).
|
| 347 |
-
- **[KOS-V4-Grounded-Guideline](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Grounded-Guideline)**, clinical-guideline grounding specialist (RGB neg-rejection 45.0).
|
|
|
|
| 18 |
<img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
|
| 19 |
</div>
|
| 20 |
|
| 21 |
+
> **Code name: Scratch.** KOS-V4 is nicknamed **Scratch LLM** because it was trained **completely from scratch** by a
|
| 22 |
+
> small team on a fraction of the data and compute of commercial models. It is not a frontier model.
|
|
|
|
|
|
|
| 23 |
|
| 24 |
> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
|
| 25 |
> clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
|
| 26 |
|
| 27 |
---
|
| 28 |
|
| 29 |
+
# KOS-V4-Instruct — a from-scratch 3B instruction & tool model
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
+
**KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a University of
|
| 32 |
+
Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research),
|
| 33 |
+
[Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer
|
| 34 |
+
(Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and tool / function calling**
|
| 35 |
+
over clinical and general English text. It was pre-trained on **180 billion tokens using 24 GPUs**, then
|
| 36 |
+
post-trained with full-parameter SFT followed by **GRPO reinforcement learning** against the official IFEval checker.
|
| 37 |
+
|
| 38 |
+
## Headline result (official benchmarks only)
|
| 39 |
+
|
| 40 |
+
Every number in this card comes from an **official benchmark suite** — **IFEval** via the EleutherAI
|
| 41 |
+
lm-evaluation-harness (`--apply_chat_template`) and **BFCL** via the official `bfcl_eval` package. No in-house
|
| 42 |
+
scorers are used.
|
| 43 |
|
| 44 |
+
**IFEval is reported here as the strict-average: `(prompt-level strict accuracy + instruction-level strict accuracy) / 2`.**
|
| 45 |
+
This is the same "IFEval" metric published on the Hugging Face Open LLM Leaderboard. We do **not** headline the looser
|
| 46 |
+
four-metric average (which reads a few points higher).
|
| 47 |
|
| 48 |
+
| IFEval **strict-avg** | model | notes |
|
| 49 |
+
| --: | :-- | :-- |
|
| 50 |
+
| **61.6** | **KOS-V4-Instruct (ours)** | from scratch, 180 B tokens, 24 GPUs |
|
| 51 |
+
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI |
|
| 52 |
+
| 64.0 | Qwen2.5-3B-Instruct | Alibaba |
|
| 53 |
+
| 75.2 | Llama-3.2-3B-Instruct | Meta |
|
| 54 |
+
| 85.0 | Qwen3-4B-Instruct | Alibaba |
|
| 55 |
+
|
| 56 |
+
**Honest standing.** KOS-V4-Instruct (61.6) is **above the original GPT-3.5-turbo generation (55.9)** and **below
|
| 57 |
+
current small instruct models** — Qwen2.5-3B (64.0), Llama-3.2-3B (75.2) and Qwen3-4B (85.0), all of which were
|
| 58 |
+
trained on far more data and instruction-tuning. This is a strong result **for its data and compute budget**, not a
|
| 59 |
+
claim to lead the current field.
|
| 60 |
+
|
| 61 |
+
Peer numbers are from the **same official lm-eval harness**; GPT-3.5-turbo-1106 is the verified snapshot from the
|
| 62 |
+
InternLM2 technical report (strict sub-metrics 50.5 / 61.2).
|
| 63 |
|
| 64 |
## Core specifications
|
| 65 |
| Attribute | Detail |
|
|
|
|
| 70 |
| **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
|
| 71 |
| **Feed-forward** | SwiGLU, intermediate size 8192 |
|
| 72 |
| **Embeddings** | Untied input/output |
|
| 73 |
+
| **Vocabulary** | 32,000, custom medical **byte-level BPE** |
|
| 74 |
| **Context length** | 24,576 (`max_position_embeddings` 65,536) |
|
| 75 |
| **Position encoding** | RoPE, θ = 25,000 |
|
| 76 |
| **Precision** | bfloat16 |
|
| 77 |
| **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
|
| 78 |
| **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
|
|
|
|
| 79 |
|
| 80 |
## Quickstart (Hugging Face Transformers)
|
| 81 |
```python
|
|
|
|
| 106 |
<|im_start|>assistant
|
| 107 |
{response}<|im_end|>
|
| 108 |
```
|
| 109 |
+
`<|im_end|>` is the true end-of-turn / eos token. Tool calling uses dedicated `<tool_call>` / `<tool_response>`
|
| 110 |
tokens; pass tools via the tokenizer's chat template `tools=` argument.
|
| 111 |
|
| 112 |
## Pre-training (the KOS-V4 base)
|
|
|
|
| 116 |
| Parameter | Value |
|
| 117 |
| :--- | :--- |
|
| 118 |
| Objective | Pure next-token cross-entropy (**no auxiliary losses**) |
|
| 119 |
+
| Optimizer | AdamW (β1 = 0.9, β2 = 0.999), weight decay 0.0 |
|
| 120 |
+
| Peak learning rate | 3.0e-4, cosine, warmup ratio 0.01, 1 epoch |
|
|
|
|
| 121 |
| Gradient clipping | 1.0 max-norm |
|
| 122 |
| Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
|
| 123 |
+
| Global batch | 589,824 tokens/step |
|
| 124 |
| Precision / kernels | bfloat16, FlashAttention-2 |
|
| 125 |
| Total steps / tokens | 305,613 / 180.3 B token-positions |
|
| 126 |
|
| 127 |
### Training data mixture
|
| 128 |
English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
|
| 129 |
+
(approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B,
|
| 130 |
+
BlueScrubs 4.6 B, plus ~40 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora.
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
### Known pre-training issues (disclosed)
|
| 133 |
+
- **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/filter
|
| 134 |
+
build bug plus PMC repetition). A fully deduplicated corpus was built but **not** trained; this base carries the
|
| 135 |
+
duplication, and dedup is a ready improvement for the next base.
|
| 136 |
+
- **38 "sink" tokens → surgical tokenizer.** 38 orphan BPE merges behaved as a localized attention sink; the merge
|
| 137 |
+
rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a fail-closed gate enforced.
|
| 138 |
+
- **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000** — pin it on export.
|
|
|
|
| 139 |
|
| 140 |
## Post-training (this model)
|
| 141 |
### 1. Instruction SFT
|
| 142 |
+
**Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch, NEFTune
|
| 143 |
+
α=5**, whole-document packing. Corpus (~0.9 M rows): a stratified medical + tool + general majority, a **23-family
|
| 144 |
+
instruction-following taxonomy** (generated with a verifier-in-the-loop loop, keeping only responses that pass the
|
| 145 |
+
official IFEval checker), generic function-calling (xLAM), and grounded cite/abstain RAG. A forgetting gate (broad
|
| 146 |
+
perplexity ≤ 1.5× base) guards retention.
|
|
|
|
| 147 |
|
| 148 |
### 2. Reinforcement learning (GRPO)
|
| 149 |
+
**GRPO** via **verl**, with **deterministic verifiable rewards — no reward model, no LLM judge.** The primary reward
|
| 150 |
+
runs the *official* IFEval constraint checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to
|
| 151 |
+
61.6 at step 1120 (shipped)**. The shipped weights are checkpoint **step 1120**, selected as the argmax of the
|
| 152 |
+
official strict-avg over all RL checkpoints.
|
|
|
|
|
|
|
| 153 |
|
| 154 |
## Evaluation & benchmarks
|
| 155 |
+
|
| 156 |
+
### IFEval (official EleutherAI lm-eval-harness, `--apply_chat_template`)
|
| 157 |
+
Strict sub-metrics and the strict-average (the headline metric):
|
| 158 |
+
|
| 159 |
+
| IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
|
| 160 |
+
| :-- | --: | --: | --: | --: | --: |
|
| 161 |
+
| prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
|
| 162 |
+
| instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
|
| 163 |
+
| **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
|
| 164 |
+
|
| 165 |
+
### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel, higher is better)
|
| 166 |
+
|
| 167 |
+
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
|
| 168 |
+
| :-- | :-- | :-- | :-- |
|
|
|
|
|
|
|
| 169 |
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 170 |
|
| 171 |
+
**Reading.** On instruction following, KOS-V4-Instruct (strict-avg 61.6) clears the original GPT-3.5-turbo generation
|
| 172 |
+
but trails today's small instruct models. On tool calling it works, but again trails the modern small models, which
|
| 173 |
+
were trained on far more tool-use data. The result is notable for a from-scratch 3B on 180 B tokens and 24 GPUs — not
|
| 174 |
+
a claim to lead the current frontier.
|
| 175 |
+
|
| 176 |
+
## Compute footprint
|
| 177 |
+
- **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, ~3,300 H200-GPU-hours.
|
| 178 |
+
- **SFT + RL (GRPO):** single 8-GPU node each.
|
| 179 |
+
|
| 180 |
+
## Deployment (inference)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
| Precision | Approx. VRAM | Notes |
|
| 182 |
| :--- | :--- | :--- |
|
| 183 |
+
| **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
|
| 184 |
+
| INT8 / INT4 (approx.) | 4 / 2.5 GB | not shipped; typical for a 3 B model |
|
|
|
|
| 185 |
|
| 186 |
## Intended use & limitations
|
| 187 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
|
| 188 |
+
workflows.
|
| 189 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
|
| 190 |
parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
|
| 191 |
+
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this
|
| 192 |
+
model's results are notable for its data/compute budget, not for leading the field.
|
| 193 |
+
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|