Text Generation
Transformers
Safetensors
English
qwen3
clinical
medical
instruction-following
tool-calling
function-calling
KOS-V4
from-scratch
conversational
text-generation-inference
Instructions to use Kentucky-Open-Science/KOS-V4-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kentucky-Open-Science/KOS-V4-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Kentucky-Open-Science/KOS-V4-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct") model = AutoModelForCausalLM.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kentucky-Open-Science/KOS-V4-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kentucky-Open-Science/KOS-V4-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
- SGLang
How to use Kentucky-Open-Science/KOS-V4-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Kentucky-Open-Science/KOS-V4-Instruct with Docker Model Runner:
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
Add GGUF quantizations section (Q8_0/Q4_K_M, IFEval preserved)
Browse files
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
library_name: transformers
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
|
@@ -12,6 +12,9 @@ tags:
|
|
| 12 |
- function-calling
|
| 13 |
- KOS-V4
|
| 14 |
- from-scratch
|
|
|
|
|
|
|
|
|
|
| 15 |
---
|
| 16 |
|
| 17 |
<div align="center">
|
|
@@ -34,19 +37,29 @@ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.ed
|
|
| 34 |
(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
|
| 35 |
instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
|
| 36 |
|
| 37 |
-
**IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
|
| 38 |
-
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
| IFEval **strict-avg** | model | who built it, and how |
|
| 41 |
| --: | :-- | :-- |
|
| 42 |
-
|
|
| 43 |
-
| **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs |
|
| 44 |
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
|
|
|
|
| 45 |
|
| 46 |
KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
|
| 47 |
commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
|
| 48 |
which the original ChatGPT lacked at launch — though modern small models score higher there.
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
## Core specifications
|
| 51 |
| Attribute | Detail |
|
| 52 |
| :--- | :--- |
|
|
@@ -111,11 +124,20 @@ the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from
|
|
| 111 |
|
| 112 |
## Evaluation & benchmarks
|
| 113 |
|
| 114 |
-
|
| 115 |
-
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
## IFEval in context (strict-avg, models our size or larger)
|
| 121 |
One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
|
|
@@ -176,11 +198,6 @@ claim to lead the current field.
|
|
| 176 |
(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
|
| 177 |
few points lower), clearly labeled.
|
| 178 |
|
| 179 |
-
### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
|
| 180 |
-
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
|
| 181 |
-
| :-- | :-- | :-- | :-- |
|
| 182 |
-
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 183 |
-
|
| 184 |
## Compute footprint
|
| 185 |
Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
|
| 186 |
|
|
@@ -188,12 +205,47 @@ Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT +
|
|
| 188 |
| Precision | Approx. VRAM | Notes |
|
| 189 |
| :--- | :--- | :--- |
|
| 190 |
| **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
|
| 191 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 192 |
|
| 193 |
## Intended use & limitations
|
| 194 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
|
| 195 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
|
| 196 |
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
|
| 197 |
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
|
| 198 |
-
- **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.
|
| 199 |
-
|
|
|
|
| 1 |
---
|
| 2 |
+
license: other
|
| 3 |
library_name: transformers
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
|
|
|
| 12 |
- function-calling
|
| 13 |
- KOS-V4
|
| 14 |
- from-scratch
|
| 15 |
+
- gguf
|
| 16 |
+
- llama.cpp
|
| 17 |
+
- quantized
|
| 18 |
---
|
| 19 |
|
| 20 |
<div align="center">
|
|
|
|
| 37 |
(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
|
| 38 |
instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
|
| 39 |
|
| 40 |
+
**Every number in this card is IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
|
| 41 |
+
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." We do **not** use the looser
|
| 42 |
+
four-metric average anywhere.
|
| 43 |
+
|
| 44 |
+
**David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs**. OpenAI trained the original
|
| 45 |
+
ChatGPT (**GPT-3.5**) on a ~10,000-GPU supercomputer, and today's open models train on **trillions** of tokens
|
| 46 |
+
(Qwen2.5 ~18T, Llama-3 ~15T). KOS-V4's 180B tokens is **~100× less data**. From that budget:
|
| 47 |
|
| 48 |
| IFEval **strict-avg** | model | who built it, and how |
|
| 49 |
| --: | :-- | :-- |
|
| 50 |
+
| **61.6** | **KOS-V4-Instruct (ours)** | a small team, from scratch, 180B tokens, 24 GPUs |
|
|
|
|
| 51 |
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
|
| 52 |
+
| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
|
| 53 |
|
| 54 |
KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
|
| 55 |
commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
|
| 56 |
which the original ChatGPT lacked at launch — though modern small models score higher there.
|
| 57 |
|
| 58 |
+
**Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
|
| 59 |
+
BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
|
| 60 |
+
is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
|
| 61 |
+
64.7, a 0.7-pt gap).
|
| 62 |
+
|
| 63 |
## Core specifications
|
| 64 |
| Attribute | Detail |
|
| 65 |
| :--- | :--- |
|
|
|
|
| 124 |
|
| 125 |
## Evaluation & benchmarks
|
| 126 |
|
| 127 |
+
### IFEval (official lm-eval-harness, `--apply_chat_template`) — strict metrics
|
| 128 |
+
| IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
|
| 129 |
+
| :-- | --: | --: | --: | --: | --: |
|
| 130 |
+
| prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
|
| 131 |
+
| instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
|
| 132 |
+
| **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
|
| 133 |
+
|
| 134 |
+
*(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
|
| 135 |
+
public Open LLM Leaderboard strict number for a wider set.)*
|
| 136 |
+
|
| 137 |
+
### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
|
| 138 |
+
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
|
| 139 |
+
| :-- | :-- | :-- | :-- |
|
| 140 |
+
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 141 |
|
| 142 |
## IFEval in context (strict-avg, models our size or larger)
|
| 143 |
One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
|
|
|
|
| 198 |
(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
|
| 199 |
few points lower), clearly labeled.
|
| 200 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
## Compute footprint
|
| 202 |
Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
|
| 203 |
|
|
|
|
| 205 |
| Precision | Approx. VRAM | Notes |
|
| 206 |
| :--- | :--- | :--- |
|
| 207 |
| **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
|
| 208 |
+
| **GGUF Q8_0 / Q4_K_M** | ~4 / ~2.5 GB | shipped for `llama.cpp` — see **GGUF quantizations** below |
|
| 209 |
+
|
| 210 |
+
## GGUF quantizations (llama.cpp)
|
| 211 |
+
Ready-to-run `llama.cpp` builds are published at
|
| 212 |
+
[**Kentucky-Open-Science/KOS-V4-Instruct-GGUF**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct-GGUF)
|
| 213 |
+
(llama.cpp `b510`/18ef86e; lm-eval-harness `0.4.12`).
|
| 214 |
+
|
| 215 |
+
| file | quant | size | notes |
|
| 216 |
+
| :-- | :-- | --: | :-- |
|
| 217 |
+
| `kosv4-f16.gguf` | F16 (16.0 bpw) | 6.03 GB | full-precision reference / requantize source |
|
| 218 |
+
| `kosv4-Q8_0.gguf` | Q8_0 (8.5 bpw) | 3.21 GB | near-lossless |
|
| 219 |
+
| `kosv4-Q4_K_M.gguf` | Q4_K_M (~4.8 bpw) | 1.83 GB | **recommended** — 3.3× smaller than f16, no measurable IFEval loss |
|
| 220 |
+
|
| 221 |
+
SHA256 in `SHA256SUMS`; raw benchmark JSON + conversion/eval scripts in `testing/`.
|
| 222 |
+
|
| 223 |
+
**Quantization preserves IFEval.** Official EleutherAI lm-eval `ifeval` (`--apply_chat_template`), all 541 prompts,
|
| 224 |
+
greedy; only the weights differ across rows. `strict-avg = (prompt-strict + inst-strict)/2`.
|
| 225 |
+
|
| 226 |
+
| format | prompt-strict | inst-strict | **strict-avg** | Δ vs bf16 |
|
| 227 |
+
| :-- | --: | --: | --: | --: |
|
| 228 |
+
| bf16 (HF reference) | 55.82 | 67.03 | **61.42** | — |
|
| 229 |
+
| Q8_0 (GGUF) | 55.27 | 66.43 | **60.85** | −0.57 |
|
| 230 |
+
| Q4_K_M (GGUF) | 56.19 | 67.39 | **61.79** | +0.37 |
|
| 231 |
+
|
| 232 |
+
All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the
|
| 233 |
+
bf16 row reproduces the card's **61.6** headline (measured 61.42).
|
| 234 |
+
|
| 235 |
+
**Serving:**
|
| 236 |
+
```bash
|
| 237 |
+
llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080
|
| 238 |
+
```
|
| 239 |
+
`--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
|
| 240 |
+
`<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
|
| 241 |
+
|
| 242 |
+
**Rebuilding (important):** the v4 tokenizer is a standard **ByteLevel / GPT-2-style BPE**. Convert with pre-type
|
| 243 |
+
**`gpt-2`** — this gives full token parity vs the HF tokenizer (`qwen2`/`default`/`llama-bpe`/`tekken` all diverge and
|
| 244 |
+
corrupt the ChatML scaffold). Set `tokenizer.ggml.pre = gpt-2`, `add_bos_token = false`, `rope.freq_base = 25000`.
|
| 245 |
+
Use `testing/convert_gpt2.py`; verify token parity with `testing/pretok_probe.py`.
|
| 246 |
|
| 247 |
## Intended use & limitations
|
| 248 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
|
| 249 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
|
| 250 |
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
|
| 251 |
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
|
|
|
|
|
|