Text Generation
Transformers
Safetensors
English
qwen3
clinical
medical
instruction-following
tool-calling
function-calling
KOS-V4
from-scratch
conversational
text-generation-inference
Instructions to use Kentucky-Open-Science/KOS-V4-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kentucky-Open-Science/KOS-V4-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Kentucky-Open-Science/KOS-V4-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct") model = AutoModelForCausalLM.from_pretrained("Kentucky-Open-Science/KOS-V4-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kentucky-Open-Science/KOS-V4-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kentucky-Open-Science/KOS-V4-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
- SGLang
How to use Kentucky-Open-Science/KOS-V4-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V4-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V4-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Kentucky-Open-Science/KOS-V4-Instruct with Docker Model Runner:
docker model run hf.co/Kentucky-Open-Science/KOS-V4-Instruct
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
-
--
|
| 2 |
-
license:
|
| 3 |
library_name: transformers
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
|
@@ -12,9 +12,6 @@ tags:
|
|
| 12 |
- function-calling
|
| 13 |
- KOS-V4
|
| 14 |
- from-scratch
|
| 15 |
-
- gguf
|
| 16 |
-
- llama.cpp
|
| 17 |
-
- quantized
|
| 18 |
---
|
| 19 |
|
| 20 |
<div align="center">
|
|
@@ -37,28 +34,19 @@ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.ed
|
|
| 37 |
(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
|
| 38 |
instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
|
| 39 |
|
| 40 |
-
**
|
| 41 |
-
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
|
| 42 |
-
four-metric average anywhere.
|
| 43 |
-
|
| 44 |
-
KOS-V4 was trained on a fraction of the resources OpenAI trained the original
|
| 45 |
-
ChatGPT (GPT-3.5).
|
| 46 |
|
| 47 |
| IFEval **strict-avg** | model | who built it, and how |
|
| 48 |
| --: | :-- | :-- |
|
| 49 |
-
| **61.6** | **KOS-V4-Instruct (ours)** | Kentucky Research Team, 180B tokens, 24 GPUs |
|
| 50 |
-
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
|
| 51 |
| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
|
|
|
|
|
|
|
| 52 |
|
| 53 |
KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
|
| 54 |
commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
|
| 55 |
which the original ChatGPT lacked at launch — though modern small models score higher there.
|
| 56 |
|
| 57 |
-
**Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
|
| 58 |
-
BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
|
| 59 |
-
is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
|
| 60 |
-
64.7, a 0.7-pt gap).
|
| 61 |
-
|
| 62 |
## Core specifications
|
| 63 |
| Attribute | Detail |
|
| 64 |
| :--- | :--- |
|
|
@@ -123,20 +111,11 @@ the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from
|
|
| 123 |
|
| 124 |
## Evaluation & benchmarks
|
| 125 |
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
| **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
|
| 132 |
-
|
| 133 |
-
*(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
|
| 134 |
-
public Open LLM Leaderboard strict number for a wider set.)*
|
| 135 |
-
|
| 136 |
-
### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
|
| 137 |
-
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
|
| 138 |
-
| :-- | :-- | :-- | :-- |
|
| 139 |
-
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 140 |
|
| 141 |
## IFEval in context (strict-avg, models our size or larger)
|
| 142 |
One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
|
|
@@ -197,8 +176,11 @@ claim to lead the current field.
|
|
| 197 |
(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
|
| 198 |
few points lower), clearly labeled.
|
| 199 |
|
| 200 |
-
##
|
| 201 |
-
|
|
|
|
|
|
|
|
|
|
| 202 |
|
| 203 |
## Deployment (inference)
|
| 204 |
| Precision | Approx. VRAM | Notes |
|
|
@@ -238,13 +220,9 @@ llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 -
|
|
| 238 |
`--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
|
| 239 |
`<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
|
| 240 |
|
| 241 |
-
**Rebuilding (important):** the v4 tokenizer is a standard **ByteLevel / GPT-2-style BPE**. Convert with pre-type
|
| 242 |
-
**`gpt-2`** — this gives full token parity vs the HF tokenizer (`qwen2`/`default`/`llama-bpe`/`tekken` all diverge and
|
| 243 |
-
corrupt the ChatML scaffold). Set `tokenizer.ggml.pre = gpt-2`, `add_bos_token = false`, `rope.freq_base = 25000`.
|
| 244 |
-
Use `testing/convert_gpt2.py`; verify token parity with `testing/pretok_probe.py`.
|
| 245 |
-
|
| 246 |
## Intended use & limitations
|
| 247 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
|
| 248 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
|
| 249 |
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
|
| 250 |
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
|
|
|
|
|
|
| 1 |
+
--
|
| 2 |
+
license: cc-by-nc-sa-4.0
|
| 3 |
library_name: transformers
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
language:
|
|
|
|
| 12 |
- function-calling
|
| 13 |
- KOS-V4
|
| 14 |
- from-scratch
|
|
|
|
|
|
|
|
|
|
| 15 |
---
|
| 16 |
|
| 17 |
<div align="center">
|
|
|
|
| 34 |
(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
|
| 35 |
instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
|
| 36 |
|
| 37 |
+
**IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
|
| 38 |
+
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
| IFEval **strict-avg** | model | who built it, and how |
|
| 41 |
| --: | :-- | :-- |
|
|
|
|
|
|
|
| 42 |
| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
|
| 43 |
+
| **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs |
|
| 44 |
+
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
|
| 45 |
|
| 46 |
KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
|
| 47 |
commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
|
| 48 |
which the original ChatGPT lacked at launch — though modern small models score higher there.
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
## Core specifications
|
| 51 |
| Attribute | Detail |
|
| 52 |
| :--- | :--- |
|
|
|
|
| 111 |
|
| 112 |
## Evaluation & benchmarks
|
| 113 |
|
| 114 |
+
**Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True
|
| 115 |
+
- greedy, task-default max_gen_toks=1280`),
|
| 116 |
+
BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
|
| 117 |
+
is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
|
| 118 |
+
64.7, a 0.7-pt gap).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
## IFEval in context (strict-avg, models our size or larger)
|
| 121 |
One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
|
|
|
|
| 176 |
(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
|
| 177 |
few points lower), clearly labeled.
|
| 178 |
|
| 179 |
+
### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
|
| 180 |
+
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
|
| 181 |
+
| :-- | :-- | :-- | :-- |
|
| 182 |
+
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
|
| 183 |
+
|
| 184 |
|
| 185 |
## Deployment (inference)
|
| 186 |
| Precision | Approx. VRAM | Notes |
|
|
|
|
| 220 |
`--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
|
| 221 |
`<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
|
| 222 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
## Intended use & limitations
|
| 224 |
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
|
| 225 |
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
|
| 226 |
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
|
| 227 |
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
|
| 228 |
+
- **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.
|