README.md

---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- clinical
- medical
- instruction-following
- tool-calling
- function-calling
- KOS-V4
- from-scratch
---

<div align="center">
  <img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
</div>

> **Code name: Scratch.** The KOS-V4 series is nicknamed **Scratch LLM**: it was trained **completely from scratch**
> by a small team on a fraction of the data and compute of commercial models. It is not a frontier model.

> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
> clinical, legal, or production-grade applications. The user assumes all risks associated with its use.

---

# KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following

**KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a University of
Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research),
[Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer
(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.

**IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."

| IFEval **strict-avg** | model | who built it, and how |
| --: | :-- | :-- |
| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
| **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs |
| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |

KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
which the original ChatGPT lacked at launch — though modern small models score higher there.

## Core specifications
| Attribute | Detail |
| :--- | :--- |
| **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |
| **Parameters** | 3.015 B |
| **Hidden / Layers** | 3072 / 28 |
| **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm |
| **Feed-forward** | SwiGLU, intermediate 8192 |
| **Vocabulary** | 32,000, custom medical **byte-level BPE** |
| **Context length** | 24,576 (`max_position_embeddings` 65,536) |
| **Position encoding** | RoPE, θ = 25,000 (pin on export) |
| **Precision** | bfloat16 |
| **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
| **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |

## Quickstart (Hugging Face Transformers)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kentucky-Open-Science/KOS-V4-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
```
Serve with vLLM / TGI as a standard `Qwen3` causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export.

## Prompt / chat format (ChatML)
```
<|im_start|>system
{system (optional)}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
```
`<|im_end|>` is the true eos. Tool calling uses `<tool_call>` / `<tool_response>` tokens (pass tools via the chat
template `tools=` argument).

## Pre-training (the KOS-V4 base)
This model is fine-tuned from **[KOS-V4-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Base)** — the
from-scratch pretrained foundation summarized here.

Trained from scratch, not distilled or continued. Pure next-token cross-entropy (**no auxiliary losses**), AdamW,
peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps /
180.3 B token-positions. Data: English-only, medical/biomedical-first, **49 sources / 130 M chunks** (PubMed Central
77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources).

**Disclosed issues:** 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but
was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as `bad_words_ids`); RoPE θ
trained at 25,000 (a planned 10,000 was never applied).

## Post-training (this model)
**SFT:** full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified
medical+tool+general majority, a **23-family instruction taxonomy** (verifier-in-the-loop; only responses passing the
official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×).

**RL (GRPO via verl):** deterministic verifiable rewards, **no reward model, no LLM judge**; the primary reward runs
the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120
(shipped)** — the argmax over all RL checkpoints (sha256-verified as the released weights).

## Evaluation & benchmarks

**Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True
- greedy, task-default max_gen_toks=1280`),
BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
64.7, a 0.7-pt gap).

## IFEval in context (strict-avg, models our size or larger)
One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
strict-avg). Commercial rows are **Proprietary**; a **\*** means the strict value is an **estimate** (no official
strict sub-metrics published — estimated from the model's published AVG4/prompt-strict, which run a few points above
strict) and a **+** means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above).

| model | weights | company | released | params | IFEval strict-avg |
| :-- | :-- | :-- | :-- | :-- | --: |
| GPT-4o-mini | Proprietary | OpenAI | Jul 2024 | 8B + | 79 \* |
| Llama-3.2-3B-Instruct | Open | Meta | Sep 2024 | 3.2B | 73.9 |
| Qwen2.5-3B-Instruct | Open | Alibaba | Sep 2024 | 3.0B | 64.7 |
| Phi-3-medium-4k-instruct | Open | Microsoft | May 2024 | 14.0B | 64.2 |
| Mistral-Large | Proprietary | Mistral AI | Feb 2024 | 46.7B + | 63 \* |
| **KOS-V4-Instruct (ours)** | **Open** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **61.6** |
| Yi-1.5-9B-Chat | Open | 01.AI | May 2024 | 8.8B | 60.5 |
| Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 57.7 |
| GPT-3.5-turbo-0613 | Proprietary | OpenAI | Jun 2023 | 20B + | 57 \* |
| Command-R | Open | Cohere | Mar 2024 | 35B | 57 \* |
| Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 56.1 |
| GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 55.9 |
| Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 55.0 |
| Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 55 \* |
| Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 55 \* |
| Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 54.7 |
| gemma-1.1-7b-it | Open | Google | Mar 2024 | 8.5B | 50.4 |
| zephyr-7b-beta | Open | Hugging Face | Oct 2023 | 7.2B | 49.5 |
| GPT-3.5-turbo-0125 | Proprietary | OpenAI | Jan 2024 | 20B + | 49 \* |
| Llama-3.1-8B-Instruct | Open | Meta | Jul 2024 | 8.0B | 44.3 |
| Qwen1.5-7B-Chat | Open | Alibaba | Jan 2024 | 7.7B | 43.7 |
| Llama-2-13b-chat | Open | Meta | Jul 2023 | 13.0B | 39.8 |
| Qwen1.5-4B-Chat | Open | Alibaba | Jan 2024 | 4.0B | 31.6 |

**\* strict estimate** — no official IFEval strict sub-metrics are published for this model; the value is estimated
from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). **\+ unofficial params** — never
disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium
~100B, Gemini-1.0-Pro ~30B).

**Reading.** At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9,
0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, **Meta's Llama-3.1-8B (44.3)**,
Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and
the frontier proprietary models. This is a strong result **for a from-scratch 3B on 180B tokens and 24 GPUs**, not a
claim to lead the current field.

## Commercial baselines (strict-avg)
| commercial model | company | snapshot | strict-avg | basis |
| :-- | :-- | :-- | --: | :-- |
| GPT-4 | OpenAI | gpt-4-0613 | 80.6 | computed from published strict sub-metrics (77.1 / 84.1), InternLM2 report |
| Command-R+ | Cohere | c4ai-command-r-plus | 76.6 | strict sub-metrics (72.8 / 80.5), Open LLM Leaderboard raw |
| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 55.9 | strict sub-metrics (50.5 / 61.2), InternLM2 report |
| Claude-3.5-Sonnet | Anthropic | 20240620 | ~83 \* | estimate from published AVG4 86.2 (Llama 3.1 report) |
| GPT-4o | OpenAI | 2024-05-13 | ~81 \* | estimate from AVG4 84.3 |
| Gemini-1.5-Pro | Google | May 2024 | ~79 \* | estimate from AVG4 82.3 |
| Claude-3-Haiku | Anthropic | Mar 2024 | ~65 \* | estimate from AVG4 68.1 |

**Sources.** IFEval definition: `google-research/instruction_following_eval`. Open-model strict-avg:
`open-llm-leaderboard/contents` (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report
(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
few points lower), clearly labeled.

### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
| BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
| :-- | :-- | :-- | :-- |
| simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |


## Deployment (inference)
| Precision | Approx. VRAM | Notes |
| :--- | :--- | :--- |
| **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
| **GGUF Q8_0 / Q4_K_M** | ~4 / ~2.5 GB | shipped for `llama.cpp` — see **GGUF quantizations** below |

## GGUF quantizations (llama.cpp)
Ready-to-run `llama.cpp` builds are published at
[**Kentucky-Open-Science/KOS-V4-Instruct-GGUF**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct-GGUF)
(llama.cpp `b510`/18ef86e; lm-eval-harness `0.4.12`).

| file | quant | size | notes |
| :-- | :-- | --: | :-- |
| `kosv4-f16.gguf` | F16 (16.0 bpw) | 6.03 GB | full-precision reference / requantize source |
| `kosv4-Q8_0.gguf` | Q8_0 (8.5 bpw) | 3.21 GB | near-lossless |
| `kosv4-Q4_K_M.gguf` | Q4_K_M (~4.8 bpw) | 1.83 GB | **recommended** — 3.3× smaller than f16, no measurable IFEval loss |

SHA256 in `SHA256SUMS`; raw benchmark JSON + conversion/eval scripts in `testing/`.

**Quantization preserves IFEval.** Official EleutherAI lm-eval `ifeval` (`--apply_chat_template`), all 541 prompts,
greedy; only the weights differ across rows. `strict-avg = (prompt-strict + inst-strict)/2`.

| format | prompt-strict | inst-strict | **strict-avg** | Δ vs bf16 |
| :-- | --: | --: | --: | --: |
| bf16 (HF reference) | 55.82 | 67.03 | **61.42** | — |
| Q8_0 (GGUF) | 55.27 | 66.43 | **60.85** | −0.57 |
| Q4_K_M (GGUF) | 56.19 | 67.39 | **61.79** | +0.37 |

All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the
bf16 row reproduces the card's **61.6** headline (measured 61.42).

**Serving:**
```bash
llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080
```
`--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
`<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.

### Edge Deployment & Performance (NVIDIA Jetson Orin Nano)
The highly compact memory footprint of the `Q4_K_M` quantization makes this model an exceptional candidate for localized, low-power edge computing platforms using unified memory architectures. 

When deployed natively via `llama.cpp` using CUDA-offloaded layers, the model achieves the following baseline performance characteristics on an **NVIDIA Jetson Orin Nano (8GB)**:

| Phase | Throughput | Bottleneck Profile |
| :--- | :--- | :--- |
| **Prefill (Prompt Processing)** | ~143.4 t/s | Compute-Bound ($GEMM$ execution over unified RAM) |
| **Decode (Token Generation)** | ~24.8 t/s | Memory Bandwidth-Bound (Saturating the 68 GB/s bus) |

**Deployment Optimization Recommendations:**
* **Lock Hardware Clocks:** Prior to initializing `llama-server`, maximize the power envelope and lock the frequency steps to prevent dynamic frequency scaling latency:
  ```bash
  sudo nvpmodel -m 1
  sudo jetson_clocks

## Intended use & limitations
- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
- **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
- **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.