--- license: cc-by-nc-sa-4.0 library_name: transformers pipeline_tag: text-generation language: - en tags: - clinical - medical - instruction-following - tool-calling - function-calling - KOS-V4 - from-scratch ---
Scratch LLM
> **Code name: Scratch.** The KOS-V4 series is nicknamed **Scratch LLM**: it was trained **completely from scratch** > by a small team on a fraction of the data and compute of commercial models. It is not a frontier model. > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial, > clinical, legal, or production-grade applications. The user assumes all risks associated with its use. --- # KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following **KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a University of Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research), [Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base. **IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2` — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." | IFEval **strict-avg** | model | who built it, and how | | --: | :-- | :-- | | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens | | **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs | | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer | KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5), which the original ChatGPT lacked at launch — though modern small models score higher there. ## Core specifications | Attribute | Detail | | :--- | :--- | | **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention | | **Parameters** | 3.015 B | | **Hidden / Layers** | 3072 / 28 | | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm | | **Feed-forward** | SwiGLU, intermediate 8192 | | **Vocabulary** | 32,000, custom medical **byte-level BPE** | | **Context length** | 24,576 (`max_position_embeddings` 65,536) | | **Position encoding** | RoPE, θ = 25,000 (pin on export) | | **Precision** | bfloat16 | | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) | | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) | ## Quickstart (Hugging Face Transformers) ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "Kentucky-Open-Science/KOS-V4-Instruct" tok = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto") messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}] inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(inputs, max_new_tokens=256, do_sample=False) print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True)) ``` Serve with vLLM / TGI as a standard `Qwen3` causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export. ## Prompt / chat format (ChatML) ``` <|im_start|>system {system (optional)}<|im_end|> <|im_start|>user {user}<|im_end|> <|im_start|>assistant {response}<|im_end|> ``` `<|im_end|>` is the true eos. Tool calling uses `` / `` tokens (pass tools via the chat template `tools=` argument). ## Pre-training (the KOS-V4 base) This model is fine-tuned from **[KOS-V4-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Base)** — the from-scratch pretrained foundation summarized here. Trained from scratch, not distilled or continued. Pure next-token cross-entropy (**no auxiliary losses**), AdamW, peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps / 180.3 B token-positions. Data: English-only, medical/biomedical-first, **49 sources / 130 M chunks** (PubMed Central 77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources). **Disclosed issues:** 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as `bad_words_ids`); RoPE θ trained at 25,000 (a planned 10,000 was never applied). ## Post-training (this model) **SFT:** full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified medical+tool+general majority, a **23-family instruction taxonomy** (verifier-in-the-loop; only responses passing the official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×). **RL (GRPO via verl):** deterministic verifiable rewards, **no reward model, no LLM judge**; the primary reward runs the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120 (shipped)** — the argmax over all RL checkpoints (sha256-verified as the released weights). ## Evaluation & benchmarks **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True - greedy, task-default max_gen_toks=1280`), BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6 is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's 64.7, a 0.7-pt gap). ## IFEval in context (strict-avg, models our size or larger) One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is* strict-avg). Commercial rows are **Proprietary**; a **\*** means the strict value is an **estimate** (no official strict sub-metrics published — estimated from the model's published AVG4/prompt-strict, which run a few points above strict) and a **+** means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above). | model | weights | company | released | params | IFEval strict-avg | | :-- | :-- | :-- | :-- | :-- | --: | | GPT-4o-mini | Proprietary | OpenAI | Jul 2024 | 8B + | 79 \* | | Llama-3.2-3B-Instruct | Open | Meta | Sep 2024 | 3.2B | 73.9 | | Qwen2.5-3B-Instruct | Open | Alibaba | Sep 2024 | 3.0B | 64.7 | | Phi-3-medium-4k-instruct | Open | Microsoft | May 2024 | 14.0B | 64.2 | | Mistral-Large | Proprietary | Mistral AI | Feb 2024 | 46.7B + | 63 \* | | **KOS-V4-Instruct (ours)** | **Open** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **61.6** | | Yi-1.5-9B-Chat | Open | 01.AI | May 2024 | 8.8B | 60.5 | | Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 57.7 | | GPT-3.5-turbo-0613 | Proprietary | OpenAI | Jun 2023 | 20B + | 57 \* | | Command-R | Open | Cohere | Mar 2024 | 35B | 57 \* | | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 56.1 | | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 55.9 | | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 55.0 | | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 55 \* | | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 55 \* | | Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 54.7 | | gemma-1.1-7b-it | Open | Google | Mar 2024 | 8.5B | 50.4 | | zephyr-7b-beta | Open | Hugging Face | Oct 2023 | 7.2B | 49.5 | | GPT-3.5-turbo-0125 | Proprietary | OpenAI | Jan 2024 | 20B + | 49 \* | | Llama-3.1-8B-Instruct | Open | Meta | Jul 2024 | 8.0B | 44.3 | | Qwen1.5-7B-Chat | Open | Alibaba | Jan 2024 | 7.7B | 43.7 | | Llama-2-13b-chat | Open | Meta | Jul 2023 | 13.0B | 39.8 | | Qwen1.5-4B-Chat | Open | Alibaba | Jan 2024 | 4.0B | 31.6 | **\* strict estimate** — no official IFEval strict sub-metrics are published for this model; the value is estimated from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). **\+ unofficial params** — never disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium ~100B, Gemini-1.0-Pro ~30B). **Reading.** At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9, 0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, **Meta's Llama-3.1-8B (44.3)**, Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and the frontier proprietary models. This is a strong result **for a from-scratch 3B on 180B tokens and 24 GPUs**, not a claim to lead the current field. ## Commercial baselines (strict-avg) | commercial model | company | snapshot | strict-avg | basis | | :-- | :-- | :-- | --: | :-- | | GPT-4 | OpenAI | gpt-4-0613 | 80.6 | computed from published strict sub-metrics (77.1 / 84.1), InternLM2 report | | Command-R+ | Cohere | c4ai-command-r-plus | 76.6 | strict sub-metrics (72.8 / 80.5), Open LLM Leaderboard raw | | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 55.9 | strict sub-metrics (50.5 / 61.2), InternLM2 report | | Claude-3.5-Sonnet | Anthropic | 20240620 | ~83 \* | estimate from published AVG4 86.2 (Llama 3.1 report) | | GPT-4o | OpenAI | 2024-05-13 | ~81 \* | estimate from AVG4 84.3 | | Gemini-1.5-Pro | Google | May 2024 | ~79 \* | estimate from AVG4 82.3 | | Claude-3-Haiku | Anthropic | Mar 2024 | ~65 \* | estimate from AVG4 68.1 | **Sources.** IFEval definition: `google-research/instruction_following_eval`. Open-model strict-avg: `open-llm-leaderboard/contents` (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a few points lower), clearly labeled. ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel) | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B | | :-- | :-- | :-- | :-- | | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 | ## Deployment (inference) | Precision | Approx. VRAM | Notes | | :--- | :--- | :--- | | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable | | **GGUF Q8_0 / Q4_K_M** | ~4 / ~2.5 GB | shipped for `llama.cpp` — see **GGUF quantizations** below | ## GGUF quantizations (llama.cpp) Ready-to-run `llama.cpp` builds are published at [**Kentucky-Open-Science/KOS-V4-Instruct-GGUF**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct-GGUF) (llama.cpp `b510`/18ef86e; lm-eval-harness `0.4.12`). | file | quant | size | notes | | :-- | :-- | --: | :-- | | `kosv4-f16.gguf` | F16 (16.0 bpw) | 6.03 GB | full-precision reference / requantize source | | `kosv4-Q8_0.gguf` | Q8_0 (8.5 bpw) | 3.21 GB | near-lossless | | `kosv4-Q4_K_M.gguf` | Q4_K_M (~4.8 bpw) | 1.83 GB | **recommended** — 3.3× smaller than f16, no measurable IFEval loss | SHA256 in `SHA256SUMS`; raw benchmark JSON + conversion/eval scripts in `testing/`. **Quantization preserves IFEval.** Official EleutherAI lm-eval `ifeval` (`--apply_chat_template`), all 541 prompts, greedy; only the weights differ across rows. `strict-avg = (prompt-strict + inst-strict)/2`. | format | prompt-strict | inst-strict | **strict-avg** | Δ vs bf16 | | :-- | --: | --: | --: | --: | | bf16 (HF reference) | 55.82 | 67.03 | **61.42** | — | | Q8_0 (GGUF) | 55.27 | 66.43 | **60.85** | −0.57 | | Q4_K_M (GGUF) | 56.19 | 67.39 | **61.79** | +0.37 | All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the bf16 row reproduces the card's **61.6** headline (measured 61.42). **Serving:** ```bash llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080 ``` `--jinja` is **required** for the model's `` output to be parsed into structured `tool_calls`; eos is `<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior. ### Edge Deployment & Performance (NVIDIA Jetson Orin Nano) The highly compact memory footprint of the `Q4_K_M` quantization makes this model an exceptional candidate for localized, low-power edge computing platforms using unified memory architectures. When deployed natively via `llama.cpp` using CUDA-offloaded layers, the model achieves the following baseline performance characteristics on an **NVIDIA Jetson Orin Nano (8GB)**: | Phase | Throughput | Bottleneck Profile | | :--- | :--- | :--- | | **Prefill (Prompt Processing)** | ~143.4 t/s | Compute-Bound ($GEMM$ execution over unified RAM) | | **Decode (Token Generation)** | ~24.8 t/s | Memory Bandwidth-Bound (Saturating the 68 GB/s bus) | **Deployment Optimization Recommendations:** * **Lock Hardware Clocks:** Prior to initializing `llama-server`, maximize the power envelope and lock the frequency steps to prevent dynamic frequency scaling latency: ```bash sudo nvpmodel -m 1 sudo jetson_clocks ## Intended use & limitations - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows. - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.** - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field. - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data. - **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.