--- license: cc-by-nc-sa-4.0 library_name: transformers pipeline_tag: text-generation language: - en tags: - clinical - medical - pretrained - base-model - KOS-V4 - from-scratch ---
Scratch LLM
> **Code name: Scratch.** The KOS-V4 series is nicknamed **Scratch LLM**: it was trained **completely from scratch** > by a small team on a fraction of the data and compute of commercial models. It is not a frontier model. > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial, > clinical, legal, or production-grade applications. The user assumes all risks associated with its use. --- # KOS-V4-Base — a from-scratch 3B medical foundation model **KOS-V4-Base** (`kos-v4-pretrain`) is an open-weights **3B language model trained completely from scratch** by a University of Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research), [Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer (Qwen3 architecture, bespoke 3B config) — **the pretrained foundation** that [**KOS-V4-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct) is fine-tuned from. This is a **base model**: it completes text; it is not instruction-tuned and has no chat template. **The headline.** On **held-out medical bits-per-byte (5-domain mean 0.4309) KOS-V4-Base ranks 1st of 17** across a 17-model benchmark pool — beating from-scratch and trillion-token peers alike (BPB is tokenizer-agnostic, so this is a fair cross-model comparison; the distillation confound flatters the trillion-token externals, so this is conservative).
KOS-V4 training-token budget vs the comparator fleet
*Training-token budget of KOS-V4 (180.3 B tokens) against the comparator fleet (log scale). Every peer was trained on **1.7–200× more data** (0.3–36 T tokens); the biomedical specialists additionally continue-pretrain on a multi-trillion-token general base. KOS-V4 is the small dot in the lower-left — the from-scratch 3B on a fraction of the budget.* ## Core specifications | Attribute | Detail | | :--- | :--- | | **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention | | **Parameters** | 3.015 B | | **Hidden / Layers** | 3072 / 28 | | **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm | | **Feed-forward** | SwiGLU, intermediate 8192 | | **Vocabulary** | 32,000, custom medical **byte-level BPE** | | **Context length** | 24,576 (`max_position_embeddings` 65,536) | | **Position encoding** | RoPE, θ = 25,000 (pin on export) | | **Precision** | bfloat16 | | **Objective** | pure next-token cross-entropy — **no auxiliary losses** | | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) | ## Quickstart (Hugging Face Transformers) ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch model_id = "Kentucky-Open-Science/KOS-V4-Base" tok = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto") prompt = "The patient presented with acute chest pain and shortness of breath. The differential diagnosis includes" inputs = tok(prompt, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=128, do_sample=False) print(tok.decode(out[0], skip_special_tokens=True)) ``` Standard `Qwen3ForCausalLM`. **This is a base (completion) model — no chat template.** For instructions / tools use [KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct). **Pin RoPE θ = 25000** on export; `add_bos_token=false`; eos = `<|endoftext|>` (id 0). ## Tokenizer **32,000-token** byte-level BPE (BBPE), NFKC-normalized, full 256-byte alphabet (no out-of-vocabulary). The vocabulary is **32,000 tokens with 100 reserved slots** (`<|reserved_0|>` … `<|reserved_99|>`) and a **single special token `<|endoftext|>` at id 0**, which serves as **EOS = BOS = PAD = UNK** and the document separator. There are **no other special tokens** — no chat / ChatML tokens; the base was not trained on any. `add_bos_token = false`. The 38 residual byte-fragment "sink" merges present at pretraining have been **removed (corrected tokenizer)**; the vocabulary stays 32,000. BPB is per-byte, so this does not confound cross-model comparison. **Where the tokenizer does well.** On raw byte-compression efficiency (tokens-per-byte **0.2591**) it ranks **4th of 17**, essentially tied with the most byte-efficient general tokenizers (Llama-3 0.2573) — fewer tokens per document, a multiplicative saving on both training and inference. It deliberately does *not* optimize for whole-word medical vocabulary, so its medical single-token rate (T1b STRR) is the lowest in the pool — but that tradeoff is **vindicated**: despite the lowest single-token rate, the model still ranks **1st of 17 on held-out medical BPB and leads the pool on medical entity extraction**. A high single-token rate is neither necessary for nor sufficient for strong downstream medical modeling. ## Pre-training Trained **from scratch, not distilled or continued**. Pure next-token cross-entropy (**no auxiliary losses**), full-parameter on the LlamaFactory trainer, AdamW, peak LR 3.0e-4 cosine (warmup 1%), grad-clip 1.0, **1 epoch**, seq 24,576 (whole-document **neat-packing**; any doc > 24,576 tokens is dropped, never split; 4-D block-diagonal mask so there is no cross-document attention; position_ids reset per doc; 99.8% fill), bf16 + FlashAttention-2 + Liger kernels, gradient checkpointing off. **305,613 steps / 180.3 B token-positions** (batch 589,824 tokens/step). Final train loss **1.6637**. **24× H200 (3 nodes × 8), pure DDP, 5.73 days, ~3,300 H200-GPU-hours.** **Representation health is handled data-side**, not with geometric regularizers: corpus cleaning that removes structural sink tokens, source-balanced neat-packing, and train-time shuffle. KOS-V4-Base is a direct test of whether a clean tokenizer + a clean corpus — *without* auxiliary losses — yields healthy representations (the attention / geometry diagnostics below measure this). ## Pre-training datasets — 49 sources, ~180.3 B tokens, English-only Cleaned (ftfy → NFKC → whitelist; PMC body-only; radiology → natural-language headers; PHI-run collapse), single-phase. Token counts are char-estimates against the packed cache. **Backbone — biomedical literature + general web (~97% of tokens):** | source | tokens | | source | tokens | | :-- | --: | -- | :-- | --: | | PubMed Central (pmc) | 76.97 B | | mimic-iv discharge (PhysioNet) | 0.83 B | | FineWeb-Edu 350BT (FK≤10) | 40.52 B | | clinical_trials (ClinicalTrials.gov) | 0.72 B | | FineWeb-Edu 10BT | 10.38 B | | wikipedia (sci/med subset) | 0.62 B | | mMedC-en | 6.33 B | | mimic-iv radiology (PhysioNet) | 0.45 B | | BlueScrubs | 4.55 B | | biorxiv / medrxiv | 0.41 B | | Hindawi OA journals | 2.93 B | | s2orc | 1.21 B | | MeDAL (PubMed abstracts) | 2.49 B | | | | **+ ~30 smaller sources:** open clinical narratives, knowledge / guidelines (stackexchange-science 0.27 B, dailymed, cpg, gene_ontology, medlineplus, orphapacket, medmentions, trialgpt), pharmacovigilance / relational rendered to NL (ctd, faers, aeolus, onsides, sider, cdc_places, cbioportal, civic, ade_corpus_v2), and deliberate register-diversity (locus legal-code). *(A small number of additional sources with unresolved licenses are intentionally omitted from this list pending license verification; they will be added once resolved.)* **Disclosed issue:** ~35% of tokens are duplicates (a FineWeb-Edu sharding build bug + PMC repetition); a deduped corpus (`v4_dedup`, 79.4 M docs, 0% dup) is ready but was **not** trained — this release is the original single-epoch corpus. ## Evaluation — 19-test suite, 75 metrics, vs a 16-model fleet KOS-V4-Base is benchmarked as a **from-scratch, single-epoch base against 16 external models trained on 1.7–200× more data** (0.3–36 T tokens). Pool of 17 models, 95 ranked metrics. BPB (bits-per-byte) is tokenizer-agnostic and included as a ranked measure. **Tally:** **20 outright rank-1 wins · 6 best-tied (frontier parity) · 9 top-3 · 60 trailing.** Wins concentrate on held-out medical BPB (rank 1/17 mean), attention health, internal representation / spectral geometry, medical entity extraction, and reasoning discrimination. **Where it wins (rank-1 of 17):** held-out medical BPB (T1 rad / cxr / clin / 5-domain mean **0.4309**), long-context BPB (T10 L/2, L/4), attention health (T2 collapsed-head frac **0.0506**, entropy-min, BOS-sink), medical entity extraction (RadGraph DR.1/2/3 macro-F1 **0.7665 / 0.7990 / 0.8150**; BLURB T7 BIOSSES / HoC), reasoning discrimination (T6b MedThink rank-1 **0.9450**), calibration (T9 mean ECE **0.1366**). Notably, the **clean-tokenizer + clean-corpus recipe produces healthy internals with no geometric regularizer**: logit-lens decodability rises through depth (T3 final-lens accuracy **0.5715** vs BioMedLM 0.2903 — a **top-3** placement) and the weight-matrix spectra are mature (T5 WeightWatcher median power-law α wins; safe / under-trained layer ratios place **top-3**). Beyond the 20 outright wins, KOS-V4-Base reaches **frontier parity (best-tied) on 6 metrics** (near-zero dead-neuron rate; saturated needle retrieval at several depths) and **top-3 on 9 more**. **Radiology and clinical text is its home turf — and there it beats every trillion-token model in the pool.** On per-domain held-out BPB (lower = better) it ranks **1st of 17** on radiology reports (**0.4761** vs Qwen3-4B/36 T 0.7995, Llama-3-8B 0.7444, Gemma-2-9B 0.7760), chest-X-ray reports (**0.5887**), and clinical narratives (**0.3221**); and **1st of 17** on radiology entity/relation extraction (RadGraph DR.1/2/3 macro-F1 **0.7665 / 0.7990 / 0.8150**). It *trails* on the general axes the trillion-token models saturate — biomedical literature (rank 17/17) and textbooks (rank 12/17) — but the clinical/radiology wins are decisive enough to carry the **5-domain BPB mean to rank 1/17**. A 3 B model on 180 B tokens out-modeling 4–9 B models trained on 8–36 T tokens, specifically on the clinical text it was built for, is the payoff of a from-scratch clinical corpus. **Where it is weak:** - **Closed-book medical MCQ** (T6 — near-chance): as a 180 B-token base it does not reliably recall parametric medical facts; it is worst on the hardest professional/college splits (mmlu_professional_medicine **0.2096**, mmlu_college_medicine **0.2312**, both below BioMedLM). Token-volume + instruction-tuning bound. - **Long-context needle retrieval** (T10): pool-relative trailing except at the saturated depths (it does beat BioMedLM on the 3-depth mean, **0.8667 vs 0.3667**). Token-volume bound. - **Hallucination discipline** (Med-HALT T8 **FCT 0.0400 / NOTA 0.6600 / FQT 0.3868**): all three trail the specialist BioMedLM (0.162 / 0.842 / 0.705) — a real-PubMed-format, instruction-format gap the SFT/RL line addresses. - **Effective-rank geometry** (T4 RankMe **170.8** vs pool leaders' ~187) and the **byte-level tokenizer single-token rate** (T1b **0.0448** vs 0.0572): the model trades a little representational rank / single-token coverage for a smaller, byte-efficient vocabulary. (Anisotropy / isotropy also run high — a known BBPE base characteristic.) - **Demographic bias** (T11 CrowS-Pairs |disparity| **1.0791** — **rank 9/17, mid-pack, and notably *more* biased than the biomedical specialists** BioMedLM 0.3705 / MedGemma 0.2978): the clean-corpus recipe did **not** remove social-stereotype bias. Treat generations as **unaudited for fairness**. The token- and format-bound axes (T6 knowledge, T10 needle, Med-HALT NOTA / FQT, IFEval) scale with raw pretraining-token volume and instruction tuning — a **token-budget** gap rather than an architecture flaw; the format-bound ones are the job of the SFT/RL line ([KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct)). Bias and effective-rank are **not** simply token-budget artifacts and remain open limitations. ### Apples-to-apples vs BioMedLM (the only non-commercial, biomedical-only peer; ~300 B PubMed tokens) **KOS-V4-Base wins 28 of 38 comparable cells.** Selected: | test | KOS-V4-Base | BioMedLM | Δ | | :-- | --: | --: | --: | | T1 BPB 5-corpus mean (lower = better) | **0.4309** | 0.9059 | −0.475 ✅ | | T2 collapsed-head frac (lower = better) | **0.0506** | 0.4844 | −0.434 ✅ | | RadGraph-XL macro-F1 (DR.1) | **0.7665** | 0.6196 | +0.147 ✅ | | RadGraph CXR macro-F1 (DR.2) | **0.7990** | 0.6304 | +0.169 ✅ | | T6b MedThink rank-1 | **0.9450** | 0.8250 | +0.120 ✅ | | T7 BLURB mean / 4 | **0.7465** | 0.6829 | +0.064 ✅ | | pubmedqa | **0.6980** | 0.4440 | +0.254 ✅ | | T3 final-lens accuracy (higher = better) | **0.5715** | 0.2903 | +0.281 ✅ | | T10 needle 3-depth mean (higher = better) | **0.8667** | 0.3667 | +0.500 ✅ | | T9 mean ECE (lower = better) | **0.1366** | 0.2231 | −0.086 ✅ | | T8 FCT (Med-HALT) | 0.0400 | **0.1620** | −0.122 ✗ | | T8 NOTA (Med-HALT) | 0.6600 | **0.8420** | −0.182 ✗ | | T8 FQT (Med-HALT) | 0.3868 | **0.7054** | −0.319 ✗ | | T4 RankMe effective rank (higher = better) | 170.8 | **186.8** | −16.0 ✗ | | T11 CrowS-Pairs \|disparity\| (lower = better) | 1.0791 | **0.3705** | +0.709 ✗ | **KOS-V4-Base wins 28 of 38 comparable cells.** Wins concentrate on attention pathology (T2), representation flow (T3), spectral health (T5), radiology entity extraction (Block C), and chemical/disease NER. **Losses** concentrate on corpus-volume- / instruction-format-bound axes (Med-HALT FCT / NOTA / FQT, long-form closed-book MCQ, BLURB linear probes at low token count) plus two that are **not** token-budget artifacts: lower effective rank (T4) and **higher demographic-bias disparity (T11)**. BioMedLM is architecturally capped at 1024 context (GPT-2 learned positions); KOS-V4-Base (RoPE, 24,576) holds the long-context axis. ## Compute footprint Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. ## Deployment (inference) | Precision | Approx. VRAM | Notes | | :--- | :--- | :--- | | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable | ## Fine-tunes built on this base - **[KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct)** — instruction following + tool / function calling (IFEval 61.6, official BFCL 72.75 / 73 / 60.5) + **[GGUF](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct-GGUF)**. ## Intended use & limitations - **Intended use:** a **base foundation** for medical-NLP research and downstream fine-tuning. Strong on held-out medical BPB, attention / representation health, and medical entity extraction. - **Not a knowledge-QA model.** As a base LM it does not reliably recall parametric medical facts (closed-book MCQ is near-chance) — fine-tune and/or ground it with retrieval. - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data. - **Measured demographic bias.** On CrowS-Pairs (T11) the model is **mid-pack (rank 9/17) and more biased than the biomedical specialists** (|disparity| 1.0791 vs BioMedLM 0.37 / MedGemma 0.30). The clean-corpus recipe removes structural sink tokens but does **not** remove social-stereotype bias — outputs are **unaudited for fairness**. - **Measured hallucination weakness.** Med-HALT (T8 FCT / NOTA / FQT) trails the specialist BioMedLM; the model may fabricate confidently. - **Not otherwise safety-tested.** Beyond CrowS-Pairs and Med-HALT above, this model has **not** been red-teamed or evaluated for toxicity or clinical safety. It may produce harmful, biased, or medically inaccurate content.