codybum commited on
Commit
7688fad
·
verified ·
1 Parent(s): bb42310

Add GGUF quantizations section (Q8_0/Q4_K_M, IFEval preserved)

Browse files
Files changed (1) hide show
  1. README.md +70 -18
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- license: cc-by-nc-sa-4.0
3
  library_name: transformers
4
  pipeline_tag: text-generation
5
  language:
@@ -12,6 +12,9 @@ tags:
12
  - function-calling
13
  - KOS-V4
14
  - from-scratch
 
 
 
15
  ---
16
 
17
  <div align="center">
@@ -34,19 +37,29 @@ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.ed
34
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
35
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
36
 
37
- **IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
38
- — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
 
 
 
 
 
39
 
40
  | IFEval **strict-avg** | model | who built it, and how |
41
  | --: | :-- | :-- |
42
- | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
43
- | **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs |
44
  | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
 
45
 
46
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
47
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
48
  which the original ChatGPT lacked at launch — though modern small models score higher there.
49
 
 
 
 
 
 
50
  ## Core specifications
51
  | Attribute | Detail |
52
  | :--- | :--- |
@@ -111,11 +124,20 @@ the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from
111
 
112
  ## Evaluation & benchmarks
113
 
114
- **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True
115
- - greedy, task-default max_gen_toks=1280`),
116
- BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
117
- is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
118
- 64.7, a 0.7-pt gap).
 
 
 
 
 
 
 
 
 
119
 
120
  ## IFEval in context (strict-avg, models our size or larger)
121
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
@@ -176,11 +198,6 @@ claim to lead the current field.
176
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
177
  few points lower), clearly labeled.
178
 
179
- ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
180
- | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
181
- | :-- | :-- | :-- | :-- |
182
- | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
183
-
184
  ## Compute footprint
185
  Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
186
 
@@ -188,12 +205,47 @@ Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT +
188
  | Precision | Approx. VRAM | Notes |
189
  | :--- | :--- | :--- |
190
  | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
191
- | INT8 / INT4 (approx.) | 4 / 2.5 GB | not shipped; typical for a 3 B model |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
192
 
193
  ## Intended use & limitations
194
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
195
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
196
  - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
197
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
198
- - **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.
199
-
 
1
  ---
2
+ license: other
3
  library_name: transformers
4
  pipeline_tag: text-generation
5
  language:
 
12
  - function-calling
13
  - KOS-V4
14
  - from-scratch
15
+ - gguf
16
+ - llama.cpp
17
+ - quantized
18
  ---
19
 
20
  <div align="center">
 
37
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
38
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
39
 
40
+ **Every number in this card is IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
41
+ — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." We do **not** use the looser
42
+ four-metric average anywhere.
43
+
44
+ **David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs**. OpenAI trained the original
45
+ ChatGPT (**GPT-3.5**) on a ~10,000-GPU supercomputer, and today's open models train on **trillions** of tokens
46
+ (Qwen2.5 ~18T, Llama-3 ~15T). KOS-V4's 180B tokens is **~100× less data**. From that budget:
47
 
48
  | IFEval **strict-avg** | model | who built it, and how |
49
  | --: | :-- | :-- |
50
+ | **61.6** | **KOS-V4-Instruct (ours)** | a small team, from scratch, 180B tokens, 24 GPUs |
 
51
  | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
52
+ | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
53
 
54
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
55
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
56
  which the original ChatGPT lacked at launch — though modern small models score higher there.
57
 
58
+ **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
59
+ BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
60
+ is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
61
+ 64.7, a 0.7-pt gap).
62
+
63
  ## Core specifications
64
  | Attribute | Detail |
65
  | :--- | :--- |
 
124
 
125
  ## Evaluation & benchmarks
126
 
127
+ ### IFEval (official lm-eval-harness, `--apply_chat_template`) — strict metrics
128
+ | IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
129
+ | :-- | --: | --: | --: | --: | --: |
130
+ | prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
131
+ | instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
132
+ | **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
133
+
134
+ *(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
135
+ public Open LLM Leaderboard strict number for a wider set.)*
136
+
137
+ ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
138
+ | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
139
+ | :-- | :-- | :-- | :-- |
140
+ | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
141
 
142
  ## IFEval in context (strict-avg, models our size or larger)
143
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
 
198
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
199
  few points lower), clearly labeled.
200
 
 
 
 
 
 
201
  ## Compute footprint
202
  Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
203
 
 
205
  | Precision | Approx. VRAM | Notes |
206
  | :--- | :--- | :--- |
207
  | **bfloat16** | 7 GB | native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable |
208
+ | **GGUF Q8_0 / Q4_K_M** | ~4 / ~2.5 GB | shipped for `llama.cpp` — see **GGUF quantizations** below |
209
+
210
+ ## GGUF quantizations (llama.cpp)
211
+ Ready-to-run `llama.cpp` builds are published at
212
+ [**Kentucky-Open-Science/KOS-V4-Instruct-GGUF**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct-GGUF)
213
+ (llama.cpp `b510`/18ef86e; lm-eval-harness `0.4.12`).
214
+
215
+ | file | quant | size | notes |
216
+ | :-- | :-- | --: | :-- |
217
+ | `kosv4-f16.gguf` | F16 (16.0 bpw) | 6.03 GB | full-precision reference / requantize source |
218
+ | `kosv4-Q8_0.gguf` | Q8_0 (8.5 bpw) | 3.21 GB | near-lossless |
219
+ | `kosv4-Q4_K_M.gguf` | Q4_K_M (~4.8 bpw) | 1.83 GB | **recommended** — 3.3× smaller than f16, no measurable IFEval loss |
220
+
221
+ SHA256 in `SHA256SUMS`; raw benchmark JSON + conversion/eval scripts in `testing/`.
222
+
223
+ **Quantization preserves IFEval.** Official EleutherAI lm-eval `ifeval` (`--apply_chat_template`), all 541 prompts,
224
+ greedy; only the weights differ across rows. `strict-avg = (prompt-strict + inst-strict)/2`.
225
+
226
+ | format | prompt-strict | inst-strict | **strict-avg** | Δ vs bf16 |
227
+ | :-- | --: | --: | --: | --: |
228
+ | bf16 (HF reference) | 55.82 | 67.03 | **61.42** | — |
229
+ | Q8_0 (GGUF) | 55.27 | 66.43 | **60.85** | −0.57 |
230
+ | Q4_K_M (GGUF) | 56.19 | 67.39 | **61.79** | +0.37 |
231
+
232
+ All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the
233
+ bf16 row reproduces the card's **61.6** headline (measured 61.42).
234
+
235
+ **Serving:**
236
+ ```bash
237
+ llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080
238
+ ```
239
+ `--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
240
+ `<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
241
+
242
+ **Rebuilding (important):** the v4 tokenizer is a standard **ByteLevel / GPT-2-style BPE**. Convert with pre-type
243
+ **`gpt-2`** — this gives full token parity vs the HF tokenizer (`qwen2`/`default`/`llama-bpe`/`tekken` all diverge and
244
+ corrupt the ChatML scaffold). Set `tokenizer.ggml.pre = gpt-2`, `add_bos_token = false`, `rope.freq_base = 25000`.
245
+ Use `testing/convert_gpt2.py`; verify token parity with `testing/pretok_probe.py`.
246
 
247
  ## Intended use & limitations
248
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
249
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
250
  - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
251
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.