codybum commited on
Commit
6571935
·
verified ·
1 Parent(s): c6ea0eb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -39
README.md CHANGED
@@ -1,5 +1,5 @@
1
- ---
2
- license: other
3
  library_name: transformers
4
  pipeline_tag: text-generation
5
  language:
@@ -12,9 +12,6 @@ tags:
12
  - function-calling
13
  - KOS-V4
14
  - from-scratch
15
- - gguf
16
- - llama.cpp
17
- - quantized
18
  ---
19
 
20
  <div align="center">
@@ -37,28 +34,19 @@ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.ed
37
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
38
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
39
 
40
- **Every number in this card is IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
41
- — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." We do **not** use the looser
42
- four-metric average anywhere.
43
-
44
- KOS-V4 was trained on a fraction of the resources OpenAI trained the original
45
- ChatGPT (GPT-3.5).
46
 
47
  | IFEval **strict-avg** | model | who built it, and how |
48
  | --: | :-- | :-- |
49
- | **61.6** | **KOS-V4-Instruct (ours)** | Kentucky Research Team, 180B tokens, 24 GPUs |
50
- | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
51
  | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
 
 
52
 
53
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
54
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
55
  which the original ChatGPT lacked at launch — though modern small models score higher there.
56
 
57
- **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
58
- BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
59
- is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
60
- 64.7, a 0.7-pt gap).
61
-
62
  ## Core specifications
63
  | Attribute | Detail |
64
  | :--- | :--- |
@@ -123,20 +111,11 @@ the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from
123
 
124
  ## Evaluation & benchmarks
125
 
126
- ### IFEval (official lm-eval-harness, `--apply_chat_template`) — strict metrics
127
- | IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
128
- | :-- | --: | --: | --: | --: | --: |
129
- | prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
130
- | instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
131
- | **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
132
-
133
- *(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
134
- public Open LLM Leaderboard strict number for a wider set.)*
135
-
136
- ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
137
- | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
138
- | :-- | :-- | :-- | :-- |
139
- | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
140
 
141
  ## IFEval in context (strict-avg, models our size or larger)
142
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
@@ -197,8 +176,11 @@ claim to lead the current field.
197
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
198
  few points lower), clearly labeled.
199
 
200
- ## Compute footprint
201
- Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
 
 
 
202
 
203
  ## Deployment (inference)
204
  | Precision | Approx. VRAM | Notes |
@@ -238,13 +220,9 @@ llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 -
238
  `--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
239
  `<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
240
 
241
- **Rebuilding (important):** the v4 tokenizer is a standard **ByteLevel / GPT-2-style BPE**. Convert with pre-type
242
- **`gpt-2`** — this gives full token parity vs the HF tokenizer (`qwen2`/`default`/`llama-bpe`/`tekken` all diverge and
243
- corrupt the ChatML scaffold). Set `tokenizer.ggml.pre = gpt-2`, `add_bos_token = false`, `rope.freq_base = 25000`.
244
- Use `testing/convert_gpt2.py`; verify token parity with `testing/pretok_probe.py`.
245
-
246
  ## Intended use & limitations
247
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
248
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
249
  - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
250
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
 
 
1
+ --
2
+ license: cc-by-nc-sa-4.0
3
  library_name: transformers
4
  pipeline_tag: text-generation
5
  language:
 
12
  - function-calling
13
  - KOS-V4
14
  - from-scratch
 
 
 
15
  ---
16
 
17
  <div align="center">
 
34
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
35
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
36
 
37
+ **IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
38
+ — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
 
 
 
 
39
 
40
  | IFEval **strict-avg** | model | who built it, and how |
41
  | --: | :-- | :-- |
 
 
42
  | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
43
+ | **61.6** | **KOS-V4-Instruct (ours)** | University research team, 180B tokens, 24 GPUs |
44
+ | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
45
 
46
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
47
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
48
  which the original ChatGPT lacked at launch — though modern small models score higher there.
49
 
 
 
 
 
 
50
  ## Core specifications
51
  | Attribute | Detail |
52
  | :--- | :--- |
 
111
 
112
  ## Evaluation & benchmarks
113
 
114
+ **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True
115
+ - greedy, task-default max_gen_toks=1280`),
116
+ BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
117
+ is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
118
+ 64.7, a 0.7-pt gap).
 
 
 
 
 
 
 
 
 
119
 
120
  ## IFEval in context (strict-avg, models our size or larger)
121
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
 
176
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
177
  few points lower), clearly labeled.
178
 
179
+ ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
180
+ | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
181
+ | :-- | :-- | :-- | :-- |
182
+ | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
183
+
184
 
185
  ## Deployment (inference)
186
  | Precision | Approx. VRAM | Notes |
 
220
  `--jinja` is **required** for the model's `<tool_call>` output to be parsed into structured `tool_calls`; eos is
221
  `<|im_end|>` (id 0) and no BOS is prepended; keep `-c ≤ 6144` for in-distribution instruct behavior.
222
 
 
 
 
 
 
223
  ## Intended use & limitations
224
  - **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
225
  - **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
226
  - **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
227
  - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.
228
+ - **No safety or bias evaluation.** This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.