codybum commited on
Commit
25c83d6
·
verified ·
1 Parent(s): 2199bcf

Manual update

Browse files
Files changed (1) hide show
  1. README.md +12 -27
README.md CHANGED
@@ -34,29 +34,19 @@ Kentucky College of Medicine team ([Office for Research](https://medicine.uky.ed
34
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
35
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
36
 
37
- **Every number in this card is IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
38
- — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." We do **not** use the looser
39
- four-metric average anywhere.
40
-
41
- **David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs**. OpenAI trained the original
42
- ChatGPT (**GPT-3.5**) on a ~10,000-GPU supercomputer, and today's open models train on **trillions** of tokens
43
- (Qwen2.5 ~18T, Llama-3 ~15T). KOS-V4's 180B tokens is **~100× less data**. From that budget:
44
 
45
  | IFEval **strict-avg** | model | who built it, and how |
46
  | --: | :-- | :-- |
 
47
  | **61.6** | **KOS-V4-Instruct (ours)** | a small team, from scratch, 180B tokens, 24 GPUs |
48
  | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
49
- | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
50
 
51
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
52
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
53
  which the original ChatGPT lacked at launch — though modern small models score higher there.
54
 
55
- **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
56
- BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
57
- is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
58
- 64.7, a 0.7-pt gap).
59
-
60
  ## Core specifications
61
  | Attribute | Detail |
62
  | :--- | :--- |
@@ -121,20 +111,10 @@ the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from
121
 
122
  ## Evaluation & benchmarks
123
 
124
- ### IFEval (official lm-eval-harness, `--apply_chat_template`) — strict metrics
125
- | IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
126
- | :-- | --: | --: | --: | --: | --: |
127
- | prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
128
- | instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
129
- | **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
130
-
131
- *(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
132
- public Open LLM Leaderboard strict number for a wider set.)*
133
-
134
- ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
135
- | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
136
- | :-- | :-- | :-- | :-- |
137
- | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
138
 
139
  ## IFEval in context (strict-avg, models our size or larger)
140
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
@@ -195,6 +175,11 @@ claim to lead the current field.
195
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
196
  few points lower), clearly labeled.
197
 
 
 
 
 
 
198
  ## Compute footprint
199
  Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
200
 
 
34
  (Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
35
  instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
36
 
37
+ **IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
38
+ — the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."
 
 
 
 
 
39
 
40
  | IFEval **strict-avg** | model | who built it, and how |
41
  | --: | :-- | :-- |
42
+ | 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
43
  | **61.6** | **KOS-V4-Instruct (ours)** | a small team, from scratch, 180B tokens, 24 GPUs |
44
  | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
 
45
 
46
  KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
47
  commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
48
  which the original ChatGPT lacked at launch — though modern small models score higher there.
49
 
 
 
 
 
 
50
  ## Core specifications
51
  | Attribute | Detail |
52
  | :--- | :--- |
 
111
 
112
  ## Evaluation & benchmarks
113
 
114
+ **Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
115
+ BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
116
+ is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
117
+ 64.7, a 0.7-pt gap).
 
 
 
 
 
 
 
 
 
 
118
 
119
  ## IFEval in context (strict-avg, models our size or larger)
120
  One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
 
175
  (arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
176
  few points lower), clearly labeled.
177
 
178
+ ### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
179
+ | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
180
+ | :-- | :-- | :-- | :-- |
181
+ | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
182
+
183
  ## Compute footprint
184
  Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
185