codybum commited on
Commit
b765fc1
·
verified ·
1 Parent(s): 265c752

Rebrand to KOS-V4; David-vs-Goliath GPT-3.5 framing; complete peer tables

Browse files
Files changed (1) hide show
  1. README.md +22 -13
README.md CHANGED
@@ -65,7 +65,8 @@ tool / function calling** over clinical and general English text. Its instructio
65
  reinforcement learning** on a from-scratch clinical base, reaching **IFEval AVG4 = 62.7**. That is on par with the
66
  reported score for the original ChatGPT (**GPT-3.5-turbo, 60.7**) and within about three points of
67
  **Qwen2.5-3B-Instruct (65.4)**, from a fraction of the data and compute. It also adds reliable **tool and function
68
- calling** (BFCL 81/73/62), a capability the original ChatGPT did not have at launch. It is the flagship of the
 
69
  **KOS-V4** family (a retriever and two grounded readers ship alongside it).
70
 
71
  ## Core specifications
@@ -169,23 +170,31 @@ AVG4 rose across GRPO steps from **47.7 (SFT base)** to **62.7 at step 1120 (shi
169
 
170
  ## Evaluation & benchmarks
171
  Public benchmarks only. **IFEval** is the EleutherAI lm-evaluation-harness with `--apply_chat_template`, AVG4 of the
172
- four strict and loose metrics. **BFCL** is the Berkeley Function-Calling AST checker. We measured KOS-V4 and
173
- Qwen2.5-3B ourselves on this harness. The GPT-3.5-turbo column is OpenAI's model **as reported** in published
174
- instruction-following evaluations, included as the original-ChatGPT reference point rather than run by us.
 
175
 
176
- | Benchmark | KOS-V4-Instruct (ours) | GPT-3.5-turbo (original ChatGPT, reported) | Qwen2.5-3B-Instruct |
177
  | :--- | :--- | :--- | :--- |
178
- | **IFEval** AVG4 | **62.7** | 60.7 | 65.4 |
179
- | IFEval prompt-strict / inst-strict / prompt-loose / inst-loose | 55.8 / 67.0 / 58.4 / 69.7 | 53.8 / 64.7 / 56.6 / 67.5 | 59.5 / 68.5 / 62.1 / 71.3 |
180
- | **BFCL** simple / multiple / parallel | **81 / 73 / 62** | none at launch | 95 / 89 / 75 |
181
- | *(reference)* KOS-V4 SFT base, pre-RL, IFEval AVG4 | 47.7 | | |
 
 
 
 
 
182
 
183
  **Reading.** From a small team's from-scratch 3B, IFEval **62.7** lands right on the original ChatGPT
184
  (GPT-3.5-turbo, a reported 60.7) and within about three points of Qwen2.5-3B-Instruct (65.4), while being
185
- **15.0 points above our own SFT base** (47.7). On tool-calling, KOS-V4 is functional (81/73/62), a capability the
186
- original ChatGPT lacked at launch; general instruct models trained on far more tool data score higher (Qwen2.5-3B
187
- at 95/89/75). Newer, more heavily distilled small models trained on 50 to 200 times more data also score higher on
188
- IFEval; KOS-V4's result is notable for its data and compute budget, not a claim to lead the current frontier.
 
 
189
 
190
  ## Compute footprint & requirements
191
  ### Training
 
65
  reinforcement learning** on a from-scratch clinical base, reaching **IFEval AVG4 = 62.7**. That is on par with the
66
  reported score for the original ChatGPT (**GPT-3.5-turbo, 60.7**) and within about three points of
67
  **Qwen2.5-3B-Instruct (65.4)**, from a fraction of the data and compute. It also adds reliable **tool and function
68
+ calling** (official BFCL 73/73/61), a capability the original ChatGPT lacked at launch, though modern small models
69
+ score higher here. It is the flagship of the
70
  **KOS-V4** family (a retriever and two grounded readers ship alongside it).
71
 
72
  ## Core specifications
 
170
 
171
  ## Evaluation & benchmarks
172
  Public benchmarks only. **IFEval** is the EleutherAI lm-evaluation-harness with `--apply_chat_template`, AVG4 of the
173
+ four strict and loose metrics. **BFCL** is the official Berkeley Function-Calling Leaderboard package (`bfcl_eval`),
174
+ run with its **official per-model native handlers** in function-calling mode: each model is prompted in its own tool
175
+ format and scored by the official AST checker. The GPT-3.5-turbo IFEval column is OpenAI's model **as reported** in
176
+ published instruction-following evaluations, included as the original-ChatGPT reference point rather than run by us.
177
 
178
+ | IFEval | KOS-V4-Instruct (ours) | GPT-3.5-turbo (original ChatGPT, reported) | Qwen2.5-3B-Instruct |
179
  | :--- | :--- | :--- | :--- |
180
+ | **AVG4** | **62.7** | 60.7 | 65.4 |
181
+ | prompt-strict / inst-strict / prompt-loose / inst-loose | 55.8 / 67.0 / 58.4 / 69.7 | 53.8 / 64.7 / 56.6 / 67.5 | 59.5 / 68.5 / 62.1 / 71.3 |
182
+ | *(reference)* KOS-V4 SFT base, pre-RL AVG4 | 47.7 | | |
183
+
184
+ Tool-calling, **official BFCL** (function-calling mode; simple / multiple / parallel accuracy, higher is better):
185
+
186
+ | BFCL (official FC) | KOS-V4-Instruct (ours) | Qwen2.5-3B-Instruct | Llama-3.2-3B-Instruct |
187
+ | :--- | :--- | :--- | :--- |
188
+ | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
189
 
190
  **Reading.** From a small team's from-scratch 3B, IFEval **62.7** lands right on the original ChatGPT
191
  (GPT-3.5-turbo, a reported 60.7) and within about three points of Qwen2.5-3B-Instruct (65.4), while being
192
+ **15.0 points above our own SFT base** (47.7). On tool-calling, KOS-V4 works (official BFCL 72.75 / 73.00 / 60.50)
193
+ but clearly **trails the modern small instruct models**: both Qwen2.5-3B and Llama-3.2-3B score around 90 or above on
194
+ simple and multiple, and far higher on parallel. Function-calling is a real capability here, one the original ChatGPT
195
+ did not have at launch, but it is **not a strength relative to today's small models**, which were trained on far more
196
+ tool-use data. The same holds on IFEval, where newer, more heavily distilled small models score higher; KOS-V4's
197
+ results are notable for its data and compute budget, not a claim to lead the current frontier.
198
 
199
  ## Compute footprint & requirements
200
  ### Training