codybum commited on
Commit
2cf9f83
·
verified ·
1 Parent(s): c7be14a

IFEval-in-context: use leaderboard-comparable strict-average (61.4) consistently

Browse files
Files changed (1) hide show
  1. README.md +12 -11
README.md CHANGED
@@ -211,19 +211,21 @@ tool-use data. The same holds on IFEval, where newer, more heavily distilled sma
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (recognizable models)
214
- Where KOS-V4-Instruct's IFEval sits among well-known open models, from the **archived Open LLM Leaderboard**
215
- (2024–2025 snapshot, `open-llm-leaderboard/contents`), using **exact canonical model IDs** (not community fine-tunes).
216
- The leaderboard's IFEval is a single aggregated metric, a near-equivalent to our AVG4: as a calibration it lists
217
- Qwen2.5-3B-Instruct at 64.7 where our own harness measures 65.4 (within ~1 point).
 
 
218
 
219
  **Size-fair view, small instruct models (≤ 4B params).** Among comparable-size models, KOS-V4 trails only two
220
  big-lab distilled 3B models and beats the rest:
221
 
222
- | model | params | IFEval |
223
  | :-- | :-- | :-- |
224
  | Llama-3.2-3B-Instruct | 3.2B | 73.9 |
225
  | Qwen2.5-3B-Instruct | 3.0B | 64.7 |
226
- | **KOS-V4-Instruct (ours)** | **3.0B** | **62.7** |
227
  | Phi-3.5-mini-instruct | 3.8B | 57.7 |
228
  | Llama-3.2-1B-Instruct | 1.2B | 57.0 |
229
  | gemma-2-2b-it | 2.6B | 56.7 |
@@ -232,11 +234,10 @@ big-lab distilled 3B models and beats the rest:
232
  | Qwen2.5-1.5B-Instruct | 1.5B | 44.8 |
233
  | Qwen1.5-4B-Chat | 4.0B | 31.6 |
234
 
235
- **It also matches or beats much larger instruct models.** On the same leaderboard, KOS-V4's 62.7 is at or above
236
- Phi-3-medium-14B (64.2, within noise), Mistral-7B-Instruct v0.2 / v0.3 (55.0 / 54.7), Llama-3.1-8B-Instruct (49.2),
237
- gemma-1.1-7b-it (50.4), Zephyr-7B (49.5), Qwen1.5-7B-Chat (43.7), Llama-2-13b-chat (39.8), and gemma-7b-it (38.7).
238
- Only Llama-3-8B-Instruct (74.1) and Llama-3.2-3B (73.9) stay clearly ahead. (Note: this is the aggregated leaderboard
239
- metric; single-metric strict prompt-level accuracy is lower for every model here, ours included.)
240
 
241
  ## Compute footprint & requirements
242
  ### Training
 
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (recognizable models)
214
+ Where KOS-V4-Instruct sits among well-known open models on the **archived Open LLM Leaderboard** (2024–2025 snapshot,
215
+ `open-llm-leaderboard/contents`), using **exact canonical model IDs** (not community fine-tunes). Both this column
216
+ and our own IFEval are **aggregates**, but of different sub-metrics: the leaderboard's IFEval is the **strict**
217
+ prompt/instruction average, while our headline **62.7 is AVG4** (all four strict + loose metrics). To compare like
218
+ with like, KOS-V4 is listed below at its **strict-average, 61.4** (our AVG4 headline is 62.7). Calibration anchor:
219
+ Qwen2.5-3B shows leaderboard 64.7 vs our strict-average 64.0.
220
 
221
  **Size-fair view, small instruct models (≤ 4B params).** Among comparable-size models, KOS-V4 trails only two
222
  big-lab distilled 3B models and beats the rest:
223
 
224
+ | model | params | IFEval (leaderboard strict) |
225
  | :-- | :-- | :-- |
226
  | Llama-3.2-3B-Instruct | 3.2B | 73.9 |
227
  | Qwen2.5-3B-Instruct | 3.0B | 64.7 |
228
+ | **KOS-V4-Instruct (ours)** | **3.0B** | **61.4** |
229
  | Phi-3.5-mini-instruct | 3.8B | 57.7 |
230
  | Llama-3.2-1B-Instruct | 1.2B | 57.0 |
231
  | gemma-2-2b-it | 2.6B | 56.7 |
 
234
  | Qwen2.5-1.5B-Instruct | 1.5B | 44.8 |
235
  | Qwen1.5-4B-Chat | 4.0B | 31.6 |
236
 
237
+ **It also matches or beats much larger instruct models.** On the same strict metric, KOS-V4 (61.4) is at or above
238
+ Mistral-7B-Instruct v0.2 / v0.3 (55.0 / 54.7), Llama-3.1-8B-Instruct (49.2), gemma-1.1-7b-it (50.4), Zephyr-7B
239
+ (49.5), Qwen1.5-7B-Chat (43.7), Llama-2-13b-chat (39.8), and gemma-7b-it (38.7). Ahead of it sit Phi-3-medium-14B
240
+ (64.2), Qwen2.5-3B (64.7), and the strongest small models, Llama-3-8B-Instruct (74.1) and Llama-3.2-3B (73.9).
 
241
 
242
  ## Compute footprint & requirements
243
  ### Training