codybum commited on
Commit
7d1480b
·
verified ·
1 Parent(s): 2cf9f83

IFEval context on the standard AVG4 metric for all models (62.7 consistent)

Browse files
Files changed (1) hide show
  1. README.md +21 -22
README.md CHANGED
@@ -211,33 +211,32 @@ tool-use data. The same holds on IFEval, where newer, more heavily distilled sma
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (recognizable models)
214
- Where KOS-V4-Instruct sits among well-known open models on the **archived Open LLM Leaderboard** (2024–2025 snapshot,
215
- `open-llm-leaderboard/contents`), using **exact canonical model IDs** (not community fine-tunes). Both this column
216
- and our own IFEval are **aggregates**, but of different sub-metrics: the leaderboard's IFEval is the **strict**
217
- prompt/instruction average, while our headline **62.7 is AVG4** (all four strict + loose metrics). To compare like
218
- with like, KOS-V4 is listed below at its **strict-average, 61.4** (our AVG4 headline is 62.7). Calibration anchor:
219
- Qwen2.5-3B shows leaderboard 64.7 vs our strict-average 64.0.
220
 
221
  **Size-fair view, small instruct models (≤ 4B params).** Among comparable-size models, KOS-V4 trails only two
222
  big-lab distilled 3B models and beats the rest:
223
 
224
- | model | params | IFEval (leaderboard strict) |
225
  | :-- | :-- | :-- |
226
- | Llama-3.2-3B-Instruct | 3.2B | 73.9 |
227
- | Qwen2.5-3B-Instruct | 3.0B | 64.7 |
228
- | **KOS-V4-Instruct (ours)** | **3.0B** | **61.4** |
229
- | Phi-3.5-mini-instruct | 3.8B | 57.7 |
230
- | Llama-3.2-1B-Instruct | 1.2B | 57.0 |
231
- | gemma-2-2b-it | 2.6B | 56.7 |
232
- | Phi-3-mini-4k-instruct | 3.8B | 56.1 |
233
- | SmolLM2-1.7B-Instruct | 1.7B | 53.7 |
234
- | Qwen2.5-1.5B-Instruct | 1.5B | 44.8 |
235
- | Qwen1.5-4B-Chat | 4.0B | 31.6 |
236
-
237
- **It also matches or beats much larger instruct models.** On the same strict metric, KOS-V4 (61.4) is at or above
238
- Mistral-7B-Instruct v0.2 / v0.3 (55.0 / 54.7), Llama-3.1-8B-Instruct (49.2), gemma-1.1-7b-it (50.4), Zephyr-7B
239
- (49.5), Qwen1.5-7B-Chat (43.7), Llama-2-13b-chat (39.8), and gemma-7b-it (38.7). Ahead of it sit Phi-3-medium-14B
240
- (64.2), Qwen2.5-3B (64.7), and the strongest small models, Llama-3-8B-Instruct (74.1) and Llama-3.2-3B (73.9).
241
 
242
  ## Compute footprint & requirements
243
  ### Training
 
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (recognizable models)
214
+ Every model here is scored on the **same standard metric, IFEval AVG4** (the mean of the four strict + loose
215
+ sub-metrics, exactly as reported for KOS-V4 above). Competitor AVG4 values are computed from the **archived Open LLM
216
+ Leaderboard** per-model results (`open-llm-leaderboard/results`, 2024–2025) using **exact canonical model IDs** (not
217
+ community fine-tunes). Calibration: our own harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0
218
+ (within ~0.6), so KOS-V4's 62.7 is directly comparable.
 
219
 
220
  **Size-fair view, small instruct models (≤ 4B params).** Among comparable-size models, KOS-V4 trails only two
221
  big-lab distilled 3B models and beats the rest:
222
 
223
+ | model | params | IFEval AVG4 |
224
  | :-- | :-- | :-- |
225
+ | Llama-3.2-3B-Instruct | 3.2B | 76.6 |
226
+ | Qwen2.5-3B-Instruct | 3.0B | 66.0 |
227
+ | **KOS-V4-Instruct (ours)** | **3.0B** | **62.7** |
228
+ | Phi-3.5-mini-instruct | 3.8B | 60.5 |
229
+ | Llama-3.2-1B-Instruct | 1.2B | 58.9 |
230
+ | gemma-2-2b-it | 2.6B | 57.8 |
231
+ | Phi-3-mini-4k-instruct | 3.8B | 57.1 |
232
+ | SmolLM2-1.7B-Instruct | 1.7B | 55.0 |
233
+ | Qwen2.5-1.5B-Instruct | 1.5B | 46.8 |
234
+ | Qwen1.5-4B-Chat | 4.0B | 33.2 |
235
+
236
+ **It also matches or beats much larger instruct models** (same AVG4 metric): Yi-1.5-9B-Chat (62.7, a tie),
237
+ OpenChat-3.5 (62.2), Mistral-7B-Instruct v0.2 / v0.3 (57.5 / 56.6), gemma-1.1-7b-it (52.9), Zephyr-7B (51.2),
238
+ Llama-3.1-8B-Instruct (51.0), Llama-2-13b-chat (46.5), and Qwen1.5-7B-Chat (46.1). Ahead of it sit Phi-3-medium-14B
239
+ (65.7) and the strongest small models, Llama-3-8B-Instruct (77.3) and Llama-3.2-3B (76.6).
240
 
241
  ## Compute footprint & requirements
242
  ### Training