codybum commited on
Commit
9e6b111
·
verified ·
1 Parent(s): f4a4430

IFEval-in-context: add sourced Company + Released (contents) + precise params

Browse files
Files changed (1) hide show
  1. README.md +28 -28
README.md CHANGED
@@ -211,34 +211,34 @@ tool-use data. The same holds on IFEval, where newer, more heavily distilled sma
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (models our size or larger)
214
- Every model is scored on the **same standard metric, IFEval AVG4** (the mean of the four strict + loose sub-metrics,
215
- exactly as reported for KOS-V4 above). Competitor values are computed identically from the **archived Open LLM
216
- Leaderboard** per-model results (`open-llm-leaderboard/results`, 2024–2025) using **exact canonical model IDs**
217
- (not community fine-tunes). Models **smaller than KOS-V4 are excluded** — this is a same-size-or-larger
218
- comparison. Calibration: our own harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0 (within ~0.6).
219
-
220
- | model | params | IFEval AVG4 |
221
- | :-- | :-- | :-- |
222
- | Llama-3-8B-Instruct | 8B | 77.3 |
223
- | Llama-3.2-3B-Instruct | 3.2B | 76.6 |
224
- | Qwen2.5-3B-Instruct | 3.0B | 66.0 |
225
- | Phi-3-medium-4k-instruct | 14B | 65.7 |
226
- | **KOS-V4-Instruct (ours)** | **3.0B** | **62.7** |
227
- | Yi-1.5-9B-Chat | 9B | 62.7 |
228
- | OpenChat-3.5 (1210) | 7B | 62.2 |
229
- | Phi-3.5-mini-instruct | 3.8B | 60.5 |
230
- | Mistral-7B-Instruct-v0.2 | 7B | 57.5 |
231
- | Phi-3-mini-4k-instruct | 3.8B | 57.1 |
232
- | Mistral-7B-Instruct-v0.3 | 7B | 56.6 |
233
- | gemma-1.1-7b-it | 7B | 52.9 |
234
- | Zephyr-7B-beta | 7B | 51.2 |
235
- | Llama-3.1-8B-Instruct | 8B | 51.0 |
236
- | Llama-2-13b-chat | 13B | 46.5 |
237
- | Qwen1.5-7B-Chat | 7B | 46.1 |
238
- | Qwen1.5-4B-Chat | 4.0B | 33.2 |
239
-
240
- **Reading.** At **3B**, KOS-V4 ties the 9B Yi-1.5 and outscores every 7B–13B instruct model below it, including
241
- both Mistral-7B-Instruct releases, **Meta's Llama-3.1-8B (8B)**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and
242
  Qwen1.5-7B. **The newest major model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four
243
  models stay clearly ahead: Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's
244
  leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived value.)
 
211
  results are notable for its data and compute budget, not a claim to lead the current frontier.
212
 
213
  ## IFEval in context (models our size or larger)
214
+ Every model is scored on the **same standard metric, IFEval AVG4** (mean of the four strict + loose sub-metrics from
215
+ the official IFEval implementation, `google-research/instruction_following_eval`). AVG4 is computed identically from
216
+ the archived **Open LLM Leaderboard** per-model results (`open-llm-leaderboard/results`); **company, parameter count,
217
+ and release date (Hugging Face upload date) are taken from `open-llm-leaderboard/contents`**. Models smaller than
218
+ KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
219
+
220
+ | model | company | released | params | IFEval AVG4 |
221
+ | :-- | :-- | :-- | :-- | :-- |
222
+ | Llama-3-8B-Instruct | Meta | Apr 2024 | 8.0B | 77.3 |
223
+ | Llama-3.2-3B-Instruct | Meta | Sep 2024 | 3.2B | 76.6 |
224
+ | Qwen2.5-3B-Instruct | Alibaba | Sep 2024 | 3.0B | 66.0 |
225
+ | Phi-3-medium-4k-instruct | Microsoft | May 2024 | 14.0B | 65.7 |
226
+ | **KOS-V4-Instruct (ours)** | **Univ. of Kentucky** | **2025** | **3.0B** | **62.7** |
227
+ | Yi-1.5-9B-Chat | 01.AI | May 2024 | 8.8B | 62.7 |
228
+ | OpenChat-3.5-1210 | OpenChat | Dec 2023 | 7.2B | 62.2 |
229
+ | Phi-3.5-mini-instruct | Microsoft | Aug 2024 | 3.8B | 60.5 |
230
+ | Mistral-7B-Instruct-v0.2 | Mistral AI | Dec 2023 | 7.2B | 57.5 |
231
+ | Phi-3-mini-4k-instruct | Microsoft | Apr 2024 | 3.8B | 57.1 |
232
+ | Mistral-7B-Instruct-v0.3 | Mistral AI | May 2024 | 7.2B | 56.6 |
233
+ | gemma-1.1-7b-it | Google | Mar 2024 | 8.5B | 52.9 |
234
+ | Zephyr-7B-beta | Hugging Face | Oct 2023 | 7.2B | 51.2 |
235
+ | Llama-3.1-8B-Instruct | Meta | Jul 2024 | 8.0B | 51.0 |
236
+ | Llama-2-13b-chat | Meta | Jul 2023 | 13.0B | 46.5 |
237
+ | Qwen1.5-7B-Chat | Alibaba | Jan 2024 | 7.7B | 46.1 |
238
+ | Qwen1.5-4B-Chat | Alibaba | Jan 2024 | 4.0B | 33.2 |
239
+
240
+ **Reading.** At **3B**, KOS-V4 ties the 8.8B Yi-1.5 and outscores every 7B-14B instruct model below it, including
241
+ both Mistral-7B-Instruct releases, **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and
242
  Qwen1.5-7B. **The newest major model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four
243
  models stay clearly ahead: Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's
244
  leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived value.)