codybum commited on
Commit
9464d72
·
verified ·
1 Parent(s): 9e6b111

IFEval-in-context: release dates + GPT-3.5-turbo commercial ref + Sources block

Browse files
Files changed (1) hide show
  1. README.md +17 -9
README.md CHANGED
@@ -212,10 +212,11 @@ results are notable for its data and compute budget, not a claim to lead the cur
212
 
213
  ## IFEval in context (models our size or larger)
214
  Every model is scored on the **same standard metric, IFEval AVG4** (mean of the four strict + loose sub-metrics from
215
- the official IFEval implementation, `google-research/instruction_following_eval`). AVG4 is computed identically from
216
- the archived **Open LLM Leaderboard** per-model results (`open-llm-leaderboard/results`); **company, parameter count,
217
- and release date (Hugging Face upload date) are taken from `open-llm-leaderboard/contents`**. Models smaller than
218
- KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
 
219
 
220
  | model | company | released | params | IFEval AVG4 |
221
  | :-- | :-- | :-- | :-- | :-- |
@@ -226,6 +227,7 @@ KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 v
226
  | **KOS-V4-Instruct (ours)** | **Univ. of Kentucky** | **2025** | **3.0B** | **62.7** |
227
  | Yi-1.5-9B-Chat | 01.AI | May 2024 | 8.8B | 62.7 |
228
  | OpenChat-3.5-1210 | OpenChat | Dec 2023 | 7.2B | 62.2 |
 
229
  | Phi-3.5-mini-instruct | Microsoft | Aug 2024 | 3.8B | 60.5 |
230
  | Mistral-7B-Instruct-v0.2 | Mistral AI | Dec 2023 | 7.2B | 57.5 |
231
  | Phi-3-mini-4k-instruct | Microsoft | Apr 2024 | 3.8B | 57.1 |
@@ -237,11 +239,17 @@ KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 v
237
  | Qwen1.5-7B-Chat | Alibaba | Jan 2024 | 7.7B | 46.1 |
238
  | Qwen1.5-4B-Chat | Alibaba | Jan 2024 | 4.0B | 33.2 |
239
 
240
- **Reading.** At **3B**, KOS-V4 ties the 8.8B Yi-1.5 and outscores every 7B-14B instruct model below it, including
241
- both Mistral-7B-Instruct releases, **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and
242
- Qwen1.5-7B. **The newest major model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four
243
- models stay clearly ahead: Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's
244
- leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived value.)
 
 
 
 
 
 
245
 
246
  ## Compute footprint & requirements
247
  ### Training
 
212
 
213
  ## IFEval in context (models our size or larger)
214
  Every model is scored on the **same standard metric, IFEval AVG4** (mean of the four strict + loose sub-metrics from
215
+ the official IFEval implementation, `google-research/instruction_following_eval`). For open models, AVG4 is computed
216
+ identically from the archived **Open LLM Leaderboard** results (`open-llm-leaderboard/results`), and **company,
217
+ parameters, and release date are taken from `open-llm-leaderboard/contents`**. GPT-3.5-turbo is a **commercial
218
+ reference** (not on the leaderboard); its AVG4 is computed from OpenAI's reported IFEval sub-metrics. Models smaller
219
+ than KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
220
 
221
  | model | company | released | params | IFEval AVG4 |
222
  | :-- | :-- | :-- | :-- | :-- |
 
227
  | **KOS-V4-Instruct (ours)** | **Univ. of Kentucky** | **2025** | **3.0B** | **62.7** |
228
  | Yi-1.5-9B-Chat | 01.AI | May 2024 | 8.8B | 62.7 |
229
  | OpenChat-3.5-1210 | OpenChat | Dec 2023 | 7.2B | 62.2 |
230
+ | *GPT-3.5-turbo* (commercial ref) | OpenAI | 2023 | undisclosed | 60.7 |
231
  | Phi-3.5-mini-instruct | Microsoft | Aug 2024 | 3.8B | 60.5 |
232
  | Mistral-7B-Instruct-v0.2 | Mistral AI | Dec 2023 | 7.2B | 57.5 |
233
  | Phi-3-mini-4k-instruct | Microsoft | Apr 2024 | 3.8B | 57.1 |
 
239
  | Qwen1.5-7B-Chat | Alibaba | Jan 2024 | 7.7B | 46.1 |
240
  | Qwen1.5-4B-Chat | Alibaba | Jan 2024 | 4.0B | 33.2 |
241
 
242
+ **Reading.** At **3B**, KOS-V4 (62.7) **edges the original commercial ChatGPT (GPT-3.5-turbo, 60.7)**, ties the 8.8B
243
+ Yi-1.5, and outscores every 7B-14B open instruct model below it, including both Mistral-7B-Instruct releases,
244
+ **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and Qwen1.5-7B. **The newest major model it
245
+ beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four models stay clearly ahead:
246
+ Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
247
+ its scores on some later evaluations; it is shown here as the archived value.)
248
+
249
+ **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
250
+ Open-model AVG4: computed from `open-llm-leaderboard/results`. Company, parameter count, and release date:
251
+ `open-llm-leaderboard/contents`. GPT-3.5-turbo: AVG4 from OpenAI's reported IFEval sub-metrics
252
+ (prompt/inst strict + loose = 53.8 / 64.7 / 56.6 / 67.5).
253
 
254
  ## Compute footprint & requirements
255
  ### Training