codybum commited on
Commit
4d4d91b
·
verified ·
1 Parent(s): dec0c7d

Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

Browse files
Files changed (1) hide show
  1. README.md +18 -11
README.md CHANGED
@@ -215,20 +215,25 @@ Every model is scored on the **same standard metric, IFEval AVG4** (mean of the
215
  the official IFEval implementation, `google-research/instruction_following_eval`). For open models, AVG4 is computed
216
  identically from the archived **Open LLM Leaderboard** results (`open-llm-leaderboard/results`), and **company,
217
  parameters, and release date are taken from `open-llm-leaderboard/contents`**. GPT-3.5-turbo is a **commercial
218
- reference** (not on the leaderboard); its AVG4 is computed from OpenAI's reported IFEval sub-metrics. Models smaller
219
- than KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
 
220
 
221
  | model | company | released | params | IFEval AVG4 |
222
  | :-- | :-- | :-- | :-- | :-- |
223
- | Llama-3-8B-Instruct | Meta | Apr 2024 | 8.0B | 77.3 |
224
  | Llama-3.2-3B-Instruct | Meta | Sep 2024 | 3.2B | 76.6 |
225
  | Qwen2.5-3B-Instruct | Alibaba | Sep 2024 | 3.0B | 66.0 |
 
226
  | Phi-3-medium-4k-instruct | Microsoft | May 2024 | 14.0B | 65.7 |
227
  | **KOS-V4-Instruct (ours)** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **62.7** |
228
  | Yi-1.5-9B-Chat | 01.AI | May 2024 | 8.8B | 62.7 |
229
- | OpenChat-3.5-1210 | OpenChat | Dec 2023 | 7.2B | 62.2 |
230
- | *GPT-3.5-turbo* (commercial ref) | OpenAI | 2023 | 20B (est.) | 60.7 |
231
  | Phi-3.5-mini-instruct | Microsoft | Aug 2024 | 3.8B | 60.5 |
 
 
 
232
  | Mistral-7B-Instruct-v0.2 | Mistral AI | Dec 2023 | 7.2B | 57.5 |
233
  | Phi-3-mini-4k-instruct | Microsoft | Apr 2024 | 3.8B | 57.1 |
234
  | Mistral-7B-Instruct-v0.3 | Mistral AI | May 2024 | 7.2B | 56.6 |
@@ -243,12 +248,14 @@ than KOS-V4 are excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 6
243
  disclosed the parameter count of GPT-3.5-turbo; the roughly 20B consensus comes from leaked research documents and
244
  industry estimates.
245
 
246
- **Reading.** At **3B**, KOS-V4 (62.7) **edges the original commercial ChatGPT (GPT-3.5-turbo, 60.7)**, ties the 8.8B
247
- Yi-1.5, and outscores every 7B-14B open instruct model below it, including both Mistral-7B-Instruct releases,
248
- **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and Qwen1.5-7B. **The newest major model it
249
- beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four models stay clearly ahead:
250
- Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
251
- its scores on some later evaluations; it is shown here as the archived value.)
 
 
252
 
253
  **Commercial baselines (IFEval AVG4, same metric as the table above).** KOS-V4's 62.7 sits at the top of the original
254
  commercial-API tier and below the 2024 frontier. Rows are grouped by evidence quality. **Verified** = AVG4 computed
 
215
  the official IFEval implementation, `google-research/instruction_following_eval`). For open models, AVG4 is computed
216
  identically from the archived **Open LLM Leaderboard** results (`open-llm-leaderboard/results`), and **company,
217
  parameters, and release date are taken from `open-llm-leaderboard/contents`**. GPT-3.5-turbo is a **commercial
218
+ reference** and the rows tagged **(commercial, est.)** are estimates from the tiered commercial baseline block below,
219
+ clearly labeled est.; **GPT-3.5-turbo (commercial ref)** uses its reported AVG4. Models smaller than KOS-V4 are
220
+ excluded. Calibration: our harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
221
 
222
  | model | company | released | params | IFEval AVG4 |
223
  | :-- | :-- | :-- | :-- | :-- |
224
+ | *GPT-4o-mini* (commercial, est.) | OpenAI | Jul 2024 | undisclosed | 82 (est.) |
225
  | Llama-3.2-3B-Instruct | Meta | Sep 2024 | 3.2B | 76.6 |
226
  | Qwen2.5-3B-Instruct | Alibaba | Sep 2024 | 3.0B | 66.0 |
227
+ | *Mistral-Large* (commercial, est.) | Mistral AI | Feb 2024 | undisclosed | 66 (est.) |
228
  | Phi-3-medium-4k-instruct | Microsoft | May 2024 | 14.0B | 65.7 |
229
  | **KOS-V4-Instruct (ours)** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **62.7** |
230
  | Yi-1.5-9B-Chat | 01.AI | May 2024 | 8.8B | 62.7 |
231
+ | *GPT-3.5-turbo-0613* (commercial, est.) | OpenAI | Jun 2023 | 20B (est.) | 61 (est.) |
232
+ | *GPT-3.5-turbo* (commercial ref) | OpenAI | Mar 2023 | 20B (est.) | 60.7 |
233
  | Phi-3.5-mini-instruct | Microsoft | Aug 2024 | 3.8B | 60.5 |
234
+ | *Command-R* (commercial, est.) | Cohere | Mar 2024 | 35B | 60 (est.) |
235
+ | *Gemini-1.0-Pro* (commercial, est.) | Google | Dec 2023 | undisclosed | 58 (est.) |
236
+ | *Mistral-Medium* (commercial, est.) | Mistral AI | Dec 2023 | undisclosed | 58 (est.) |
237
  | Mistral-7B-Instruct-v0.2 | Mistral AI | Dec 2023 | 7.2B | 57.5 |
238
  | Phi-3-mini-4k-instruct | Microsoft | Apr 2024 | 3.8B | 57.1 |
239
  | Mistral-7B-Instruct-v0.3 | Mistral AI | May 2024 | 7.2B | 56.6 |
 
248
  disclosed the parameter count of GPT-3.5-turbo; the roughly 20B consensus comes from leaked research documents and
249
  industry estimates.
250
 
251
+ **Reading.** At **3B**, KOS-V4 (62.7) **edges the original commercial ChatGPT (GPT-3.5-turbo, 60.7)** and its estimated
252
+ June-2023 snapshot (61), ties the 8.8B Yi-1.5, and clears the estimated commercial tier of Command-R, Gemini-1.0-Pro
253
+ and Mistral-Medium. It outscores every 7B-14B open instruct model below it, including both Mistral-7B-Instruct
254
+ releases, **Meta's Llama-3.1-8B**, Llama-2-13b-chat, gemma-1.1-7b-it, Zephyr-7B and Qwen1.5-7B. **The newest major
255
+ model it beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Ahead of it sit three open models
256
+ (Llama-3.2-3B, Qwen2.5-3B, Phi-3-medium-14B) and two commercial estimates (GPT-4o-mini, Mistral-Large).
257
+ (Llama-3.1-8B's leaderboard IFEval is lower than its scores on some later evaluations; it is shown here as the archived
258
+ value.)
259
 
260
  **Commercial baselines (IFEval AVG4, same metric as the table above).** KOS-V4's 62.7 sits at the top of the original
261
  commercial-API tier and below the 2024 frontier. Rows are grouped by evidence quality. **Verified** = AVG4 computed