codybum commited on
Commit
b4eae0a
·
verified ·
1 Parent(s): 68e3039

Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -233,7 +233,7 @@ harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
233
  | Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 60.5 |
234
  | Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
235
  | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
236
- | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | undisclosed | 58 \* |
237
  | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 70-176B + | 58 \* |
238
  | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
239
  | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
@@ -252,8 +252,8 @@ and confidence are in the baseline block below and in `COMMERCIAL_IFEVAL_BASELIN
252
  measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
253
 
254
  **\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
255
- GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 70B to 176B. Gemini-1.0-Pro is left
256
- as undisclosed because public estimates span 13B to over 500B, too wide to cite.
257
 
258
  **Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**
259
  (1106 = 58.0, 0125 = 52.1, and the estimated 0613 = 61), ties the 8.8B Yi-1.5, and clears the estimated Command-R,
 
233
  | Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 60.5 |
234
  | Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
235
  | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
236
+ | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 58 \* |
237
  | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 70-176B + | 58 \* |
238
  | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
239
  | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
 
252
  measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
253
 
254
  **\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
255
+ GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 70B to 176B, Gemini-1.0-Pro about 30B
256
+ (public estimates span 13B to over 500B; 30B is a low-end estimate).
257
 
258
  **Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**
259
  (1106 = 58.0, 0125 = 52.1, and the estimated 0613 = 61), ties the 8.8B Yi-1.5, and clears the estimated Command-R,