codybum commited on
Commit
601d866
·
verified ·
1 Parent(s): b4eae0a

Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -234,7 +234,7 @@ harness measures Qwen2.5-3B at AVG4 65.4 vs the leaderboard's 66.0.
234
  | Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
235
  | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
236
  | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 58 \* |
237
- | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 70-176B + | 58 \* |
238
  | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
239
  | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
240
  | Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 56.6 |
@@ -252,7 +252,7 @@ and confidence are in the baseline block below and in `COMMERCIAL_IFEVAL_BASELIN
252
  measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
253
 
254
  **\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
255
- GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 70B to 176B, Gemini-1.0-Pro about 30B
256
  (public estimates span 13B to over 500B; 30B is a low-end estimate).
257
 
258
  **Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**
 
234
  | Command-R | Open | Cohere | Mar 2024 | 35B | 60 \* |
235
  | GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 58.0 |
236
  | Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 58 \* |
237
+ | Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 58 \* |
238
  | Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 57.5 |
239
  | Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 57.1 |
240
  | Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 56.6 |
 
252
  measured or provider-published four-metric AVG4 (for example GPT-3.5-turbo-1106 = 58.0 and -0125 = 52.1).
253
 
254
  **\+ Parameter count is unofficial** (never disclosed by the provider; industry estimate): GPT-3.5-turbo about 20B,
255
+ GPT-4o-mini about 8B, Mistral-Large (Feb 2024) about 46.7B, Mistral-Medium about 100B, Gemini-1.0-Pro about 30B
256
  (public estimates span 13B to over 500B; 30B is a low-end estimate).
257
 
258
  **Reading.** At **3B**, KOS-V4 (62.7) **clears every measured GPT-3.5-turbo snapshot of the original ChatGPT**