codybum commited on
Commit
2d5aa65
·
verified ·
1 Parent(s): bddd03f

Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

Browse files
Files changed (1) hide show
  1. README.md +49 -16
README.md CHANGED
@@ -246,26 +246,59 @@ beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four mode
246
  Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
247
  its scores on some later evaluations; it is shown here as the archived value.)
248
 
249
- **Commercial reference baselines (reported).** KOS-V4's 62.7 sits squarely in the original commercial-API tier.
250
- These are provider/endpoint figures (commercial models are not on the Open LLM Leaderboard), reported as published by
251
- each provider, so treat them as approximate, commercial IFEval is often reported as strict or 2-metric rather than
252
- the 4-metric AVG4 used in the open-model table above.
 
253
 
254
- | commercial model | company | endpoint | reported IFEval |
255
- | :-- | :-- | :-- | :-- |
256
- | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | ~60.7 (AVG4) / ~63 (strict-prompt) |
257
- | Cohere Command R | Cohere | command-r (early 2024) | ~62.0 |
258
- | Gemini 1.0 Pro | Google | gemini-1.0-pro-001 | ~58-61 |
259
 
260
- **Takeaway.** From a fraction of the data and 24 GPUs, KOS-V4 (62.7) runs **neck-and-neck with the original
261
- commercial API generation**, GPT-3.5-turbo, Cohere Command R, and Gemini 1.0 Pro.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
262
 
263
  **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
264
- Open-model AVG4: computed from `open-llm-leaderboard/results`. Company, parameter count, and release date:
265
- `open-llm-leaderboard/contents`. GPT-3.5-turbo: AVG4 from OpenAI's reported IFEval sub-metrics
266
- (prompt/inst strict + loose = 53.8 / 64.7 / 56.6 / 67.5). Commercial reference baselines (GPT-3.5-turbo, Cohere
267
- Command R, Gemini 1.0 Pro): figures reported by the respective providers for the named endpoints; these are not
268
- Open-LLM-Leaderboard entries and their metric basis is provider-defined, hence marked approximate.
 
269
 
270
  ## Compute footprint & requirements
271
  ### Training
 
246
  Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
247
  its scores on some later evaluations; it is shown here as the archived value.)
248
 
249
+ **Commercial baselines (IFEval AVG4, same metric as the table above).** KOS-V4's 62.7 sits at the top of the original
250
+ commercial-API tier and below the 2024 frontier. Rows are grouped by evidence quality. **Verified** = AVG4 computed
251
+ here from four published sub-metrics. **Reported** = a four-metric AVG4 published by a primary source (sub-metrics not
252
+ broken out). **Estimate (EST)** = no IFEval AVG4 is published; estimated from adjacent evidence and clearly labeled,
253
+ not to be cited as measured.
254
 
255
+ Verified (four sub-metrics, AVG4 computed here):
 
 
 
 
256
 
257
+ | commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
258
+ | :-- | :-- | :-- | :-- | :-- |
259
+ | GPT-4 | OpenAI | gpt-4-0613 | 82.1 | InternLM2 report (arXiv:2403.17297) |
260
+ | Command-R+ | Cohere | c4ai-command-r-plus | 77.9 | Open LLM Leaderboard raw results |
261
+ | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 58.0 | InternLM2 report (arXiv:2403.17297) |
262
+
263
+ Reported (four-metric AVG4 published, sub-metrics not broken out):
264
+
265
+ | commercial model | company | endpoint / snapshot | IFEval AVG4 | source |
266
+ | :-- | :-- | :-- | :-- | :-- |
267
+ | Claude-3.5-Sonnet | Anthropic | claude-3-5-sonnet-20240620 | 86.2 | Llama 3.1 report (arXiv:2407.21783) |
268
+ | GPT-4o | OpenAI | gpt-4o-2024-05-13 | 84.3 | Llama 3.1 report |
269
+ | Gemini-1.5-Pro | Google | May 2024 | 82.3 | Llama 3.1 report |
270
+ | GPT-4-turbo | OpenAI | gpt-4-1106-preview | 81.4 | Llama 3.1 report |
271
+ | Gemini-1.5-Flash | Google | May 2024 | 77.2 | Llama 3.1 report |
272
+ | Claude-3-Sonnet | Anthropic | Mar 2024 | 72.3 | Llama 3.1 report |
273
+ | Claude-3-Haiku | Anthropic | Mar 2024 | 68.1 | Llama 3.1 report |
274
+ | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | 52.1 | Llama 3.1 report |
275
+
276
+ Estimate (no published IFEval AVG4; EST values, not measured):
277
+
278
+ | commercial model | company | endpoint / snapshot | IFEval AVG4 | basis |
279
+ | :-- | :-- | :-- | :-- | :-- |
280
+ | GPT-4o-mini | OpenAI | gpt-4o-mini-2024-07-18 | 82 EST (79-84) | class anchor to GPT-4o |
281
+ | Mistral-Large | Mistral AI | mistral-large-2402 | 66 EST (60-72) | class anchor |
282
+ | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0613 | 61 EST (60-63) | uplift from prompt-strict 54.4 (Conifer) |
283
+ | Gemini-1.0-Pro | Google | gemini-1.0-pro-001 | 58 EST (55-61) | class anchor (report uses InFoBench, not IFEval) |
284
+ | Command-R | Cohere | command-r (Mar 2024) | 60 EST (54-68) | anchor to Command-R+ |
285
+ | Mistral-Medium | Mistral AI | mistral-medium-2312 | 58 EST (54-63) | class anchor |
286
+
287
+ The GPT-3.5-turbo family spans AVG4 52.1 (0125) to 58.0 (1106) to 61 est. (0613) across snapshots. KOS-V4's 62.7 sits
288
+ at or above the top of that band on every snapshot with data.
289
+
290
+ **Takeaway.** KOS-V4 (62.7) clears the entire original ChatGPT / GPT-3.5-turbo generation and the estimated
291
+ Gemini-1.0-Pro and Command-R tier. The 2024 frontier (GPT-4, GPT-4o, Claude-3.5-Sonnet, Gemini-1.5, Command-R+) stays
292
+ well ahead. A from-scratch 3B on 180B tokens and 24 GPUs, at the instruction-following level of the first commercial
293
+ ChatGPT generation.
294
 
295
  **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
296
+ Open-model AVG4: computed from `open-llm-leaderboard/results`, with company, parameters and release date from
297
+ `open-llm-leaderboard/contents`. Commercial verified sub-metrics: InternLM2 technical report (arXiv:2403.17297) for
298
+ GPT-4-0613 and GPT-3.5-turbo-1106; Open LLM Leaderboard raw results for Command-R+. Commercial reported AVG4: Llama 3.1
299
+ technical report (arXiv:2407.21783), Table 2. GPT-3.5-turbo-0613 prompt-strict: Conifer (arXiv:2404.02823). Estimate
300
+ rows are derived, not measured, and are labeled EST. Full dataset with methods and confidence:
301
+ `COMMERCIAL_IFEVAL_BASELINES.md`.
302
 
303
  ## Compute footprint & requirements
304
  ### Training