codybum commited on
Commit
bddd03f
·
verified ·
1 Parent(s): 9464d72

Add commercial reference baselines (GPT-3.5-turbo, Command R, Gemini 1.0 Pro), sourced/attributed

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -246,10 +246,26 @@ beats is Microsoft's Phi-3.5-mini-instruct (August 2024, 60.5).** Only four mode
246
  Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
247
  its scores on some later evaluations; it is shown here as the archived value.)
248
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
249
  **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
250
  Open-model AVG4: computed from `open-llm-leaderboard/results`. Company, parameter count, and release date:
251
  `open-llm-leaderboard/contents`. GPT-3.5-turbo: AVG4 from OpenAI's reported IFEval sub-metrics
252
- (prompt/inst strict + loose = 53.8 / 64.7 / 56.6 / 67.5).
 
 
253
 
254
  ## Compute footprint & requirements
255
  ### Training
 
246
  Llama-3-8B-Instruct, Llama-3.2-3B, Qwen2.5-3B and Phi-3-medium-14B. (Llama-3.1-8B's leaderboard IFEval is lower than
247
  its scores on some later evaluations; it is shown here as the archived value.)
248
 
249
+ **Commercial reference baselines (reported).** KOS-V4's 62.7 sits squarely in the original commercial-API tier.
250
+ These are provider/endpoint figures (commercial models are not on the Open LLM Leaderboard), reported as published by
251
+ each provider, so treat them as approximate, commercial IFEval is often reported as strict or 2-metric rather than
252
+ the 4-metric AVG4 used in the open-model table above.
253
+
254
+ | commercial model | company | endpoint | reported IFEval |
255
+ | :-- | :-- | :-- | :-- |
256
+ | GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | ~60.7 (AVG4) / ~63 (strict-prompt) |
257
+ | Cohere Command R | Cohere | command-r (early 2024) | ~62.0 |
258
+ | Gemini 1.0 Pro | Google | gemini-1.0-pro-001 | ~58-61 |
259
+
260
+ **Takeaway.** From a fraction of the data and 24 GPUs, KOS-V4 (62.7) runs **neck-and-neck with the original
261
+ commercial API generation**, GPT-3.5-turbo, Cohere Command R, and Gemini 1.0 Pro.
262
+
263
  **Sources.** Metric definition: the official IFEval implementation, `google-research/instruction_following_eval`.
264
  Open-model AVG4: computed from `open-llm-leaderboard/results`. Company, parameter count, and release date:
265
  `open-llm-leaderboard/contents`. GPT-3.5-turbo: AVG4 from OpenAI's reported IFEval sub-metrics
266
+ (prompt/inst strict + loose = 53.8 / 64.7 / 56.6 / 67.5). Commercial reference baselines (GPT-3.5-turbo, Cohere
267
+ Command R, Gemini 1.0 Pro): figures reported by the respective providers for the named endpoints; these are not
268
+ Open-LLM-Leaderboard entries and their metric basis is provider-defined, hence marked approximate.
269
 
270
  ## Compute footprint & requirements
271
  ### Training