GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

Paper describes GLM-4.5; later GLM-4.x releases reuse the same tag/suite on the Hub.


GLM-5: from Vibe Coding to Agentic Engineering


Kimi K2.5: Visual Agentic Intelligence


DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

DeepSeek-V4-Flash-0731 is the official release superseding the V4-Flash preview (same arXiv tag); its agentic benchmark table comes from its own HF model card and is scoped to it ('Agents' Last Exam' -> Agent's Last Exam; 'AutomationBench Public' -> AutomationBench, same 600-task public subset other sources use). DSBench-FullStack and DSBench-Hard are internal DeepSeek test sets. The paper suite is scoped to the V4-Pro/V4-Flash preview models.


Qwen3.5-397B-A17B / Qwen3.5-122B-A10B model card


Qwen3.6-27B model card


NVIDIA Nemotron 3 Ultra Technical Report


LongCat-2.0 announcement blog / HF discussion

HF model card has no benchmark table; numbers are blog-only self-reported. Only SWE-bench Pro and GPQA-Diamond have Hub-registered eval.yaml mappings.


Hy3-preview GitHub README / model card


Agents-A1 technical report


K-EXAONE-236B-A23B technical report


LFM2 Technical Report

LFM2.5-1.2B-Instruct benchmarks (GPQA, MMLU-Pro, IFEval, IFBench, Multi-IF, AIME 2025, BFCL v3) come from its own HF model card, which cites the same LFM2 Technical Report (arXiv 2511.23404); card's 'AIME25'/'BFCLv3' canonicalized. The math/multilingual/MMLU suite is scoped to LFM2.5-350M (from the report).


Nanbeige4.1-3B technical report


OpenSeeker-v2-30B-SFT technical report


A.X-K1 technical report


MiniMax-M2 / M2.1 model card (tagged scaffold papers: R2E-Gym 2504.07164, WebExplorer 2509.06501, FinSearchComp 2509.13160)

No dedicated M2 technical report; tagged arxiv IDs are scaffold/harness methodology references, not the model's own eval report.


MiniMax Sparse Attention (M3's tagged technical report; primarily an architecture paper)

Benchmarks are from a 109B MSA-vs-GQA ablation model in the paper, not the production M3 release (M3's own card shows only an image with no text table).


Qwen3 Technical Report

Qwen3-Coder-480B and Qwen3-Coder-Next have no text benchmark table on their own model cards (image-only charts); Qwen3-Coder-Next's chart filename implies SWE-bench Pro beyond the base Qwen3 suite.


Nemotron 3 Nano (2512.20848) + NVIDIA Nemotron 3 family whitepaper (2512.20856)

2512.20848 is the Nano-specific technical report. 2512.20856 is a family-wide architecture whitepaper; at time of writing only Nano had shipped, so its benchmark tables are internal ablations, not full Super/Ultra evals. RedHatAI and exolabs repos are community requantizations of the NVIDIA originals and inherit the same suite.


ToolOrchestra


On Data Engineering for Scaling LLM Terminal Capabilities

AIME/LiveCodeBench/SWE-bench in the paper are used only to validate the teacher model (DeepSeek-V3.2), not to score the Nemotron-Terminal models themselves.


TMax: A simple recipe for terminal agents

Only the 9B size is additionally evaluated on SWE-Bench Verified, AIME 2024/2025, and alternate agent harnesses for generalization checks.


Gemma 3 Technical Report


The Llama 3 Herd of Models (Llama-3.1); Llama 4 Maverick has no formal Meta arXiv paper, benchmarks from HF card/blog

The 2204.05149 tag on both Hub repos is a carbon-footprint-methodology citation, not the technical report. Llama 3.1's real paper is 2407.21783; Llama 4 Maverick is blog/card-only.


MiroThinker v1.5 technical report

No coding or math benchmarks used - purely an agentic search/deep-research model.


gpt-oss Model Card

Preparedness Framework (frontier-risk) evals were run on gpt-oss-120b only, not 20b.


Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters (+ Step-3.7-Flash model card, no paper)

Step-3.7-Flash has no arXiv paper; it's VLM-oriented and shares only partial overlap with 3.5-Flash's suite (Terminal-Bench, SWE-Bench, HLE).


MiniMax-M2.5 / M2.7 model cards


Laguna family model cards (MuVeraAI + poolside)

MuVeraAI/Laguna-XS.2 is an unedited older repack of poolside/Laguna-XS.2's card (stale scores, leftover self-references to the poolside repo).


OrionLLM model cards


Obscure single-model cards (CohereLabs, mindlab-research, internlm, HelpingAI)

HelpingAI/Dhanishtha-2.0-0126's model card is an unfilled auto-generated stub - no benchmark data exists.


XiaomiMiMo V2 family model cards


Ornith-1.0 family model cards

All 3 sizes share one 10-item agentic coding suite; no knowledge/math benchmarks used at all.


DeepSeek-V3.2 technical report (PDF hosted in HF repo)


IBM Granite 4.0/4.1 model cards

4.1-8B additionally adds SimpleQA, MTBench, aggregated Eval+, MULTIPLE, Tulu3 Safety Eval vs. the 4.0-h generation.


Qwen3.5/3.6 smaller-size model cards (full benchmark tables, per model)

Full tables extracted directly from each model's own card (not diffed against the flagship). Qwen3.5-27B and 35B-A3B share a byte-identical table; Qwen3.5-0.8B and 2B likewise share one. Qwen3.5-4B and Qwen3.6-35B-A3B each have their own distinct suite.


Kimi K2: Open Agentic Intelligence

Kimi-K2-Thinking additions (IMO-AnswerBench, HealthBench, BrowseComp, BrowseComp-ZH, Seal-0, FinSearchComp-T3, SciCode, Longform Writing) come from its own HF model card, not the original paper.


Anthropic system cards for Claude Opus 4.8, Sonnet 5, Haiku 4.5, and Fable 5

Each model has its own system card (not one shared document) but the suites overlap heavily. Not all benchmarks are run on every tier — frontier-only evals (USAMO, GDPval-AA, BioMysteryBench, ExploitBench, domain-specific agentic benchmarks) cluster on Opus/Fable and are largely skipped for Haiku. Not on any HF leaderboard (closed API models); scores were cross-referenced partly via other labs' model cards that cite Claude comparison numbers.


GPT-5.5 System Card + "Introducing GPT-5.5" announcement blog

Released April 23, 2026. Capability numbers from the announcement blog comparison chart (vs GPT-5.4, Claude, Gemini 3.1 Pro); safety/preparedness from the official system card PDF.

Production Benchmarks (disallowed content)


GPT-5.6 Preview (Sol/Terra/Luna) System Card + secondary coverage

Limited preview as of June 2026 (codenamed Sol/Terra/Luna), no full ChatGPT release yet. Capability numbers (Terminal-Bench 2.1, Agent's Last Exam) are OpenAI-published; SWE-bench Pro was notably NOT published at preview; other capability figures circulating (FrontierMath T4, SWE-bench Verified) are explicitly unconfirmed third-party estimates, excluded here. Safety suite mostly retains GPT-5.5's evals with several additions/replacements.

Production Benchmarks (disallowed content, reorganized categories)

Search and Function-Calling prompt-injection eval


Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

Not on any HF leaderboard (closed API models). Paper also compares against Gemini 1.5 Flash/Pro as legacy baselines (not added as separate model entries here). HiddenMath-Hard, Global-MMLU-Lite, ECLeKTic, ZeroBench, Vibe-Eval, and BetterChartQA are evaluated on Gemini models only, not run on competitor models in the paper's comparison tables. Section 2's HTML rendering was empty on arXiv; benchmarks extracted from the PDF tables.


tencent/Hy3 full-release model card (chart images, no arXiv paper)

Full/final release, distinct from tencent/Hy3-preview (already in this DB under hy3-preview-card). Benchmarks come entirely from two chart images (assets/benchmark.png, assets/benchmark-appendix.png), no text table or paper. Roughly 2x broader suite than the preview: retains most preview benchmarks (with version bumps, e.g. Terminal-Bench 2.0->2.1) while dropping Tsinghua Qiuzhen Math PhD Exam, CHSBO 2025, AdvancedIF, and LongBench v2 from the preview suite, and adding many new public + internal evals.


DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Internal safety/jailbreak suite (unnamed individually)


Llama 2: Open Foundation and Fine-Tuned Chat Models


Mistral 7B

Internal refusal-rate / content-moderation prompt set


Phi-4 Technical Report

Internal Responsible AI eval (Grounding, Jailbreak DR1, multi-turn harm)


Phi-3-mini-4k-instruct model card (report at aka.ms/phi3-tech-report, no arXiv ID)


Jamba: A Hybrid Transformer-Mamba Language Model

Base/pretrained checkpoint, not instruction-tuned; no safety/multilingual/alignment benchmarks.


Apertus Technical Report


"Announcing Grok" xAI blog (no formal paper; HF card is sparse)

Benchmark table (GSM8K, MMLU, HumanEval, MATH) verified via user-provided screenshot of the original xAI announcement table (comparing Grok-0 33B, LLaMa 2 70B, Inflection-1, GPT-3.5, Grok-1, Palm 2, Claude 2, GPT-4). Very limited public benchmarking beyond this - no safety/multilingual/instruction-following disclosure.

2023 Hungarian national high school math final exam


Command R+ model card (Open LLM Leaderboard results)

Card explicitly notes these don't capture RAG/multilingual/tool-use performance (pointed to a separate blog/Chatbot Arena instead, no numeric data given there).


Olmo 3


HyperCLOVA X 32B Think


GigaChat3.5-432B-A28B model card (base + instruct benchmarks)

Sber (Russian lab). Base and instruct checkpoints are evaluated on different suites; scoped per-model below.


MiniCPM4/4.1 report (2506.07900) + data-management companion paper (2602.09003) - no dedicated MiniCPM5 report exists

None of MiniCPM5-1B's 4 tagged arXiv IDs is actually titled 'MiniCPM5' - benchmarks are proxied from the MiniCPM4/4.1 report (predecessor generation, same architecture lineage/scale) plus a training-data companion paper.


Llama-Nemotron: Efficient Reasoning Models

Distinct from the separate 'Nemotron 3' family already in this DB - this is NVIDIA's efficiency distillation of Meta's Llama 3.3.


Gemma 3n model card / blog (no dedicated arXiv report; same dataset-citation-tag style as Gemma 3)


"Grok-1.5" xAI announcement (primary page 403-blocked to fetchers; sourced from secondary aggregators)

Benchmark table (MMLU, MATH, GSM8K, HumanEval) verified via user-provided screenshot of the original xAI announcement table (comparing Grok-1, Grok-1.5, Mistral Large, Claude 2, Claude 3 Sonnet, Gemini Pro 1.5, GPT-4, Claude 3 Opus).


"Grok-2" xAI announcement (primary page 403-blocked; sourced from secondary aggregators)

Benchmark table (GPQA, MMLU, MMLU-Pro, MATH, HumanEval, MMMU, MathVista, DocVQA) verified via user-provided screenshot of the original xAI announcement table (comparing Grok-1.5, Grok-2 mini, Grok-2, GPT-4 Turbo, Claude 3 Opus, Gemini Pro 1.5, Llama 3 405B, GPT-4o, Claude 3.5 Sonnet). Most granular scores in the table are for the Grok-2 mini variant rather than full Grok-2, but the benchmark names/columns are identical for both.


"Grok 4" xAI announcement (primary page 403-blocked; sourced from secondary aggregators)

GPQA, LiveCodeBench, USAMO 2025, HMMT 2025, AIME 2025, ARC-AGI-2, and HLE (incl. w/ Python+Internet tools variant) verified via user-provided screenshots of the original xAI announcement charts (these also show a 'Grok 4 Heavy w/ Python' multi-agent configuration scoring higher on every chart - treated as an eval setting of Grok 4, not a separate model). SWE-bench Verified, Vending-Bench, and Artificial Analysis Intelligence Index were not visible in the screenshots provided and remain sourced from secondary aggregators only.


"Grok 4.1" xAI announcement (primary page 403-blocked; sourced from secondary aggregators)

EQ-Bench and Creative Writing v3 verified via user-provided screenshots of the original xAI announcement charts (both Elo-normalized leaderboards comparing Grok 4.1/4.1 Thinking against Kimi K2 Instruct, Gemini 2.5 Pro, GPT-5 Chat, Claude Opus 4/Sonnet 4.5, Grok 3/4, and others). Confirms the benchmark name is 'EQ-Bench', not 'EQ-Bench3' as earlier secondary sourcing suggested. LMArena was not visible in the screenshots provided and remains sourced from secondary aggregators only.


Gemma 4 Technical Report

Supersedes the earlier gemma-4-family-card model-card source, which lacked per-model scoping (it applied audio benchmarks to all 5 sizes uniformly). Per the actual paper tables: LMArena (Table 4) is only reported for the 31B and 26B-A4B sizes; CoVoST/FLEURS audio benchmarks (Tables 7-8) are only reported for E2B, E4B, and 12B. Static/vision/long-context benchmarks (Tables 5, 6, 9) are reported for all 5 sizes.


Muse Spark 1.1 Evaluation Report (Meta)

Extracted from the primary evaluation report PDF (112 pp, high confidence); the announcement blog (https://ai.meta.com/blog/introducing-muse-spark-meta-model-api/) has charts only. General capabilities from §5.2; safety/preparedness from Tables 1-2 (named external evals only; internal metrics like Curated CTFs, Internal Sycophancy, BioDesign Tools omitted). Name mappings: 'MCP Atlas'→MCP-Atlas, 'SWE-Bench Pro'→SWE-bench Pro, 'DeepSWE v1.1'→DeepSWE, 'VibeCodeBench v1.1'→VibeCodeBench, 'HealthBench Pro'→HealthBench Professional (assumed same OpenAI benchmark). Kept separate on purpose: ProtocolQA (report doesn't say Open-Ended) vs ProtocolQA Open-Ended; CharXiv Reasoning (explicit reasoning split) vs CharXiv; StrongREJECT v2 vs StrongReject (explicit v2); WebArena-Verified vs WebArena; Toolathlon-Verified vs Toolathlon; OSWorld 2.0 vs OSWorld-Verified.


"Grok 4.5" SpaceXAI announcement + Cursor co-announcement

Primary launch page 403-blocked to fetchers (standard for x.ai/news/*). Suite reconstructed from the Cursor co-announcement (https://cursor.com/blog/grok-4-5 — Cursor co-trained the model; names SWE-Bench Pro, Terminal-Bench, SWE-Bench multilingual, CursorBench, scores in charts only) plus secondary aggregators (MarkTechPost, Crypto Briefing) quoting the launch page: DeepSWE 1.0 62.0 / DeepSWE 1.1 53 (both logged as DeepSWE), Terminal-Bench 2.1 83.3, SWE-bench Pro 64.7, SWE Marathon 29.0 (#1), #1 on Harvey's Legal Agent Benchmark. Medium confidence — no screenshot of the primary chart yet; scores self-reported by xAI, no third-party verification at time of logging. Cursor footnote: CursorBench result inflated by accidental inclusion of an earlier Cursor codebase snapshot in training. Lab logged as 'xAI' for consistency (announcement branding is 'SpaceXAI').


Long-Horizon-Terminal-Bench (LHTB): Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

Third-party benchmark report (Tencent HY LLM Frontier / IntelligenceLab, July 2026), not a lab release source: 46 containerized long-horizon terminal tasks, hidden replay-based verifiers, continuous partial-credit reward, Terminus-2 harness, 90-min budget. HF dataset: IntelligenceLab/Long-Horizon-Terminal-Bench. 'models' lists the evaluated models already present in this dataset (they keep their own release source_id); also evaluated but not dataset rows: Claude Sonnet 4.6, Kimi K2.7 Code, Qwen3.6 Plus, Qwen3.7 Max, Doubao Seed 2.1 Pro, Gemini 3.1 Pro, GPT-5.4, GPT-5.3 Codex, Grok 4.20. High confidence: primary sources (blog + leaderboard data + arXiv paper + HF dataset card). GPT-5.6 entry is the 'GPT-5.6-sol' variant.


EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

Third-party benchmark report (ByteDance Seed, July 2026), not a lab release source: 134 real-world long-horizon tasks (51 open-sourced) across 6 categories; agents iterate in executable environments for 12+ hours per task, scored over the full improvement trajectory (log-sigmoid time-scaling law). HF dataset: ByteDance-Seed/EdgeBench. 'models' lists the evaluated models already present in this dataset (they keep their own release source_id); also evaluated but not a dataset row: GPT-5.4. High confidence: primary sources (HF dataset card leaderboard + arXiv tech report).


Kimi K3 tech report / model card

Benchmarks extracted from the HF model card evaluation table (full tech report PDF on GitHub; no arXiv id, tech blog at kimi.com/blog/kimi-k3). Card's 'HLE-Full' reports without/with tools -> logged as HLE + HLE w/ Tools. Card's tau3-Banking logged as TAU3-Bench Banking (banking split of TAU3-Bench); MCPMark-Verified logged as MCP-Mark Verified (verified split of existing MCP-Mark); 'APEX-Agents' collapsed to existing 'Apex-Agent' (same Mercor leaderboard). PostTrainBench is the full benchmark, distinct from existing PostTrainBench Lite. Kimi Code Bench 2.0 and PerceptionBench are in-house Moonshot benchmarks. High confidence: primary source.


Inkling model card (Thinking Machines Lab)

975B-total/41B-active multimodal MoE (text+image+audio in), Thinking Machines Lab's first open-weights model. Benchmarks from the card's evaluation table (results at effort=0.99). Card variants collapsed to canonical names: 'HLE (text only)'/'HLE (with tools)' -> HLE / HLE w/ Tools; 'SWEBench Pro (Public)' -> SWE-bench Pro; 'MMMU Pro (Standard 10)' -> MMMU-Pro; 'Charxiv RQ (with python)' -> CharXiv; 'BrowseComp (w/ Ctx)' -> BrowseComp; 'Tau 3 Banking' -> TAU3-Bench Banking; 'AA Omniscience' -> AA-Omniscience; 'GDPVal-AA v2' -> GDPval-AA v2. FORTRESS covers the card's Adversarial and Benign splits. High confidence: primary source.


A.X K2 Technical Report

688B-A33B MoE, successor to A.X K1 (Korean Sovereign AI project). Benchmarks from the HF model card eval table (all in thinking mode; tech report PDF on GitHub, no arXiv). Card spellings canonicalized: 'AIME26' -> AIME 2026; 'Humanity's Last Exam' -> HLE; 'GPQA Diamond' -> GPQA-Diamond; 'LiveCodeBench v6 (Feb-May)' -> LiveCodeBench v6; 'BrowseComp (<=10 searches)' -> BrowseComp; 'tau2-Bench (Telecom)' -> TAU2-Bench Telecom (telecom split, AA-evaluated); 'KMO26 (1st round)' -> KMO 2026 (1st round). IMO 2025 (35/42, gold threshold) reported in card prose. High confidence: primary source.


MiniMax H3: An Open Model Breaking the Boundaries Between Tasks and Modalities

Omni-modal audio+video GENERATION system (Hailuo family; text/image/video/audio in, 2K video with native stereo audio out) - not an LLM. Neither the HF model card nor the official blog reports any benchmark table, so no benchmark rows. MiniMax's launch communications cite third-party arena rankings only: #1 open model on LMArena Video Arena (T2V and I2V) and on Artificial Analysis video leaderboards (T2V Elo 1242, #2 overall; Video Editing Elo 1130, #1) as of 2026-07-31.


Muse Glimmer 30B model card

Full benchmark table extracted from the HF model card. Cyber-domain benchmarks (CyberGym, CyberBench) are not in the card's visible table but are described in the linked methodology report (https://research.meta.ai/static/muse-glimmer-methodology, 'Muse Glimmer per-benchmark details'); note the card itself only reports an 'inferred' (not directly measured) Cyber risk designation, so treat these two as lower-confidence. The two arXiv tags on the repo (2504.13181, 2602.06036) are not the model's own technical report -- they are the cited Perception Encoder architecture paper and DFlash speculative-decoding paper, component references rather than a Muse Glimmer paper.


Motif 3: Technical Report

Benchmarks from Table 6 (main evaluation of the released Motif 3 post-trained/unified model). The report also includes a separate Table 4 pretraining evaluation (MMLU, MMLU-Pro, ARC-C, WinoGrande, HellaSwag, PIQA, GSM8K, MATH, HumanEval, MBPP) for the Motif-3-Base checkpoint, which is a distinct HF repo (Motif-Technologies/Motif-3-Base) not logged here since it wasn't the model requested. AA-Omniscience is reported here as two distinct sub-metrics (Accuracy and Non-Hallucination), unlike other sources in this dataset that log a single 'AA-Omniscience' score.