Add evaluation results

#4
by SaylorTwift HF Staff - opened

Add evaluation results for thinkingmachines/Inkling-2-Small

Summary

This PR adds evaluation results from the model card's benchmark table for thinkingmachines/Inkling-2-Small to the .eval_results/ directory, following the Hugging Face Hub evaluation-results specification.

Benchmarks Added

Mapping notes:

  • SWE-Bench Pro v2 (Hard) → SWE_Bench_Pro_hard, the 51-task HARD subset of SWE-Bench Pro V2.
  • ScreenSpot-Pro and HLE carry notes for their eval conditions (Python tool, image downsampling, text-only subset), as stated on the card.

Benchmarks Skipped (Not Registered on Hub)

The following benchmarks are in the model card but have no registered eval.yaml on the Hugging Face Hub:

  • Coding: FrontierCode v2.0 (25.4), Vibe Code Bench v1.1 (81.6), SWE-Atlas v1.1 (61.9), Sudo L7 (14.1)
  • Autoresearch: PostTrainBench v1.2 (31.9), PMPP-Hard (494.27 AUC), BioMysteryBench (70.0)
  • Search and tools: BrowseComp (90.2), DeepSearchQA (90.9), MCP Atlas (85.4 full / 85.0 public)
  • Knowledge work: AA-Briefcase v1.1 (1527), GDPval-AA v2.1 (1630), JobBench (70.5), DAYJOB Healthcare (6.4)
  • Enterprise: EnterpriseOps-Gym (46.8), EnterpriseBench-CoreCraft (71.8)
  • Finance: Finance Agent v2.1 (67.5), Excel Modeling Benchmark v1.1 (69.5), Tax Agent Bench (65.8), APEX-Accounting v1.1 (14.5), DAYJOB Finance (14.3), DeepFinance (29.8), Diligence Bench (58.1), AA-AnalystAgent (51.2), Tau 3 Banking v1.0.1 (40.4), ATLAS-Finance v1.1 (6.0)
  • Legal: Harvey LAB v1.1 (21.7), Legal Research Bench (46.6)
  • Visual creation: DesignArena (Web Apps 1286, Website 1331, Game Dev 1424, 3D Design 1457), Code Arena WebDev (1653), AA-3D (1021), Code2Video-AA (1020), BVB (71.6)
  • Visual perception: PerceptionBench (64.7), Humanity's Sixth Sense (33.3), BabyVision (68.6 / 93.6 with Python), OmniDocBench v1.6 (94.6; not the same set as the registered Real5-OmniDocBench), SimpleVQA (68.2)
  • Visual reasoning: GDP.pdf (25.2), CharXiv RQ (85.6 / 89.1), Chartography (36.7 / 82.1), VIALS (34.6 / 57.8), SAGE v1.1 (88.6), MuirBench (79.2), ZeroBench main (29.0 / 44.0)
  • STEM reasoning: CritPt v2 (55.7 full / 35.0 hard), HLE-Diamond (28.7; not a task of cais/hle), ArxivMath 10/2026 (27.7), BrokenArxiv 10/2026 (63.3), ARC-AGI-1 (96.5), ARC-AGI-2 (81.7), MysteryMechanism (25.2)
  • Long context: AA-LCR v1.1 (83.3), LongBench v2 (70.6)
  • Factuality: AA-Omniscience v1.1 (16.1 score / 73.3 non-hallucination)
  • Instruction following and multilingual: ComplexConstraints v1.0 (44.5), Global-MMLU-Lite (91.0)
  • Safety: HoneyBench v0.2 (52.6, lower is better), DistressBench (82.0), FORTRESS (83.8 adversarial / 79.7 benign), StrongREJECT (97.1)
  • Cybersecurity: CyberBench v1.1 (72.7), PatchEval (79.6)
  • Indexes (aggregates, not benchmarks): AA Index v5.0 (30.6), Vals Index v2.2 (56.1)

These can be added once the benchmark authors register an eval.yaml on the Hub.

Source

Files Added

  • .eval_results/Inkling-2-Small.yaml

Verification

These results come from the model card's published benchmark table, with the harnesses listed there (mini-swe-agent v2.4.6 for the coding entries). No verified token is provided, since these were not run via HF Jobs with inspect-ai.

Thinking Machines Lab org

Thank you!

aurick changed pull request status to merged

Sign up or log in to comment