🚀 Benchmarks are Not Enough: RAMP for Evaluating Agentic Models in Real-World Production Systems with RAMP

Hi Hugging Face Community! 👋 We are excited to share our latest work that challenges the current paradigm of LLM agent evaluation benchmarks.

While Large Language Models (LLMs) are rapidly evolving into autonomous software engineering systems, existing evaluation methodologies are largely centered on static, isolated, and short-horizon benchmarks. We found that high scores on traditional benchmarks poorly reflect practical capabilities under realistic runtime environments that involve long execution chains, tool interactions, and iterative feedback loops.

To bridge this gap, we introduce RAMP (Runtime Assessment of Models in Production), an infrastructure designed to evaluate agents in continuous, stateful, and resource-constrained engineering workflows.

How Far Can Today’s Agentic Models Go on the Ramp to Real System Development?

🛠️ The Workload: Real-World Compiler Construction

Instead of isolated coding puzzles, RAMP evaluates agents on a 6-stage compiler-construction pipeline (based on YatCC). The tasks range from environment setup (T0) and lexer generation (T1) all the way to LLVM IR optimization (T4) and RV64 assembly generation (T5). Each task consumes the output artifact of its predecessor, creating a strict serial dependency chain.

✨ Core Innovation: The "Resurrection Protocol"

In long-horizon tasks, a failure at an early stage usually invalidates all downstream steps, obscuring the model's true capabilities. To solve this, RAMP introduces a Resurrection Protocol.

When an agent fails an intermediate task, the orchestrator automatically transparently injects a "golden artifact" (a perfect intermediate state) and lets the agent continue. This allows us to separate "cannot reach" from "cannot solve," providing unprecedented diagnostic granularity.

Figure 3: The RAMP Pipeline demonstrating Serial Evolution and the Resurrection Protocol.

📊 Shocking Findings from 15 SOTA Models

We evaluated 15 models (including Opus-4.7, GPT-5.5, DeepSeek-v4-Pro, and Qwen-3.6-Max). The results were eye-opening:

• A Clear Capability Ceiling: None of the 15 evaluated models successfully completed the entire pipeline. Even the top-performing model, Opus-4.7, stalled at the IR Generation stage.
• The 2525x Efficiency Gap: Process efficiency varied wildly. Total inference costs ranged from $0.05 (Qwen3-Coder) to $126.24 (Opus-4.7) — an extreme 2525x difference.
• The "Context" Killer: We mapped a detailed failure taxonomy and discovered that Context Failure is the most prevalent hard-stop reason (accounting for 60.0% of failures), predominantly occurring in the middle stages (T2-T3) as history and code artifacts accumulate.

Figure 5: Trade-off of Cost and Performance: Elapsed time and API cost versus mean reward.

⚖️ Beyond Accuracy: The Agent Efficiency Index (AEI)

In production, a model that brute-forces a solution using massive context and time isn't always the best choice. We propose the Agent Efficiency Index (AEI), a composite metric jointly measuring task effectiveness, time, cost, and token utilization.

Under AEI, the rankings flip: GPT-5.5 achieved the highest composite efficiency (AEI 81.57), whereas Opus-4.7, despite having the highest raw task reward, dropped to an AEI of 40.00 due to massive resource overhead.

Read the full paper to explore the deep diagnostics of model behavior and why we need to move past static benchmarks!

• 🔗 Code: https://github.com/Nexa-Language/RAMP
• 💻 Code & Leaderboard: http://ramp.yatcc-ai.com/