Upload BLOG_POST.md

d8c398f verified about 2 months ago

preview code

Raw

History Blame Contribute Delete

9.46 kB

Building a Cost Control Layer for AI Agents

85.5% cost reduction at iso-quality. Here's how.

The Problem

Most agent cost is wasted. When you send "What is the capital of France?" to a frontier model, you're paying $0.10+ for a question a tier-1 model can answer for $0.001. When your coding agent sends 50k tokens of error logs to Claude Opus, you're paying frontier prices for context that could be compressed 5×. When your agent calls a web search tool to answer "What is 2+2?", you're wasting a tool call.

We built ACO (Agent Cost Optimizer) to fix this. It's a proxy that sits between your agent and LLM providers, applying cost optimizations transparently — no agent code changes required.

What We Built

A FastAPI proxy that intercepts /v1/chat/completions calls and applies five optimizations:

Model routing — routes to cheapest adequate model based on query complexity
Tool gating — suppresses unnecessary tool calls using a trained DistilBERT classifier (F1=0.92)
Context compression — trims stack traces, thinking-only turns, and verbose outputs
Cache-aware layout — reorders prompts (system first, dynamic last) for provider prefix-cache discounts
Telemetry — live dashboard + JSON API for cost tracking

The proxy is OpenAI-compatible. Point your agent at http://localhost:8080/v1 and it works.

The 10-Module Architecture

The full spec defines 10 optimization modules. We implemented all 10:

Module	What It Does	Status
Cost Telemetry Collector	Normalized trace schema + dashboard	✅ Production
Task Cost Classifier	Classifies by difficulty/risk/domain	✅ Heuristic
Model Cascade Router	Cheapest adequate model selection	✅ Production
Context Budgeter	Decides what to compress/omit	✅ Production
Cache-Aware Layout	Reorders for prefix-cache reuse	✅ Production
Tool-Use Cost Gate	ML classifier (F1=0.92)	✅ Production
Verifier Budgeter	Selective verification	✅ Heuristic
Retry/Recovery Optimizer	Cascade retry with model escalation	✅ Heuristic
Meta-Tool Miner	Repeated workflow compression	✅ Heuristic
Early Termination	Doom detection for failing runs	✅ Heuristic

Training the Tool-Gater

The tool-gater is the only module with a trained ML model. It's a DistilBERT (67M params) classifier that predicts whether a query needs tools.

Training data: 3,841 positive examples (queries needing tools) from ToolACE + RouterArena, 48,666 negative examples (queries that don't). Published as narcolepticchicken/aco-traces.

Result: F1=0.92 on held-out test set. The classifier correctly gates trivia, definitions, and simple explanations — saving a tool call that would have cost $0.0001+ each.

What didn't work: We also trained a v2 with ModernBERT (149M params) on the same data. It regressed to F1=0.72 because ModernBERT's higher capacity overfits the 5.6% positive class. Smaller model + shorter context (512 vs 2048) acts as regularization. Published honestly with per-class metrics.

Benchmark Results

We simulated 100 tasks across 5 domains (coding, research, tool-use, doc/QA, long-horizon) through 9 configurations:

Config	Success Rate	Total Cost	Cost/Succ	vs Frontier
A. always frontier	89.0%	$10.79	$0.100	baseline
B. always cheap	61.0%	$0.11	$0.001	-99% cost, -28pp quality
C. static routing	91.0%	$1.22	$0.013	-89% cost, +2pp quality
D. prompt-only router	86.0%	$1.05	$0.010	-90% cost, -3pp quality
E. rules-only optimizer	81.0%	$0.89	$0.009	-92% cost, -8pp quality
F. learned model router	80.0%	$1.03	$0.010	-90% cost, -9pp quality
G. learned + context	78.0%	$0.90	$0.008	-92% cost, -11pp quality
H. learned + context + verifier	83.0%	$0.91	$0.007	-92% cost, -6pp quality
I. full ACO	91.0%	$1.56	$0.016	-86% cost, +2pp quality

Key finding: Full ACO achieves iso-quality (actually +2pp better) at 85.5% cost reduction. Cost per successful task drops from $0.10 to $0.016 — a 6.2× improvement.

The full ACO uses more cost than configs E-H because it includes retry cascades (escalating to stronger models on failure) and verifier calls. These add cost but recover quality — the exact tradeoff we want.

Ablation Study: Which Modules Actually Matter?

We removed each module from the full ACO and re-ran the benchmark:

Module Removed	Quality Δ	Cost Δ	Verdict
Model router	-13pp	-66%	CRITICAL
Verifier budgeter	-8pp	-6%	CRITICAL
Retry optimizer	-8pp	-55%	CRITICAL
Cache layout	+2pp	+2%	SAVES MONEY
Tool gate	+1pp	+3%	SAVES MONEY
Context budgeter	+2pp	+1%	MARGINAL
Meta-tools	-1pp	-4%	MARGINAL
Early termination	+0pp	-20%	COST INCREASE*
Specialist models	+0pp	-10%	COST INCREASE*
Telemetry feedback	+3pp	-10%	COST INCREASE*

*"COST INCREASE" means removing the module reduces cost without hurting quality — the module is spending money without ROI. However, early termination and specialist models serve as safety nets for edge cases not captured in the simulation.

Three modules are critical: model router, verifier budgeter, and retry optimizer. Remove any one and quality drops by 8-13 percentage points.

Two modules save money: cache layout and tool gate. Remove them and cost goes up — they're the pure cost-savers.

Cost-Quality Frontier

The Pareto frontier shows which configs are not dominated:

B. always cheap:       61% quality at $0.11    (cheapest, worst quality)
E. rules-only:         81% quality at $0.89
H. learned+verifier:   83% quality at $0.91
D. prompt-only:        86% quality at $1.05
C. static routing:     91% quality at $1.22    (best value)
I. full ACO:           91% quality at $1.56    (same quality, more expensive)
A. always frontier:   89% quality at $10.79    (most expensive)

Interesting finding: Static routing (C) is Pareto-optimal — it achieves the same 91% quality as full ACO at 22% lower cost. The full ACO's extra cost comes from retry cascades and verifier calls that help on hard tasks but add overhead on easy ones.

For production deployment: use static routing as the default, enable full ACO modules for high-risk/long-horizon tasks.

How It Works: The Routing Logic

Request: model="gemini-2.5-pro" (tier 3), query="What is 2+2?"

1. Extract user text (10 chars)
2. Check: tier >= 3 AND text < 300 chars? → YES
3. Route to: deepseek-v4-flash (tier 1)
4. Cost: $0.0001 instead of $0.01 (100× savings)

Request: model="gemini-2.5-pro", query="Implement a distributed rate limiter in Go"

1. Extract user text (42 chars)
2. Check: tier >= 3 AND text < 300? → YES, but...
3. Check: coding keywords? → YES ("implement")
4. Route to: gpt-5-mini (tier 2, coding floor)
5. Cost: $0.001 instead of $0.01 (10× savings)

How It Works: The Tool-Gater

# Query: "What is the capital of France?"
# Tools: [web_search, calculator, ...]

text = "Query: What is the capital of France?"
inputs = tokenizer(text, truncation=True, max_length=512, return_tensors="pt")
logits = model(**inputs).logits
probs = softmax(logits)

# probs = [0.94, 0.06]  → P(skip_tool)=0.94, P(call_tool)=0.06
# Gate: remove tools from request (save $0.0001 tool overhead)

The classifier was trained on 52,507 examples from ToolACE and RouterArena. It correctly identifies that trivia questions don't need tools — even when tools are offered.

What's Honest About This

The 85.5% savings is simulated. We haven't validated against live LLM APIs yet. The simulation uses realistic cost models and quality estimates, but real-world savings will differ.
The tool-gater is the only trained model that works. The tier-router (F1=0.67) and verifier-gater (F1=0.65) are too weak to deploy. We published them honestly with their actual metrics.
The proxy is tested end-to-end (9/9 smoke tests passed) but only against a mock upstream. Real provider responses may have different formats, error modes, and edge cases.
Static routing beats full ACO on the Pareto frontier. This is an honest finding: the full system's retry cascades add cost without proportional quality gains on easy tasks.

What's Next

Live validation: Run the proxy against real LLM APIs and measure actual savings
Better tier-router: The current F1=0.67 is too weak. Need better training data or a different approach
Adaptive routing: Learn from telemetry which routing decisions were correct
Multi-turn optimization: Currently only first-turn tool gating. Need to track tool usage across conversation
Provider-specific caching: Anthropic's cache (90% discount) vs OpenAI's (50% discount) need different strategies

narcolepticchicken
/

agent-cost-optimizer