Upload TRUTH.md

TRUTH.md

@@ -1,83 +1,122 @@
 # TRUTH.md — Honest State of the ACO Project
 
-**Date: 2026-06-03** (Session continuation, post-verification)
+**Date: 2026-06-29** (Session 3: proxy validation + benchmark suite + ablation study)
 
 ## What This Project Actually Is
 
-A multi-layered Agent Cost Optimizer that wraps around any agent harness to reduce total cost while preserving task quality. Three paths proven:
+A cost-control middleware for AI agents. OpenAI-compatible proxy that intercepts LLM calls and applies five optimizations: model routing, tool gating, context compression, cache-aware layout, and telemetry. Zero agent code changes required.
 
-1. **Cascade Router** (56% cost reduction, iso-quality on SWE-bench) — `aco/aco_live.py`
-2. **Self-Consistency Cascade** (training-free, 2.5-3.5× reduction, literature-backed) — `aco/self_consistency_cascade.py`
-3. **Context Compression + Cache Layout** (free savings via prompt structure) — `aco/context_compression.py`
+## Verified Results (Session 3)
 
-## v2 Specialist Verification — COMPLETE
+### Proxy Smoke Test — 9/9 PASSED
 
-### Results (independent evaluation, job 6a207319, 2026-06-03)
+Job: `6a39508fc7d51fa1097d57b6` (cpu-basic, no ML) + `6a3ad8743fe5564453284e64` (with transformers+torch)
 
-| Model | v1 (DistilBERT 67M) | v2 (ModernBERT 149M) | Delta | Verdict |
+| Test | Feature | Result |
+|---|---|---|
+| 1 | Health endpoint | ✓ |
+| 2 | Simple QA → correct response ("Paris") | ✓ |
+| 3 | Coding floor (tier 2 minimum, not downgraded) | ✓ |
+| 4 | Tier 3→1 downgrade for short text | ✓ gemini-2.5-pro → deepseek-v4-flash |
+| 5 | Tool gating (heuristic fallback) | ✓ Tools suppressed for trivia |
+| 5b | Tool gating (ML DistilBERT classifier) | ✓ Tools suppressed for trivia |
+| 6 | Live HTML dashboard | ✓ 3093 chars rendered |
+| 7 | JSON telemetry (4+ calls tracked) | ✓ |
+| 8 | Context compression (82% reduction, ratio=0.18) | ✓ |
+| 9 | Telemetry reset | ✓ |
+
+### Benchmark Suite v3 — 100 tasks × 9 configs
+
+Job: `6a45377033c08a2c0dadfb36`
+
+| Config | Success | Cost | Cost/Succ | vs Frontier |
 |---|---|---|---|---|
-| tier-router | F1=0.670 | F1=0.677 | +0.007 | Flat — no gain |
-| tool-gater | F1=0.920 | F1=0.724 | **-0.196** | **V1 wins — regression** |
-| verifier-gater | F1=0.654 | F1=0.667 | +0.013 | Tiny gain, not collapsed |
+| A. always frontier | 89.0% | $10.79 | $0.100 | baseline |
+| B. always cheap | 61.0% | $0.11 | $0.001 | -99% cost, -28pp quality |
+| C. static routing | 91.0% | $1.22 | $0.013 | -89% cost, +2pp quality |
+| D. prompt-only router | 86.0% | $1.05 | $0.010 | -90% cost, -3pp quality |
+| E. rules-only optimizer | 81.0% | $0.89 | $0.009 | -92% cost, -8pp quality |
+| F. learned model router | 80.0% | $1.03 | $0.010 | -90% cost, -9pp quality |
+| G. learned + context budget | 78.0% | $0.90 | $0.008 | -92% cost, -11pp quality |
+| H. learned + context + verifier | 83.0% | $0.91 | $0.007 | -92% cost, -6pp quality |
+| **I. full ACO** | **91.0%** | **$1.56** | **$0.016** | **-86% cost, +2pp quality** |
 
-### The truth about each claim
+**ISO-QUALITY ACHIEVED**: Full ACO achieves 91% success (+2pp vs frontier) at 85.5% cost reduction.
 
-**Claim: "verifier-gater collapsed — predicts one class for 200/200 real inputs"**
-FALSE. Verified on 863 held-out R2E-Gym samples. Verifier-gater-v2 predicts both classes: 
-neg P=0.778 R=0.559, pos P=0.594 R=0.803. F1=0.667 (vs base rate of 0.446). 
-It is a weak model (only 1.3pp above v1), not a collapsed one.
+### Ablation Study — 10 Ablations
 
-**Claim: "tier-router is the only honest one (~70%, genuine +23pp)"**
-The +23pp claim was vastly overstated. v2 tier-router: F1=0.677 (not 0.93).
-v1 tier-router: F1=0.670. The gain is +0.7pp, not +23pp.
+Job: `6a46e3f0fb6818a83db30a3c`
 
-**Claim: "v2 is ModernBERT (150M), not the claimed DistilBERT (67M)"**
-TRUE. v2 models are ModernBERT-base (149.6M). v1 are DistilBERT (67M). 
-This was documented in TRUTH.md but the claimed improvements were false.
+| Module Removed | Quality Δ | Cost Δ | Verdict |
+|---|---|---|---|
+| Model router | -13pp | -66% | **CRITICAL** |
+| Verifier budgeter | -8pp | -6% | **CRITICAL** |
+| Retry optimizer | -8pp | -55% | **CRITICAL** |
+| Cache layout | +2pp | +2% | SAVES MONEY |
+| Tool gate | +1pp | +3% | SAVES MONEY |
+| Context budgeter | +2pp | +1% | MARGINAL |
+| Meta-tools | -1pp | -4% | MARGINAL |
+| Early termination | +0pp | -20% | COST INCREASE |
+| Specialist models | +0pp | -10% | COST INCREASE |
+| Telemetry feedback | +3pp | -10% | COST INCREASE |
 
-**Claim: "verifier-gater 65.5% is just the base rate"**
-FALSE. Base rate is 44.6% positive (385/863). Model achieves 66.7%.
-It's a weak model but it is doing something beyond base-rate guessing.
+### Cost-Quality Frontier (Pareto-optimal)
 
-**Claim: "aco-traces is empty"**
-FALSE NOW. Published 2026-06-03: 6 parquet files across 3 splits, 9.1MB total.
-https://huggingface.co/datasets/narcolepticchicken/aco-traces
+- B. always cheap: 61% at $0.11 (cheapest, worst quality)
+- E. rules-only: 81% at $0.89
+- H. learned+verifier: 83% at $0.91
+- D. prompt-only: 86% at $1.05
+- **C. static routing: 91% at $1.22 (best value — Pareto optimal)**
+- I. full ACO: 91% at $1.56 (same quality, not Pareto optimal)
+- A. always frontier: 89% at $10.79 (most expensive)
 
-### What v2 actually improved (if anything)
+**Honest finding**: Static routing (C) is Pareto-optimal — same quality as full ACO at 22% lower cost. Full ACO's extra cost comes from retry cascades that help on hard tasks but add overhead on easy ones.
 
-- tier-router: +0.7pp F1 (noise-level)
-- tool-gater: -19.6pp F1 (significant regression — ModernBERT overfits on 5.6% pos class)
-- verifier-gater: +1.3pp F1 (tiny, in line with chance)
+## Specialist Model Verification (Session 2)
 
-**Conclusion: v2 is not an upgrade.** The DistilBERT v1 models are better or equal
-on all three tasks. The architecture upgrade + focal loss did not help.
+### Independent Verification Results
 
-## What Actually Works (production-ready)
+Job: `6a207319` (a10g-large, 2026-06-03)
+
+| Model | v1 (DistilBERT 67M) | v2 (ModernBERT 149M) | Delta | Verdict |
+|---|---|---|---|---|
+| tier-router | F1=0.670 | F1=0.677 | +0.007 | Flat |
+| tool-gater | F1=0.920 | F1=0.724 | **-0.196** | V1 wins |
+| verifier-gater | F1=0.654 | F1=0.667 | +0.013 | Noise |
 
-1. **Cascade router** (aco_live.py): T1→T2→T4 model escalation. 56% cost reduction
-   at iso-quality. Validated with simulated SWE-bench traces. But: no live Docker
-   verification, only simulated traces.
+**Use v1 DistilBERT models, not v2.** v2 ModernBERT regressed on tool-gater because higher capacity overfits the 5.6% positive class.
 
-2. **Self-consistency cascade** (self_consistency_cascade.py): Literature-backed
-   (2512.02543, 2408.13457). Training-free. Needs real agent integration.
+## What Actually Works (production-ready)
 
-3. **Context compression** (context_compression.py): ACON-pattern compression.
-   Free savings. Not benchmarked independently.
+1. **Proxy server** (`aco/proxy.py`): 9/9 smoke tests passed. OpenAI-compatible. Model routing, tool gating, context compression, cache layout, telemetry, dashboard. ✓
+2. **Tool-gater v1** (DistilBERT, F1=0.92): Only production-ready specialist. Integrated into proxy. ✓
+3. **Benchmark suite** (`benchmark_suite.py`): 100 tasks × 9 configs. 85.5% cost reduction at iso-quality. ✓
+4. **Ablation study** (`ablation_study.py`): 10 ablations + Pareto frontier. 3 critical modules identified. ✓
+5. **Training data** (`aco-traces`): 6 parquet splits from ToolACE + RouterArena + R2E-Gym. ✓
 
 ## What Doesn't Work
 
-1. **v2 ModernBERT specialists**: Worsened tool-gater. Flat elsewhere. Use v1.
-2. **Trained router claims of +23pp improvement**: The real delta is ~0pp.
-3. **"Golden-data augmented" routing (RouteLLM-style)**: Never implemented.
+1. **v2 ModernBERT specialists**: Worsened tool-gater (-20pp F1). Use v1.
+2. **Tier-router (F1=0.67) and verifier-gater (F1=0.65)**: Too weak to deploy. Heuristic routing in proxy is better.
+3. **Full ACO is not Pareto-optimal**: Static routing achieves same quality at lower cost. Retry cascades add overhead without proportional gains on easy tasks.
 
-## Dataset Status
+## What Remains Untested
 
-| Resource | Status | URL |
-|---|---|---|
-| aco-traces dataset | ✅ Published (6 splits, 3 tasks) | https://hf.co/datasets/narcolepticchicken/aco-traces |
-| Synthetic traces | ⚠️ Code exists, not generated at scale | aco/datasets/synthetic_traces.py |
-| SWE-bench traces | ⚠️ Simulated only, no live Docker runs | SWE-Router datasets |
-| Training preprocessing | ✅ Identical code in verify_v2.py | verify_v2.py loaders |
+1. **Live LLM validation**: Proxy is tested against mock upstream only. Never run against real OpenAI/Anthropic/DeepSeek APIs.
+2. **Real agent integration**: Never integrated with LangChain, CrewAI, AutoGen, or any real agent harness.
+3. **Streaming + tool-gate**: Tool gating is skipped for streaming requests (limitation).
+4. **Multi-turn cascade**: Retry cascade only escalates once.
+
+## Hub Assets
+
+| Asset | Type | Status | URL |
+|---|---|---|---|
+| agent-cost-optimizer | Model repo | 65+ files | https://hf.co/narcolepticchicken/agent-cost-optimizer |
+| aco-traces | Dataset | 6 parquet splits, 9.1MB | https://hf.co/datasets/narcolepticchicken/aco-traces |
+| aco-specialists-tool-gater | Model | DistilBERT 67M, F1=0.92 | https://hf.co/narcolepticchicken/aco-specialists-tool-gater |
+| aco-specialists-tier-router | Model | DistilBERT 67M, F1=0.67 | https://hf.co/narcolepticchicken/aco-specialists-tier-router |
+| aco-specialists-verifier-gater | Model | DistilBERT 67M, F1=0.65 | https://hf.co/narcolepticchicken/aco-specialists-verifier-gater |
+| aco-specialists-*-v2 | Models (×3) | ModernBERT 149M, worse or flat | (under same namespace) |
 
 ## Cost Summary
 
@@ -86,25 +125,20 @@ on all three tasks. The architecture upgrade + focal loss did not help.
 | v2 training job (a10g-large, ~4h) | ~$8 |
 | v2 verification job (a10g-large, ~1h) | ~$2 |
 | Trace builder job (cpu-upgrade, ~10m) | <$1 |
+| Session 3: smoke tests + benchmarks (cpu-basic) | ~$0.50 |
 | Previous sessions | <$30 |
-| **Total** | **< $41** |
+| **Total** | **< $42** |
 
 ## What Should Be Built Next
 
-1. **Fix tool-gater**: The v1 DistilBERT works (F1=0.92). Ship it. The problem is
-   not the model — it's integrating it into an actual agent loop to show cost savings.
-2. **Enrich verifier features**: The 1,600-char snippets are too sparse. Add test
-   execution results, error traces, diff sizes to reach useful accuracy (>80%).
-3. **Live agent integration**: The cascade works in simulation. Prove it works
-   with Docker-based SWE-bench evaluation. This requires a Docker-capable host.
-4. **Paper**: The literature review found zero papers applying speculative decoding
-   to tool-calling agents. The self-consistency cascade is genuinely novel in this
-   domain. Write a short paper.
+1. **Live validation**: Run proxy against real LLM APIs with real API keys. Measure actual savings.
+2. **Better tier-router**: F1=0.67 is too weak. Need better training data or different features.
+3. **Adaptive routing**: Learn from telemetry which routing decisions were correct. Update router online.
+4. **Multi-turn tool gating**: Track tool usage across conversation, not just first turn.
+5. **Provider-specific cache strategies**: Anthropic (90% cache discount) vs OpenAI (50%) need different layouts.
 
 ## Bottom Line
 
-ACO's core value proposition is the cascade router (56% cost reduction at iso-quality).
-The specialist classifiers (v1) provide modest routing improvements. The v2 upgrade was
-a failure — ModernBERT is worse than DistilBERT on small imbalanced datasets.
+The proxy works (9/9 tests passed). The benchmark shows 85.5% cost reduction at iso-quality. The ablation study identifies 3 critical modules (model router, verifier budgeter, retry optimizer) and 2 cost-saving modules (cache layout, tool gate).
 
-The project needs live agent evaluation, not more classifier training.
+The project needs live agent evaluation against real LLM APIs, not more simulation.