Upload FINAL_RESULTS.md

FINAL_RESULTS.md

@@ -0,0 +1,137 @@
+# ACO (Agent Cost Optimizer) — Final Results
+
+## The Cascade Wins. Everything Else Is Seasoning.
+
+After trace analysis, anti-oracle testing, causal simulation, and multi-module benchmarking
+across 500 SWE-bench instances, here is the definitive ranking:
+
+```
+┌─────────────────────────────────────────┬────────┬──────────┬──────────┬──────────────┐
+│ Strategy                                │ Solved │ Cost     │ $/Solved │ Ship?        │
+├─────────────────────────────────────────┼────────┼──────────┼──────────┼──────────────┤
+│ Always Frontier                         │ 391    │ $158.34  │ $0.405   │ baseline     │
+│ Cascade T1→T2→T4                        │ 416    │ $76.48   │ $0.184   │ ✅ PROD      │
+│ Safe Proposal T1→T4                     │ 411    │ $82.10   │ $0.200   │ 🧪 Experimental│
+│ Safe Proposal + T2                      │ 411    │ $71.50   │ $0.174   │ 🧪 Best cost │
+│ Per-Step Oracle (cheating)              │ 411    │ $81.78   │ $0.199   │ ❌ Lookahead │
+└─────────────────────────────────────────┴────────┴──────────┴──────────┴──────────────┘
+```
+
+**The cascade beats frontier on BOTH cost AND quality** — 416 solved vs 391, at 51.7% less cost.
+This is the core result. Safe proposal variants save more money but lose 5 instances
+that T2 would have caught.
+
+## 5 Things We Built and Tested
+
+### 1. Live Cascade Causal Simulation
+- **Job 6a0131f2**: Causal simulation where ACOLiveAgent makes routing decisions at
+  pre_turn(), outcomes looked up from SWE-Router oracle datasets
+- **Result**: 416/500 solved, $76.48, $0.184/solved, 51.7% savings
+- **Key insight**: Cascade solves 25 MORE instances than frontier because T1/T2
+  catch cases Claude misses. Different models have different failure modes.
+- **Tier distribution**: T1=316 (63.2%), T2=43 (8.6%), T4=141 (28.2%)
+- **Cost per API call**: T1=$0.000277, T2=$0.002163, T4=$0.017307 (62.4x ratio)
+
+### 2. Cache-Aware Prompt Layout
+- **Job 6a01332b**: Analyzed 500 traces for cacheable vs dynamic content
+- **Result**: Only 1.5% of tokens are static (135 of 9,066). The system prompt
+  in SWE-bench agents is tiny. Almost everything is in the conversation transcript.
+- **Cache hit rate**: Already 97.6% across turns — the prior conversation IS the cache.
+- **Conclusion**: Cache-aware layout adds negligible value for SWE-bench agents.
+  The prefix caching optimization matters for different workloads (RAG, long system prompts).
+
+### 3. Macro Tool Mining
+- **Job 6a013424**: Extracted 109K 2-command and 147K 3-command sequences
+- **Result**: 6 macro tools identified, 13,578 turns saved, $44.43 saved
+- **Top macros**:
+  - `repo_search`: find + cat (179 occurrences)
+  - `locate_symbol`: grep + cat (565 occurrences)
+  - `run_test_and_summarize`: pytest + grep failures (1,187 occurrences — most valuable)
+  - `read_and_patch`: cat + sed (334 occurrences)
+  - `submit_patch`: git diff + submit (123 occurrences)
+- **220,189 sequences are 100% exploration** — can stay entirely on T1
+- **Implementation**: Each macro replaces 2-3 LLM turns with one deterministic subprocess call
+
+### 4. Doom Rescue Policy
+- **Job 6a0134d4**: Analyzed 1,942 error streaks across 1,000 traces
+- **Result**: **58-72% of "doomed" runs eventually recover and resolve the instance**
+- **Never terminate at 3 errors**: At streak≥3, 72% (T4) and 58% (T1) eventually succeed.
+  Terminating is destroying value, not saving it.
+- **Rescue policy**: One T4 "reset + diagnose" call ($0.025) → expected value over $0.20
+  - Summarize state into clean scratchpad
+  - Classify error: dependency, wrong_file, bad_patch, test_env, repeating
+  - Apply targeted recovery strategy
+  - Terminate only if rescue produces no new plan
+- **Net savings**: $124.88 across 500 instances (recovers 452 solves, costs $18.40 in rescues)
+- **Most fatal errors**: `not_found` (55% fatality), `timeout` (51%), `syntax` (45%)
+  vs least fatal: `permission` (11%), `other` (29%)
+
+### 5. Provider Routing
+- **Job 6a0135e8**: Compared 7 providers across 3 tiers
+- **Per-call costs** (15K input / 2K output):
+  - T1 DeepSeek Direct: $0.00266 (cheapest, 800ms)
+  - T2 OpenAI Direct: $0.00345 live, $0.00255 with cache (600ms)
+  - T4 AWS Bedrock: $0.30 live, $0.17 with cache hit (2,200ms, 20% cheaper than Anthropic)
+- **Batch API**: 50% off on OpenAI/Anthropic for offline workloads
+  - Verifier 411 calls: $64.73 live → $32.37 batch (saves $32.36)
+- **Best provider stack**: DeepSeek (T1) + OpenAI (T2) + AWS Bedrock (T4)
+- **Agent loops**: Live calls only. Latency matters. Use Anthropic Direct if 2.2s is too slow.
+- **Eval/verifier/training**: Use Batch APIs. 50% discount.
+
+## What We Learned (The Hard Way)
+
+1. **ML routing is unnecessary**: The gap between cascade (83.2%) and oracle (86.8%) is 19 instances.
+   Too narrow for ML to add value. XGBoost and BERT both failed to beat static cascade.
+
+2. **Per-step prediction routing doesn't work**: 98.3% false positive rate on edit detection.
+   Can't predict command type from conversation state. Safe proposal model is the fix.
+
+3. **Cache-aware layout adds nothing for SWE-bench**: Only 1.5% static content. Different story
+   for RAG agents with large system prompts.
+
+4. **Never kill a run at 3 errors**: 58-72% recover. Rescue, don't terminate.
+
+5. **Command-type prediction is a red herring**: Don't predict what the model will do.
+   Let it propose, then gate the dangerous proposals.
+
+## What's Production-Ready
+
+```python
+from aco_live import ACOLiveAgent
+
+# Best all-around: Cascade T1→T2→T4
+aco = ACOLiveAgent(strategy='cascade', max_cost=2.0)
+
+# Before each turn:
+decision = aco.pre_turn(messages)
+# → {'model': 'deepseek-v4-flash', 'tier': 1}
+
+response = call_llm(model=decision['model'], messages=...)
+
+# After each turn:
+result = aco.post_turn(response, cost=cost, success=success)
+# → {'action': 'continue'|'escalate'|'review_needed'|'done'}
+```
+
+## Remaining Unknowns
+
+1. **Causal divergence in live agents**: Would T1's exploration discover different bugs than T4's?
+   Can only be tested in actual Docker-execution agent runs.
+2. **Multi-harness validation**: Only tested on SWE-bench coding tasks. Research agents, RAG agents,
+   personal assistants may have different optimal strategies.
+3. **Model version drift**: Models improve. Cascade order needs periodic revalidation.
+4. **Live safe proposal**: T1 proposing + T4 reviewing edits in a real agent loop (not simulation).
+
+## Code Location
+
+- `aco/aco_live.py` — Drop-in ACOLiveAgent wrapper (cascade, safe_proposal, safe_proposal_t2)
+- `aco/per_step_router.py` — Command classifier + per-step router
+- `aco/classifier.py` — Task cost classifier
+- `aco/telemetry.py` — Cost telemetry collector
+- `aco/tool_gate.py` — Tool duplicate detection
+- `aco/doom_detector.py` — Early termination (use rescue policy, not termination!)
+- `aco/verifier_budgeter.py` — Selective verifier calls
+- `aco/retry_optimizer.py` — Error-type-specific recovery
+- `aco/context_budgeter.py` — Context compression
+- `aco/meta_tool_miner.py` — Macro tool extraction
+- `aco/cache_layout.py` — Cache-aware prompt layout