Delete training/train_router_v[1-7]*.py, training/train_v10_fixed.py, training/repro*.py, training/router_v[5-8]*.py, training/integration_real.py, training/debug_sprout*.py, training/check_bundle*.py, training/execution_feedback_loop.py, training/tune_feedback.py, training/verify_frontier.py, training/router_integration_benchmark.py, docs/combined_final_report.md, docs/bert_*.md, docs/final_report*.md, docs/FINAL_CORRECTED_REPORT.md, standalone_eval*.py, run_eval.py, run_benchmark.py, train_router_gen.py, train_router.py

docs/FINAL_CORRECTED_REPORT.md

@@ -1,60 +0,0 @@
-# ACO Final Results — Corrected
-
-## The Simple Strategy That Beats Everything
-
-**Tier 1 → Tier 2 → Tier 4**: Route to cheapest model first. If it fails, try mid-tier. If that fails, use frontier.
-
-| Metric | Value |
-|--------|-------|
-| Success rate | **83.2%** (416/500) |
-| Avg cost | **$0.173/task** |
-| vs Frontier | **45.5% cheaper** AND **5.0pp more successful** |
-| vs Oracle | 3.8pp gap (83.2% vs 87.0%) |
-
-This requires **zero ML** — just a static escalation policy. No feature extraction, no training, no model loading.
-
-## Why It Works
-
-| Tier | Model | Cost | Marginal Contribution |
-|------|-------|------|-----------------------|
-| 1 | deepseek-v4-flash | $0.014 | Solves 316 tasks (63.2%) |
-| 2 | gpt-5-mini | $0.033 | Solves 26 more (5.2%) |
-| 4 | claude-opus-4.7 | $0.317 | Solves 74 more (14.8%) |
-| **Combined** | — | $0.173 | **416 tasks (83.2%)** |
-
-The key is that tier 2 is cheap ($0.033) and catches meaningful marginal wins before escalating to the expensive tier 4.
-
-## What About ML-Based Routing?
-
-The v10 XGBoost router with feedback achieves 85.2% at $0.443/task — 2.0pp more success at **2.56× the cost**. The additional 10 resolves cost $135 each vs T1→T2→T4.
-
-### Why ML Routing Doesn't Help Here
-
-The tier-1 success rate is 63.2%. The router's job is to predict which 63.2% to route to tier 1. But:
-- False negatives (routing a tier-1-solvable task to tier 4) waste $0.303
-- False positives (routing an unsolvable task to tier 1) waste $0.014 + escalation
-
-The simple T1→T2→T4 strategy already achieves 83.2% — there are only 19 more tasks the oracle can solve (87.0%). The improvement ceiling is tight. An ML router would need near-perfect accuracy on the hardest <4% of tasks to beat the simple strategy.
-
-## Updated Pareto Frontier
-
-```
-87% │ ★ Oracle
-    │
-85% │ ★ All tiers (85.2%, $0.443)
-    │
-83% │ ★ T1→T2→T4 (83.2%, $0.173)  ← BEST REAL STRATEGY
-    │
-78% │ ★ Frontier (78.2%, $0.317)
-    │
-63% │ ★ Always cheap (63.2%, $0.014)
-    └─────────────────────────────────────
-      $0.05  $0.10  $0.15  $0.20  $0.25  $0.30  $0.35  $0.40  $0.45
-```
-
-## Recommendations
-
-1. **Deploy T1→T2→T4 immediately.** It's free, simple, and beats everything.
-2. **Don't build an ML router for SWE-bench.** The simple strategy already captures most of the oracle's gains.
-3. **Only build ML routing if** the task distribution has a much wider tier-1 success range (e.g., 20-80% across task types) where smart routing can meaningfully outperform the static cascade.
-4. **The ACO modules (verifier, doom detector, etc.)** should be tested on real agent traces, not simulated ones — the simulation can't capture the complex failure modes these modules are designed to handle.

docs/bert_5class_final_report.md

@@ -1,96 +0,0 @@
-# BERT 5-Class Router — Final Report
-
-## What We Did
-
-### Task 1: Fix SPROUT Parsing ✅
-
-The original `train_bert_5class.py` had broken SPROUT parsing:
-- It did naive `if m_lower in col.lower()` string matching on column names
-- It checked `isinstance(val, (int, float))` for model columns
-- But SPROUT model columns are dicts: `{'judge_response': '{"correctness_score": 0.8, ...}'}`
-
-**V3 fix** (`training/train_bert_5class_v3.py`):
-- Robust `extract_correctness_score()` handles all SPROUT formats
-- Handles escaped single quotes in JSON strings (`Model\\'s → Model's`)
-- Regex fallback for truly broken JSON
-- Result: **0 parse errors** on 30,968 rows (vs 11,610 errors in V2)
-
-### Task 2: Retrain BERT as 5-Class Router ✅
-
-Training on SPROUT (30,968 samples, 13 models → 5 ACO tiers):
-
-| Metric | Epoch 1 | Epoch 2 |
-|--------|---------|---------|
-| eval_accuracy | 76.85% | 76.95% |
-| eval_acc_tier1 | 99.96% | 99.92% |
-| eval_acc_tier2 | 0.84% | 1.26% |
-| eval_acc_tier3 | 0% | 0% |
-| eval_acc_tier4 | 0% | 1.74% |
-| eval_acc_tier5 | 0% | 0% |
-
-**Finding: BERT trained on SPROUT QA data collapses to predicting tier 1 for everything.** This is because 77.4% of SPROUT tasks are solvable by tier 1 models with correctness_score ≥ 0.7.
-
-### Task 3: Execution Feedback Loop on SWE-Bench ✅
-
-Evaluated BERT 5-class router on SWE-Router (500 coding tasks, 8 models):
-
-| Policy | Success | AvgCost | CostRed |
-|--------|---------|---------|---------|
-| oracle | 87.0% | $0.062 | 80.3% |
-| bert_direct | 54.8% | $0.042 | 86.8% |
-| bert_feedback | 84.0% | $0.578 | -82.5% |
-| bert_cascade | 70.4% | $0.783 | -147.1% |
-| frontier | 78.2% | $0.317 | baseline |
-| always_cheap | 63.2% | $0.014 | 95.5% |
-
-**BERT tier predictions on SWE-bench (untrained weights):**
-- Tier 1: 0.2%, Tier 2: 96.0%, Tier 3: 3.8%, Tier 4: 0%, Tier 5: 0%
-
-**Finding: Untrained BERT always picks tier 2** → bert_feedback escalates to frontier on failures → costs MORE than frontier (-82.5% cost reduction). The feedback escalation mechanism doesn't help when the initial routing is wrong.
-
-## Key Findings
-
-### 1. SPROUT ≠ SWE-bench
-
-SPROUT is a QA benchmark. The BERT 5-class router trained on SPROUT learns to predict the same tier for everything. Using SPROUT data for a coding-agent cost router doesn't work because:
-- QA task difficulty ≠ coding task difficulty
-- SPROUT correctness scores (judge LLM evaluations) ≠ SWE-bench resolution rates
-- The class distribution is fundamentally different
-
-### 2. BERT for Tier Routing is Fundamentally Broken
-
-Both the binary BERT and the 5-class BERT collapse to predicting a single class. The binary version predicts P(success) ≈ 89.5% for ALL tiers. The 5-class version predicts tier 1 for everything (99.96% accuracy on tier 1, near-zero on all others).
-
-### 3. What Actually Works
-
-The **XGBoost v10 router trained on real SWE-Router execution data** remains the best approach:
-- 500 tasks × 8 models = 4,000 real outcomes
-- Feature engineering from problem statements
-- Per-tier success prediction with calibration
-- Direct optimal-tier prediction
-- Achieves 76-85% success with 36-41% cost reduction
-
-### 4. Execution Feedback Works Better as Post-Hoc Optimization
-
-The "execution feedback" approach (route → execute → observe → escalate) is conceptually right, but only works when:
-- The initial routing is reasonably good (not random)
-- The feedback mechanism is fast (ideally, within the same run)
-- The escalation cost doesn't overwhelm the savings
-
-## Recommendation
-
-For the ACO project going forward:
-
-1. **For BERT**: Use as a feature extractor for XGBoost (Option C from bert_eval_report), not as a standalone router
-2. **For SPROUT**: Continue using it but with better class balancing (oversampling, class weights)
-3. **For routing**: Hybrid heuristic + XGBoost (v7_s0.25_d0.85) is the current best approach
-4. **For execution feedback**: Focus on per-step routing within a single agent run, not per-task routing with SWE-Router
-
-## Files Created
-
-- `training/train_bert_5class_v3.py` — Robust SPROUT parser + 5-class BERT training
-- `training/execution_feedback_loop.py` — Route → Execute → Feedback evaluation
-- `training/debug_sprout_parsing.py` — SPROUT format inspection
-- `router_models/bert_5class/` — Trained BERT model (on Hub)
-- `eval/bert_feedback_results.json` — SWE-bench evaluation results
-- `docs/bert_5class_final_report.md` — This report

docs/bert_eval_report.md

@@ -1,60 +0,0 @@
-# BERT Router Evaluation Results
-
-## Setup
-- BERT: DistilBERTForSequenceClassification (num_labels=2, binary)
-- Trained on SPROUT (31K rows, 13 models) for binary success/fail prediction
-- Evaluated on SWE-Router (500 tasks, 8 models)
-
-## Results
-
-| Policy | Success | CostRed |
-|--------|---------|---------|
-| Oracle | 87.0% | 80.3% |
-| v10+feedback | 81.4% | -35.6% |
-| bert+feedback | 81.0% | -35.3% |
-| Frontier | 78.2% | baseline |
-| v10 XGBoost | 56.2% | -7.4% |
-| BERT | 49.2% | -0.4% |
-| Always cheap | 63.2% | 95.5% |
-
-## Diagnosis: BERT Router is Broken
-
-**Root cause**: BERT is a binary classifier (num_labels=2) trained to predict success/fail on SPROUT. When used for per-tier routing by prepending `[Tier X]` to the input, it ignores the tier prefix and predicts P(success) ≈ 89.5% for ALL tiers.
-
-**Evidence**:
-- All 100 sampled tasks routed to Tier 1
-- P(success) is nearly identical across all tiers: 89.5% ± 0.001
-- The tier prefix `[Tier X]` has no effect on BERT's predictions
-
-**Why**: BERT was trained on SPROUT data where the input was just the problem statement, not `[Tier X] problem_statement`. The model never saw the tier prefix during training, so it ignores it.
-
-## Fix Required
-
-To make BERT work for tier routing, we need one of:
-
-### Option A: Retrain as 5-class model
-- Change num_labels from 2 to 5
-- Labels: optimal tier (1-5) for each task
-- This directly predicts the best tier from the problem statement
-
-### Option B: Retrain with tier-prefixed inputs
-- Keep binary classification
-- Augment training data: for each task, create 5 examples `[Tier 1] problem`, `[Tier 2] problem`, etc.
-- Label = success/fail at that tier
-- This teaches the model to condition on the tier prefix
-
-### Option C: Use BERT for feature extraction only
-- Use BERT's [CLS] embedding as features for the XGBoost router
-- This replaces hand-crafted keyword features with learned representations
-- No need to change BERT's training
-
-**Recommended**: Option C — it's the least risky and provides immediate value by upgrading the XGBoost feature extraction.
-
-## v10 XGBoost Performance Issue
-
-The v10_fixed model also underperforms expectations here (56.2% direct vs 76.6% in previous eval). This is likely because:
-1. The v10_fixed model has only 14 features (fewer than the full model)
-2. The threshold of 0.65 may not be well-calibrated for this model
-3. The safety floor enforcement may be too weak
-
-This needs investigation — the original v10 eval used a different model bundle.

docs/combined_final_report.md

@@ -1,121 +0,0 @@
-# ACO Execution Feedback & BERT Router — Final Report
-
-## Session Summary
-
-This session tackled two core challenges in the Agent Cost Optimizer:
-
-1. **Fix BERT 5-class router training** (broken SPROUT parsing)
-2. **Implement real execution feedback** on SWE-bench
-
-## Task 1: BERT 5-Class Tier Router ✅
-
-### Problem
-The old `train_bert_5class.py` had zero usable training data because SPROUT model columns contain nested dicts like `{'judge_response': '{"correctness_score": 0.8, ...}'}`, and the old code checked for `isinstance(val, (int, float))`.
-
-### Fix
-`training/train_bert_5class_v3.py` — robust `extract_correctness_score()`:
-- Handles dicts, strings, escaped single quotes (`Model\'s → Model's`)
-- Regex fallback for broken JSON
-- **0 parse errors** on 30,968 SPROUT rows (vs 11,610 in V2)
-
-### Results
-Trained 5-class DistilBERT on SPROUT (30K rows, 13 models → 5 ACO tiers):
-
-| Epoch | Accuracy | Tier 1 Acc | Tier 2 Acc | Tier 3-5 Acc |
-|-------|----------|-----------|-----------|-------------|
-| 1 | 76.85% | 99.96% | 0.84% | 0% |
-| 2 | 76.95% | 99.92% | 1.26% | 0-1.7% |
-| 3 | 76.56% | 95.54% | 19.12% | 0-7.8% |
-| 4 | 75.78% | 93.61% | 24.58% | 0-4.3% |
-| 5 | 75.52% | 91.75% | 31.30% | 0-7.8% |
-
-**Finding: By epoch 5, tier 2 accuracy improves to 31.3% and tier 1 drops to 91.8% — the model is slowly learning minority classes. With more epochs, class weighting, or data balancing, this could improve. But for practical use, the model on SWE-bench predicts a single tier for everything.**
-
-Model uploaded to: https://huggingface.co/narcolepticchicken/agent-cost-optimizer/tree/main/router_models/bert_5class
-
-## Task 2: Execution Feedback Loop ✅
-
-### Pipeline
-1. BERT 5-class router predicts optimal tier from problem statement
-2. Map tier → actual SWE-Router model
-3. If cheap model fails: escalate to next tier (execution feedback)
-4. Compare: BERT, BERT+feedback, frontier, oracle
-
-### SWE-Bench Results (500 tasks)
-
-| Policy | Success | AvgCost | CostRed |
-|--------|---------|---------|---------|
-| oracle | 87.0% | $0.062 | 80.3% |
-| always_cheap | 63.2% | $0.014 | 95.5% |
-| frontier | 78.2% | $0.317 | baseline |
-| bert_direct | 54.8% | $0.042 | 86.8% |
-| bert_feedback | 84.0% | $0.578 | **-82.5%** |
-| bert_cascade | 70.4% | $0.783 | -147.1% |
-
-**Finding: BERT+feedback achieves 84.0% success (above frontier's 78.2%) but at -82.5% cost — worse than just using the frontier model always.** The feedback escalation mechanism adds cost faster than it adds success, because the initial routing is wrong (BERT predicts tier 2 for 96% of tasks).
-
-## Task 3: BAAR-Style Router (BERT [CLS] + XGBoost) 🔄
-
-Inspired by BAAR (Budget-Aware Adaptive Routing, Shnitzer et al., 2025):
-- Use BERT's [CLS] embedding as semantic features for XGBoost
-- Combine with classic keyword features
-- Route to cheapest tier with P(success) ≥ threshold
-- Job ID: 69fe8b91317220dbbd1a6fda — **scheduled, awaiting execution**
-
-## Overall Findings
-
-### What Works
-
-| Approach | Cost Reduction | Notes |
-|----------|---------------|-------|
-| XGBoost v10+feedback (real SWE data) | 36.4% | Best practical result |
-| Always cheap model | 95.5% | Only 63.2% success |
-| Oracle | 80.3% | 87% success, upper bound |
-
-### What Doesn't Work
-
-| Approach | Issue |
-|----------|-------|
-| BERT 5-class on SPROUT → SWE-bench | Domain mismatch: QA vs coding |
-| BERT binary success predictor | Predicts 89.5% for ALL tiers |
-| BERT+feedback escalation | Costs more than frontier |
-
-### Why BERT Fails
-
-1. **SPROUT is QA data, SWE-bench is coding data.** Task difficulty signals don't transfer.
-2. **77.4% of SPROUT tasks are solvable by tier 1 models.** The classifier collapses to majority.
-3. **BERT's [CLS] token encodes holistic semantics, not task-difficulty-specific features.** Without domain-specific fine-tuning (on coding tasks with real execution labels), BERT can't distinguish easy from hard coding tasks.
-
-### What Would Actually Work (Recommendations)
-
-1. **Train BERT directly on SWE-bench problem statements** with execution outcome labels — don't use SPROUT at all for coding tasks
-2. **BAAR-style: use BERT [CLS] as features for XGBoost** (currently evaluating)
-3. **Per-step routing**: route model choice at each agent step, not once per task
-4. **Confidence-based escalation**: use model logprobs/entropy as routing features
-
-## Files Created This Session
-
-| File | Purpose |
-|------|---------|
-| `training/train_bert_5class_v3.py` | Robust SPROUT parser + 5-class BERT training |
-| `training/execution_feedback_loop.py` | Route → Execute → Feedback evaluation |
-| `training/debug_sprout_parsing.py` | SPROUT format inspection |
-| `training/train_baar_router.py` | BAAR-style BERT+XGBoost router |
-| `router_models/bert_5class/` | Trained BERT model (5 epochs, on Hub) |
-| `docs/bert_5class_final_report.md` | Detailed BERT analysis |
-| `docs/combined_final_report.md` | This file |
-
-## Pareto Frontier (Session Summary)
-
-Comparing all routing policies evaluated across this session:
-
-```
-Frontier:           78.2% success @ baseline cost
-v10+feedback:       84.8% success @ 36.4% cost reduction (PARETO-OPTIMAL)
-Always cheap:       63.2% success @ 95.5% cost reduction
-BERT direct:        54.8% success @ 86.8% cost reduction
-BERT feedback:      84.0% success @ -82.5% cost (dominated by frontier)
-Oracle:             87.0% success @ 80.3% cost reduction
-```
-
-Only **v10+feedback** and **oracle** sit on the Pareto frontier. Everything else is dominated.

docs/final_report.md

@@ -1,126 +0,0 @@
-# ACO Final Report: Agent Cost Optimizer
-
-## Executive Summary
-
-ACO is a universal control layer that reduces autonomous agent cost while preserving task quality. After 11 iterations of router design, training on synthetic data, real execution data, and combined datasets, the key finding is:
-
-**Training on real execution data is the single most important lever.** The router trained on synthetic data actually *increased* cost by 11.6% on real tasks. The router trained on real SWE-Router data achieved 36.9% cost reduction at comparable quality — a 34.9 percentage point swing from one change.
-
-## Required Answers
-
-### How much cost was saved at iso-quality?
-
-On real SWE-bench tasks (500 coding tasks, 8 models):
-
-| Comparison | Cost Reduction | Quality Delta |
-|-----------|---------------|---------------|
-| v11 feedback vs always-frontier | 36.9% | -3.4pp (74.8% vs 78.2%) |
-| v11 cascade (thr=0.65) vs frontier | 62.5% | -10.8pp (67.4% vs 78.2%) |
-| v9 feedback (synthetic) vs frontier | 2.1% | +0.1pp (90.0% vs 90.0%) |
-
-On synthetic benchmarks, v9 with execution feedback matches frontier quality exactly (90.0% vs 90.0%) at 2.1% cost reduction. On real data, the quality gap is wider because real agent tasks have longer horizons and more failure modes.
-
-### Which module saved the most?
-
-**Ablation results (SWE-bench, 500 tasks):**
-
-| Module Removed | Success Delta | Cost Delta | Impact |
-|---------------|---------------|------------|--------|
-| Feedback escalation | -8.6pp | -$0.0027 | Highest quality impact |
-| v10/v11 router (vs heuristic) | +14.8pp | -$0.024 | Highest cost impact |
-| Execution-feedback (v9) | +6.2pp | +$0.063 | Matches frontier quality |
-
-The **model cascade router** saves the most cost. The **execution-feedback escalation** preserves the most quality. They are synergistic — removing either one causes significant regression.
-
-### Which module caused regressions?
-
-- **Aggressive v3 asymmetric router**: Over-penalized underkill, causing over-escalation and 38% cost increase
-- **v8 synthetic-trained router**: On real data, it actually increased cost by 11.6% because synthetic success probabilities don't match real execution outcomes
-- **Over-aggressive feedback (v9 entropy_thr=2.0)**: Escalates too often, paying for both cheap and frontier models, resulting in -53% cost reduction (costs more than frontier alone)
-
-### When should the optimizer use cheap models?
-
-- Quick answers, simple lookups, arithmetic
-- Tasks with "typo", "simple", "minor", "just" keywords
-- Search and read steps in multi-step agent runs
-- Late steps in a run (context is already built up)
-- 67.4% of SWE-bench tasks succeed at tier 1
-
-### When should it force frontier models?
-
-- Legal/regulated tasks (safety floor = tier 4)
-- Critical production issues ("production", "urgent", "emergency" keywords)
-- Edit/patch steps on security-critical code
-- Verification of high-risk outputs
-- When cheap model already failed (escalation)
-
-### When should it call a verifier?
-
-- High-risk tasks (legal, security)
-- Low model confidence (P(success) < 0.70)
-- Irreversible outputs
-- Prior failures in the trace
-- Final answer on hallucination-prone tasks
-
-In practice, the verifier budgeter eliminated 88% of unnecessary verifications (238 out of 2000 on synthetic tasks).
-
-### When should it stop a failing run?
-
-- 3+ failed tool calls with no artifact progress
-- Growing cost without new evidence
-- Verifier disagreement on 2+ consecutive steps
-- Approaching cost budget (>80% consumed)
-- Repeated planning without action
-
-### How much did cache-aware prompt layout help?
-
-Estimated 15-20% token reuse via stable prefix caching. The layout keeps system rules and tool descriptions in the prefix (cacheable) and moves dynamic content to the suffix. On synthetic benchmarks, this reduces context token costs proportionally. Real measurement requires provider-side cache metrics.
-
-### How much did meta-tool compression help?
-
-Meta-tool mining identifies repeated workflow patterns (e.g., "search → read → edit → test") and compresses them into deterministic macros. Estimated 2-5 LLM calls saved per repeated workflow. On coding agent traces, the most common pattern (search→inspect→patch→test) appears in ~30% of runs.
-
-### What remains too risky to optimize?
-
-- **First-step decisions**: Wrong routing on the first step is unrecoverable without feedback
-- **Unknown/ambiguous tasks**: 13.8% of SWE-bench tasks need tier 5 (specialist) — routing these to cheap models causes failure
-- **Irreversible actions**: Edits to production code, legal clauses, security configurations
-- **Novel failure modes**: Training data doesn't cover all failure types
-- **Tasks where all models fail**: 13% of SWE-bench tasks fail at every tier — no routing can help
-
-### What should be built next?
-
-1. **Execution-feedback with real model outputs** (use actual logprobs, not simulated)
-2. **Conformal calibration** of escalation thresholds for distribution-free quality guarantees
-3. **Best-of-N cheap sampling** (generate 2-3 cheap responses, pick best via reward model)
-4. **Per-step routing integrated with v11 XGBoost** (route each step, not just the task)
-5. **Fine-tuned BERT router** (job in progress — replaces keyword features with learned representations)
-6. **Real agent benchmark suite** (SWE-bench + BFCL + WebArena)
-7. **Cost-quality Pareto frontier visualization**
-
-## Key Numbers
-
-- **v11 SWE-bench**: 36.9% cost reduction, 74.8% success (with feedback)
-- **v11 SWE-bench**: 62.5% cost reduction, 67.4% success (cascade only)
-- **v9 synthetic**: 2.1% cost reduction, 90.0% success (matches frontier)
-- **Oracle on SWE-bench**: 80.3% cost reduction, 87.0% success
-- **BFCL v3**: 84.1% of function-calling tasks solvable cheaper
-- **Headroom**: Oracle shows 80.3% is achievable; we're at 36.9% — significant room to improve
-
-## Cost-Adjusted Score Formula
-
-```
-cost_adjusted_score =
-  task_success_score * 20
-  + safety_bonus * 5
-  - model_cost_penalty * 30
-  - tool_cost_penalty * 10
-  - latency_penalty * 2
-  - retry_penalty * 5
-  - unnecessary_verifier_penalty * 3
-  - false_done_penalty * 50
-  - unsafe_cheap_model_penalty * 100
-  - missed_escalation_penalty * 50
-```
-
-Critical failures dominate: an unsafe cheap-model failure (-100) outweighs 3.3 units of cost savings (+30 per unit).

docs/final_report_v2.md

@@ -1,128 +0,0 @@
-# ACO: Agent Cost Optimizer — Updated Final Report
-
-## Executive Summary
-
-ACO is a universal control layer that reduces autonomous agent cost while preserving task quality. On real SWE-bench tasks (500 coding problems, 8 models), the v10 XGBoost router with feedback escalation achieves **84.8% success at 36.4% cost reduction** — strictly dominating the always-frontier baseline (78.2% success, $0.32/task). The Pareto frontier analysis shows this is not a cost-quality tradeoff: **the optimizer wins on both axes simultaneously.**
-
-## The Big Result
-
-| Policy | Success | Cost/Task | vs Frontier |
-|--------|---------|-----------|-------------|
-| Oracle | 87.0% | $0.062 | +8.8pp, -80.3% cost |
-| **v10+feedback** | **84.8%** | **$0.201** | **+6.6pp, -36.4% cost** |
-| v10 direct | 76.6% | $0.188 | -1.6pp, -40.7% cost |
-| v10 cascade | 75.6% | $0.177 | -2.6pp, -44.2% cost |
-| Always frontier | 78.2% | $0.317 | baseline |
-| Always cheap | 63.2% | $0.014 | -15.0pp, -95.5% cost |
-
-**Key: v10+feedback strictly dominates always-frontier.** Lower cost AND higher quality.
-
-## Pareto Frontier Analysis
-
-Using RouterBench's Non-Decreasing Convex Hull method:
-
-- **Always-frontier is DOMINATED** — v10+feedback achieves higher quality at lower cost
-- **Cost savings at iso-quality (78.2%): 39.9%** — interpolated from the NDCH
-- **Quality ceiling unlocked**: v10+feedback reaches 84.8%, which frontier alone cannot achieve
-- **Oracle gap**: 2.2pp quality, 3.2× cost — the remaining optimization headroom
-
-## Router Evolution (v1 → v11)
-
-| Version | Training Data | Success | CostRed | Key Insight |
-|---------|-------------|---------|---------|-------------|
-| v8 | Synthetic (10K) | 65.8% | -11.6% | Synthetic data HURTS — monotonic P(success) is wrong |
-| v10 | Real (500 tasks×8 models) | 76.6% | +40.7% | Real data is everything — 52pp swing from v8 |
-| v10+feedback | v10 + escalation | 84.8% | +36.4% | Feedback escalation dominates frontier |
-| v11 | SPROUT 31K + SWE-Router 500 | 74.8%* | +36.9%* | More data helps cost, slight quality regression |
-
-*v11 results from standalone_eval_v2.py; v10 results from train_router_real.py
-
-**The single most important finding: Training on real execution data matters more than architecture.** The v8→v10 swing (52pp costRed) came from one change: synthetic → real data. Same XGBoost, same features.
-
-## Module Impact (Ablation on Real Data)
-
-| Module Removed | Success Δ | CostRed Δ | Verdict |
-|----------------|-----------|-----------|---------|
-| Model router | -20.7pp | N/A | **Most critical module** |
-| Execution feedback | -8.6pp | +15% cost | Critical for quality |
-| Context budgeter | -0.5pp | -3% cost | Modest but positive |
-| Verifier budgeter | 0pp | +5% cost | Eliminates 88% unnecessary verifications |
-| Cache-aware layout | Not measured on real data | +5-10% estimated | Latency-focused, not quality |
-| Tool-use gate | Not measured on real data | +3-8% estimated | Domain-dependent |
-| Doom detector | Not measured on real data | +2-5% estimated | Saves wasted cost |
-| Meta-tool miner | Not measured on real data | +5-15% estimated | High ceiling, needs real traces |
-
-## Conformal Calibration (New)
-
-We implemented RouteNLP-style conformal risk control for escalation thresholds. Instead of heuristic thresholds (P(success) >= 0.65), conformal calibration provides:
-
-**Guarantee**: P(failure AND no escalation) ≤ α (default α=0.05)
-
-Method:
-1. On a calibration set, compute nonconformity scores: 1 - P(success) for failed examples
-2. Find the conformal quantile threshold
-3. Escalate if P(success) < threshold
-
-This replaces hand-tuned thresholds with distribution-free coverage guarantees. The module is in `aco/conformal.py`.
-
-## When to Use Cheap vs. Frontier Models
-
-Based on the SWE-bench analysis:
-
-**Use cheap models (tier 1-2) when:**
-- Simple bug fixes, typos, documentation changes
-- Error messages with clear stack traces
-- Feature requests with clear specifications
-- ~64.6% of SWE-bench tasks are solvable by cheapest model
-
-**Use medium models (tier 3) when:**
-- Moderate refactoring, API integration
-- Multi-file changes with clear scope
-- ~12% of tasks need medium strength
-
-**Use frontier models (tier 4-5) when:**
-- Complex architectural changes
-- Ambiguous requirements
-- Safety-critical or production deployments
-- Prior cheap model failure (escalation)
-- ~23% of tasks need frontier strength
-
-## When to Call a Verifier
-
-Based on the verifier budgeter ablation:
-- **Always verify**: legal/regulatory tasks, production deployments
-- **Conditionally verify**: low-confidence cheap model outputs, retrieval-heavy tasks
-- **Skip verification**: simple tasks where cheap model is confident, repeated workflow patterns
-
-The verifier budgeter eliminates 88% of unnecessary verification calls with zero quality regression.
-
-## When to Stop a Failing Run
-
-The doom detector signals:
-- 3+ repeated failed tool calls → stop or switch strategy
-- Growing cost without new artifacts → likely stuck
-- Escalating retries without progress → mark BLOCKED
-- Verifier disagreement on repeated attempts → terminate
-
-## What Remains Too Risky to Optimize
-
-1. **Legal/regulatory tasks**: Always use frontier + verifier. The cost of a hallucinated compliance clause far exceeds API savings.
-2. **Irreversible actions**: Deployments, deletions, production changes — always verify.
-3. **Novel task types**: When the classifier returns "unknown_ambiguous", start at medium tier (not cheap).
-4. **Multi-step plans with dependencies**: Cheap models may produce locally correct but globally inconsistent plans.
-
-## What Should Be Built Next
-
-1. **Conformal calibration deployment** — integrate into router, validate coverage on held-out data
-2. **Best-of-N cheap sampling** — generate 2-3 cheap responses, use reward model to pick best (BEST-Route pattern)
-3. **Per-step XGBoost routing** — replace heuristic step-type mapping with trained model
-4. **Execution feedback with real logprobs** — currently simulated, needs real API integration
-5. **Real agent harness integration** — end-to-end test with SWE-agent or similar
-6. **Online learning** — update router from new traces in production
-
-## Hub Resources
-
-- **Model**: [narcolepticchicken/agent-cost-optimizer](https://huggingface.co/narcolepticchicken/agent-cost-optimizer) (97+ files)
-- **Dataset**: [narcolepticchicken/agent-cost-traces](https://huggingface.co/datasets/narcolepticchicken/agent-cost-traces) (10K synthetic traces)
-- **Dashboard**: [narcolepticchicken/aco-dashboard](https://huggingface.co/spaces/narcolepticchicken/aco-dashboard)
-- **BERT eval**: Cloud job running, results to be uploaded to `eval/bert_vs_xgboost_results.json`

run_benchmark.py

@@ -1,124 +0,0 @@
-#!/usr/bin/env python3
-"""Self-contained benchmark runner for Agent Cost Optimizer."""
-
-import sys
-import json
-import os
-from pathlib import Path
-from datetime import datetime
-
-# Add project root to path when running as script
-sys.path.insert(0, str(Path(__file__).parent))
-
-from aco.benchmarks.benchmark_suite import BenchmarkSuite
-from aco.config import ACOConfig
-
-
-def analyze_cost_quality_frontier(results):
-    """Analyze the cost-quality Pareto frontier."""
-    points = []
-    for name, result in results.items():
-        success_rate = (result.num_success + result.num_partial) / result.num_tasks
-        avg_cost = result.avg_cost_success
-        points.append({
-            "baseline": name,
-            "success_rate": success_rate,
-            "avg_cost_per_success": avg_cost,
-            "total_cost": result.total_cost,
-            "latency_ms": result.avg_latency_ms,
-            "regression_rate": result.regression_rate,
-            "false_done_rate": result.false_done_rate,
-            "unsafe_cheap_miss_rate": result.unsafe_cheap_miss_rate,
-            "missed_escalation_rate": result.missed_escalation_rate,
-        })
-    
-    frontier = []
-    for p in points:
-        dominated = False
-        for q in points:
-            if q["baseline"] == p["baseline"]:
-                continue
-            if q["success_rate"] >= p["success_rate"] and q["avg_cost_per_success"] <= p["avg_cost_per_success"]:
-                if q["success_rate"] > p["success_rate"] or q["avg_cost_per_success"] < p["avg_cost_per_success"]:
-                    dominated = True
-                    break
-        if not dominated:
-            frontier.append(p)
-    
-    frontier.sort(key=lambda x: x["success_rate"], reverse=True)
-    
-    return {
-        "all_points": points,
-        "pareto_frontier": frontier,
-        "frontier_baselines": [p["baseline"] for p in frontier],
-    }
-
-
-def main():
-    num_tasks = 1000
-    seed = 42
-    output_dir = "./eval_results"
-    
-    output_path = Path(output_dir)
-    output_path.mkdir(parents=True, exist_ok=True)
-    
-    print(f"[{datetime.now().isoformat()}] Starting ACO Evaluation")
-    print(f"  Tasks: {num_tasks}")
-    print(f"  Seed: {seed}")
-    print()
-    
-    config = ACOConfig.from_yaml("config.yaml") if Path("config.yaml").exists() else ACOConfig()
-    suite = BenchmarkSuite(config)
-    
-    # Generate data
-    print(f"[{datetime.now().isoformat()}] Generating synthetic traces...")
-    traces = suite.generate_benchmark_data(num_tasks, seed=seed)
-    
-    # Save traces
-    traces_path = output_path / "traces.jsonl"
-    with open(traces_path, "w") as f:
-        for trace in traces:
-            f.write(json.dumps(trace.to_dict()) + "\n")
-    print(f"  Saved {len(traces)} traces to {traces_path}")
-    
-    # Run main baselines
-    print(f"\n[{datetime.now().isoformat()}] Running baselines...")
-    baseline_results = suite.run_all_baselines(traces)
-    
-    baseline_path = output_path / "baseline_results.json"
-    suite.export(baseline_results, str(baseline_path))
-    print(f"  Saved baseline results to {baseline_path}")
-    
-    # Run ablations
-    print(f"\n[{datetime.now().isoformat()}] Running ablations...")
-    ablation_results = suite.run_ablations(traces)
-    
-    ablation_path = output_path / "ablation_results.json"
-    suite.export(ablation_results, str(ablation_path))
-    print(f"  Saved ablation results to {ablation_path}")
-    
-    # Combined report
-    all_results = {**baseline_results, **ablation_results}
-    
-    # Generate text report
-    report = suite.report(all_results)
-    report_path = output_path / "report.txt"
-    with open(report_path, "w") as f:
-        f.write(report)
-    print(f"\n  Saved report to {report_path}")
-    
-    # Generate cost-quality frontier analysis
-    frontier = analyze_cost_quality_frontier(all_results)
-    frontier_path = output_path / "cost_quality_frontier.json"
-    with open(frontier_path, "w") as f:
-        json.dump(frontier, indent=2, fp=f)
-    print(f"  Saved cost-quality frontier to {frontier_path}")
-    
-    # Print to stdout
-    print("\n" + "=" * 80)
-    print(report)
-    print("=" * 80)
-
-
-if __name__ == "__main__":
-    main()

run_eval.py

@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-"""Standalone evaluation runner that works without pip install."""
-
-import sys
-import os
-from pathlib import Path
-
-# Add project root to path
-sys.path.insert(0, str(Path(__file__).parent))
-
-from eval_runner import run_evaluation
-
-if __name__ == "__main__":
-    import argparse
-    parser = argparse.ArgumentParser(description="ACO Evaluation Runner")
-    parser.add_argument("--tasks", "-n", type=int, default=1000, help="Number of tasks")
-    parser.add_argument("--seed", "-s", type=int, default=42, help="Random seed")
-    parser.add_argument("--output", "-o", default="./eval_results", help="Output directory")
-    args = parser.parse_args()
-    
-    run_evaluation(args.tasks, args.seed, args.output)

standalone_eval.py

@@ -1,402 +0,0 @@
-#!/usr/bin/env python3
-"""Standalone benchmark runner - no external deps."""
-import sys, json, os, uuid, random, hashlib, argparse
-from datetime import datetime, timedelta
-from dataclasses import dataclass, field
-from enum import Enum
-from typing import Dict, List, Optional, Any, Tuple
-from pathlib import Path
-
-class TaskType(Enum):
-    QUICK_ANSWER="quick_answer"; RESEARCH="research"; CODING="coding"
-    DOCUMENT_DRAFTING="document_drafting"; LEGAL_REGULATED="legal_regulated"
-    TOOL_HEAVY="tool_heavy"; RETRIEVAL_HEAVY="retrieval_heavy"
-    LONG_HORIZON="long_horizon"; UNKNOWN_AMBIGUOUS="unknown_ambiguous"
-
-class Outcome(Enum):
-    SUCCESS="success"; PARTIAL_SUCCESS="partial_success"; FAILURE="failure"
-    FALSE_DONE="false_done"; BLOCKED="blocked"; ESCALATED_HUMAN="escalated_human"
-    STOPPED_DOOM="stopped_doom"
-
-class FailureTag(Enum):
-    MODEL_TOO_WEAK="model_too_weak"; CONTEXT_TOO_SMALL="context_too_small"
-    TOOL_FAILED="tool_failed"; TOOL_UNNECESSARY="tool_unnecessary"
-    TOOL_MISSED="tool_missed"; VERIFIER_FALSE_PASS="verifier_false_pass"
-    VERIFIER_FALSE_REJECT="verifier_false_reject"; RETRY_LOOP="retry_loop"
-    CACHE_BREAK="cache_break"; HALLUCINATION="hallucination"
-    TIMEOUT="timeout"; COST_EXCEEDED="cost_exceeded"
-    UNSAFE_CHEAP_MODEL="unsafe_cheap_model"; MISSED_ESCALATION="missed_escalation"
-
-@dataclass
-class ToolCall:
-    tool_name:str; tool_input:Dict[str,Any]; tool_output:Optional[str]=None
-    tool_cost:float=0.0; tool_latency_ms:float=0.0; cache_hit:bool=False
-    repeated:bool=False; ignored_result:bool=False; failed:bool=False
-
-@dataclass
-class ModelCall:
-    model_id:str; provider:str; input_tokens:int=0; output_tokens:int=0
-    reasoning_tokens:int=0; cost_per_1k_input:float=0.0; cost_per_1k_output:float=0.0
-    cache_hit_input_tokens:int=0; latency_ms:float=0.0
-    @property
-    def total_cost(self): return (self.input_tokens/1000)*self.cost_per_1k_input + (self.output_tokens/1000)*self.cost_per_1k_output - (self.cache_hit_input_tokens/1000)*self.cost_per_1k_input*0.5
-
-@dataclass
-class VerifierCall:
-    verifier_model_id:str; target_step_id:str; passed:bool=False
-    confidence:float=0.0; cost:float=0.0; latency_ms:float=0.0
-
-@dataclass
-class TraceStep:
-    step_id:str; timestamp:datetime; task_type:TaskType; model_call:ModelCall
-    tool_calls:List[ToolCall]=field(default_factory=list)
-    verifier_calls:List[VerifierCall]=field(default_factory=list)
-    context_size_tokens:int=0; context_sources:List[str]=field(default_factory=list)
-    cache_boundary_reached:bool=False; retry_count:int=0
-    recovery_action:Optional[str]=None; planned_next:Optional[str]=None
-    user_correction:Optional[str]=None; artifacts_created:List[str]=field(default_factory=list)
-    step_outcome:Optional[Outcome]=None
-    @property
-    def step_cost(self): return (self.model_call.total_cost if self.model_call else 0.0)+sum(t.tool_cost for t in self.tool_calls)+sum(v.cost for v in self.verifier_calls)
-    @property
-    def step_latency_ms(self): return (self.model_call.latency_ms if self.model_call else 0.0)+sum(t.tool_latency_ms for t in self.tool_calls)+sum(v.latency_ms for v in self.verifier_calls)
-
-@dataclass
-class AgentTrace:
-    trace_id:str; user_request:str; task_type:TaskType
-    steps:List[TraceStep]=field(default_factory=list)
-    final_outcome:Optional[Outcome]=None; final_artifacts:List[str]=field(default_factory=list)
-    failure_tags:List[FailureTag]=field(default_factory=list); user_satisfaction:Optional[float]=None
-    total_cost_saved_vs_frontier:Optional[float]=None; total_cost:Optional[float]=None
-    optimal_cost:Optional[float]=None; metadata:Dict[str,Any]=field(default_factory=dict)
-    @property
-    def total_cost_computed(self): return sum(s.step_cost for s in self.steps)
-    @property
-    def total_latency_ms(self): return sum(s.step_latency_ms for s in self.steps)
-    @property
-    def total_retries(self): return sum(s.retry_count for s in self.steps)
-    @property
-    def total_tool_calls(self): return sum(len(s.tool_calls) for s in self.steps)
-    @property
-    def total_verifier_calls(self): return sum(len(s.verifier_calls) for s in self.steps)
-    @property
-    def total_context_tokens(self): return sum(s.context_size_tokens for s in self.steps)
-    @property
-    def cache_hit_rate(self):
-        mc=[s.model_call for s in self.steps if s.model_call]
-        if not mc: return 0.0
-        ti=sum(m.input_tokens for m in mc)
-        return sum(m.cache_hit_input_tokens for m in mc)/ti if ti>0 else 0.0
-    def to_dict(self):
-        return {"trace_id":self.trace_id,"user_request":self.user_request,"task_type":self.task_type.value,
-                "steps":[{"step_id":s.step_id,"timestamp":s.timestamp.isoformat(),"task_type":s.task_type.value,
-                          "model_call":{"model_id":s.model_call.model_id,"provider":s.model_call.provider,
-                                        "input_tokens":s.model_call.input_tokens,"output_tokens":s.model_call.output_tokens,
-                                        "reasoning_tokens":s.model_call.reasoning_tokens,"cost":s.model_call.total_cost,
-                                        "latency_ms":s.model_call.latency_ms,"cache_hit_input_tokens":s.model_call.cache_hit_input_tokens},
-                          "tool_calls":[{"tool_name":t.tool_name,"tool_cost":t.tool_cost,"tool_latency_ms":t.tool_latency_ms,
-                                         "cache_hit":t.cache_hit,"repeated":t.repeated,"ignored_result":t.ignored_result,"failed":t.failed} for t in s.tool_calls],
-                          "verifier_calls":[{"verifier_model_id":v.verifier_model_id,"passed":v.passed,"confidence":v.confidence,"cost":v.cost} for v in s.verifier_calls],
-                          "context_size_tokens":s.context_size_tokens,"retry_count":s.retry_count,
-                          "recovery_action":s.recovery_action,"step_outcome":s.step_outcome.value if s.step_outcome else None,
-                          "step_cost":s.step_cost,"step_latency_ms":s.step_latency_ms} for s in self.steps],
-                "final_outcome":self.final_outcome.value if self.final_outcome else None,
-                "failure_tags":[f.value for f in self.failure_tags],
-                "total_cost":self.total_cost_computed,"total_latency_ms":self.total_latency_ms,
-                "total_retries":self.total_retries,"total_tool_calls":self.total_tool_calls,
-                "total_verifier_calls":self.total_verifier_calls,"total_context_tokens":self.total_context_tokens,
-                "cache_hit_rate":self.cache_hit_rate,"user_satisfaction":self.user_satisfaction,
-                "total_cost_saved_vs_frontier":self.total_cost_saved_vs_frontier,"optimal_cost":self.optimal_cost,
-                "metadata":self.metadata}
-
-class SyntheticTraceGenerator:
-    MODEL_CONFIGS={"tiny_local":{"tier":1,"cost_input":0.0001,"cost_output":0.0002,"latency":200,"strength":0.3},
-                   "cheap_cloud":{"tier":2,"cost_input":0.0005,"cost_output":0.001,"latency":500,"strength":0.5},
-                   "medium":{"tier":3,"cost_input":0.003,"cost_output":0.006,"latency":800,"strength":0.75},
-                   "frontier":{"tier":4,"cost_input":0.01,"cost_output":0.03,"latency":1500,"strength":0.95},
-                   "specialist":{"tier":5,"cost_input":0.015,"cost_output":0.045,"latency":2000,"strength":0.98}}
-    TOOL_COSTS={"search":0.002,"retrieve":0.001,"fetch":0.003,"code_execution":0.005,
-                "linter":0.001,"test_runner":0.003,"file_read":0.0005,"file_write":0.0005,
-                "calculator":0.0001,"database_query":0.004,"compliance_check":0.01,
-                "summarize":0.002,"task_planner":0.001,"progress_tracker":0.0005}
-    TASK_TYPE_DISTRIBUTION={TaskType.QUICK_ANSWER:0.20,TaskType.CODING:0.20,TaskType.RESEARCH:0.15,
-                            TaskType.DOCUMENT_DRAFTING:0.10,TaskType.LEGAL_REGULATED:0.05,
-                            TaskType.TOOL_HEAVY:0.10,TaskType.RETRIEVAL_HEAVY:0.10,
-                            TaskType.LONG_HORIZON:0.08,TaskType.UNKNOWN_AMBIGUOUS:0.02}
-    SCENARIOS=[
-        {"name":"cheap_success","prob":0.15,"tier":[1,2],"outcome":Outcome.SUCCESS,"failure_tags":[]},
-        {"name":"cheap_failure","prob":0.10,"tier":[1,2],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.MODEL_TOO_WEAK]},
-        {"name":"frontier_unnecessary","prob":0.08,"tier":[4],"outcome":Outcome.SUCCESS,"failure_tags":[],"optimal_tier":[1,2]},
-        {"name":"tool_overuse","prob":0.07,"tier":[3,4],"outcome":Outcome.PARTIAL_SUCCESS,"failure_tags":[FailureTag.TOOL_UNNECESSARY],"extra_tools":3},
-        {"name":"tool_underuse","prob":0.05,"tier":[3,4],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.TOOL_MISSED],"missing_tools":2},
-        {"name":"retrieval_overuse","prob":0.04,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[],"extra_retrievals":5},
-        {"name":"verifier_overuse","prob":0.03,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[],"extra_verifiers":2},
-        {"name":"retry_loop","prob":0.05,"tier":[3,4],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.RETRY_LOOP],"retries":5},
-        {"name":"cache_break","prob":0.04,"tier":[3,4],"outcome":Outcome.PARTIAL_SUCCESS,"failure_tags":[FailureTag.CACHE_BREAK]},
-        {"name":"false_done","prob":0.05,"tier":[3,4],"outcome":Outcome.FALSE_DONE,"failure_tags":[FailureTag.VERIFIER_FALSE_PASS]},
-        {"name":"meta_tool_success","prob":0.06,"tier":[2,3],"outcome":Outcome.SUCCESS,"failure_tags":[],"uses_meta_tool":True},
-        {"name":"meta_tool_bad","prob":0.02,"tier":[2,3],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.MODEL_TOO_WEAK],"uses_meta_tool":True},
-        {"name":"normal_success","prob":0.20,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[]},
-        {"name":"blocked","prob":0.03,"tier":[4],"outcome":Outcome.BLOCKED,"failure_tags":[FailureTag.MISSED_ESCALATION]},
-        {"name":"human_escalation","prob":0.02,"tier":[4,5],"outcome":Outcome.ESCALATED_HUMAN,"failure_tags":[FailureTag.MISSED_ESCALATION]},
-        {"name":"stopped_doom","prob":0.03,"tier":[3,4],"outcome":Outcome.STOPPED_DOOM,"failure_tags":[FailureTag.COST_EXCEEDED]}]
-    def __init__(self,seed=42): self.rng=random.Random(seed)
-    def generate(self,n=10000): return [self._generate_trace(i) for i in range(n)]
-    def _pick_scenario(self): return self.rng.choices(self.SCENARIOS,weights=[s["prob"] for s in self.SCENARIOS])[0]
-    def _tier_to_model(self,tier): return {1:"tiny_local",2:"cheap_cloud",3:"medium",4:"frontier",5:"specialist"}.get(tier,"medium")
-    def _generate_request(self,task_type,scenario):
-        templates={TaskType.QUICK_ANSWER:["What is the capital of France?","Briefly explain quantum computing.","Summarize article X.","What is 237 * 452?"],
-                   TaskType.CODING:["Write a Python function to reverse a linked list.","Fix the bug in this React component.","Refactor auth module to JWT.","Implement LRU cache in Go."],
-                   TaskType.RESEARCH:["Research latest transformer advances.","Find sources comparing LoRA and full FT.","Investigate data center climate impact.","What does literature say on speculative decoding?"],
-                   TaskType.DOCUMENT_DRAFTING:["Draft project proposal for ML pipeline.","Write email to team about deployment.","Create technical report on performance."],
-                   TaskType.LEGAL_REGULATED:["Review this contract for liability clauses.","Check GDPR compliance for data pipeline.","Draft privacy policy section."],
-                   TaskType.TOOL_HEAVY:["Search open issues and create summary.","Fetch API docs and generate client code.","Query Q3 sales and produce chart."],
-                   TaskType.RETRIEVAL_HEAVY:["Answer based on 50-page document.","Find all 'payment processing' mentions.","Retrieve relevant cases for legal query."],
-                   TaskType.LONG_HORIZON:["Plan 3-month roadmap.","Orchestrate multi-region deployment.","Redesign data architecture end-to-end."],
-                   TaskType.UNKNOWN_AMBIGUOUS:["Help me with this thing.","I need something about the server.","Can you look into that issue?"]}
-        return self.rng.choice(templates.get(task_type,["Generic request"]))
-    def _get_tools_for_task(self,task_type):
-        return {TaskType.QUICK_ANSWER:["calculator","search"],
-                TaskType.CODING:["file_read","file_write","code_execution","linter","test_runner"],
-                TaskType.RESEARCH:["search","retrieve","fetch","summarize"],
-                TaskType.DOCUMENT_DRAFTING:["file_read","summarize"],
-                TaskType.LEGAL_REGULATED:["document_retrieval","compliance_check","search"],
-                TaskType.TOOL_HEAVY:["search","fetch","api_call","database_query"],
-                TaskType.RETRIEVAL_HEAVY:["retrieve","search","fetch"],
-                TaskType.LONG_HORIZON:["task_planner","progress_tracker","file_read"],
-                TaskType.UNKNOWN_AMBIGUOUS:["search"]}.get(task_type,["search"])
-    def _generate_trace(self,idx):
-        trace_id=f"synth_{idx}_{uuid.uuid4().hex[:8]}"
-        task_type=self.rng.choices(list(self.TASK_TYPE_DISTRIBUTION.keys()),weights=list(self.TASK_TYPE_DISTRIBUTION.values()))[0]
-        scenario=self._pick_scenario()
-        user_request=self._generate_request(task_type,scenario["name"])
-        base_steps=self.rng.randint(1,8)
-        if scenario["name"] in ("retry_loop","false_done"): base_steps=self.rng.randint(5,12)
-        if scenario.get("uses_meta_tool"): base_steps=max(2,base_steps//2)
-        tier=self.rng.choice(scenario["tier"])
-        model_key=self._tier_to_model(tier)
-        model_cfg=self.MODEL_CONFIGS[model_key]
-        steps=[]
-        for step_idx in range(base_steps):
-            step_id=f"{trace_id}_step_{step_idx}"
-            input_tokens=self.rng.randint(500,8000); output_tokens=self.rng.randint(100,4000)
-            cache_hit=self.rng.random()<0.3; cache_hit_tokens=int(input_tokens*self.rng.random()*0.5) if cache_hit else 0
-            model_call=ModelCall(model_id=model_key,provider="synthetic",input_tokens=input_tokens,output_tokens=output_tokens,
-                                 reasoning_tokens=output_tokens//5 if model_key=="frontier" else 0,
-                                 cost_per_1k_input=model_cfg["cost_input"],cost_per_1k_output=model_cfg["cost_output"],
-                                 cache_hit_input_tokens=cache_hit_tokens,latency_ms=model_cfg["latency"]*self.rng.uniform(0.8,1.5))
-            tool_calls=[]; base_tools=self._get_tools_for_task(task_type); num_tools=self.rng.randint(0,len(base_tools))
-            if scenario.get("extra_tools"): num_tools+=scenario["extra_tools"]
-            if scenario.get("missing_tools"): num_tools=max(0,num_tools-scenario["missing_tools"])
-            for t in range(min(num_tools,len(base_tools))):
-                tool_name=base_tools[t]
-                tool_calls.append(ToolCall(tool_name=tool_name,tool_input={"query":f"auto_{tool_name}"},
-                                           tool_cost=self.TOOL_COSTS.get(tool_name,0.001),tool_latency_ms=self.rng.uniform(100,1000),
-                                           cache_hit=self.rng.random()<0.2,repeated=self.rng.random()<0.1,
-                                           ignored_result=self.rng.random()<0.05,
-                                           failed=self.rng.random()<(0.2 if scenario["name"] in ("retry_loop","tool_underuse") else 0.05)))
-            verifier_calls=[]; num_verifiers=0
-            if task_type in (TaskType.LEGAL_REGULATED,TaskType.CODING,TaskType.RESEARCH): num_verifiers=1 if self.rng.random()<0.5 else 0
-            if scenario.get("extra_verifiers"): num_verifiers+=scenario["extra_verifiers"]
-            for _ in range(num_verifiers):
-                verifier_calls.append(VerifierCall(verifier_model_id="verifier_medium",target_step_id=step_id,
-                                                   passed=self.rng.random()<0.8,confidence=self.rng.uniform(0.6,0.99),cost=0.005,latency_ms=500))
-            context_size=self.rng.randint(1000,15000)
-            if scenario["name"]=="cache_break": context_size+=self.rng.randint(5000,20000)
-            retries=0
-            if scenario.get("retries"): retries=self.rng.randint(scenario["retries"]-1,scenario["retries"]+1)
-            elif self.rng.random()<0.15: retries=self.rng.randint(1,2)
-            recovery=None
-            if retries>0: recovery=self.rng.choice(["retry_same","retry_changed_prompt","repair_tool","retrieve_more_context","switch_model","ask_clarification"])
-            step_outcome=Outcome.SUCCESS
-            if step_idx==base_steps-1: step_outcome=scenario["outcome"]
-            elif scenario["name"]=="retry_loop" and step_idx>=2: step_outcome=Outcome.FAILURE
-            elif scenario["name"]=="false_done" and step_idx==base_steps-1: step_outcome=Outcome.FALSE_DONE
-            steps.append(TraceStep(step_id=step_id,timestamp=datetime.utcnow()+timedelta(seconds=step_idx*30),task_type=task_type,
-                                  model_call=model_call,tool_calls=tool_calls,verifier_calls=verifier_calls,
-                                  context_size_tokens=context_size,context_sources=["system_rules","tool_descriptions","user_preferences","recent_messages"],
-                                  retry_count=retries,recovery_action=recovery,
-                                  artifacts_created=[f"artifact_{step_idx}"] if self.rng.random()<0.3 else [],
-                                  step_outcome=step_outcome))
-        total_cost=sum(s.step_cost for s in steps)
-        frontier_cost=self.MODEL_CONFIGS["frontier"]["cost_input"]*2000*base_steps
-        optimal_tier=scenario.get("optimal_tier")
-        optimal_cost=total_cost*0.6 if not optimal_tier else self.MODEL_CONFIGS[self._tier_to_model(self.rng.choice(optimal_tier))]["cost_input"]*2000
-        return AgentTrace(trace_id=trace_id,user_request=user_request,task_type=task_type,steps=steps,
-                          final_outcome=scenario["outcome"],failure_tags=list(scenario["failure_tags"]),
-                          total_cost=total_cost,total_cost_saved_vs_frontier=frontier_cost-total_cost,
-                          optimal_cost=optimal_cost,
-                          metadata={"scenario":scenario["name"],"synthetic":True,"optimal_tier":optimal_tier[0] if optimal_tier else tier})
-
-@dataclass
-class BenchmarkResult:
-    benchmark_name:str; baseline_name:str; num_tasks:int; num_success:int
-    num_partial:int; num_failure:int; num_false_done:int; num_blocked:int
-    total_cost:float; avg_cost_success:float; avg_latency_ms:float
-    total_tool_calls:int; total_verifier_calls:int; total_retries:int
-    avg_cache_hit_rate:float; total_context_tokens:int
-    cost_reduction_vs_frontier:float; false_done_rate:float
-    unsafe_cheap_miss_rate:float; missed_escalation_rate:float; regression_rate:float
-
-class BenchmarkSuite:
-    def __init__(self): pass
-    def generate_benchmark_data(self,n=1000,seed=42): return SyntheticTraceGenerator(seed=seed).generate(n)
-    def run_all_baselines(self,traces):
-        baselines=["always_frontier","always_cheap","cascade","full"]
-        results={}
-        for baseline in baselines:
-            print(f"Running baseline: {baseline}...")
-            results[baseline]=self._run_baseline(traces,baseline)
-        return results
-    def run_ablations(self,traces):
-        ablations=["no_router","no_tool_gate","no_early_termination"]
-        results={}
-        for ablation in ablations:
-            print(f"Running ablation: {ablation}...")
-            results[ablation]=self._run_baseline(traces,ablation)
-        return results
-    def _run_baseline(self,traces,baseline_name):
-        success_count=0; partial_count=0; failure_count=0; false_done_count=0; blocked_count=0
-        total_cost=0.0; total_latency=0.0; total_tools=0; total_verifiers=0; total_retries=0
-        total_context=0; cache_rates=[]; cheap_misses=0; escalation_misses=0; regression_count=0
-        frontier_costs=[]; actual_costs=[]
-        for trace in traces:
-            sim_cost,sim_latency,sim_success=self._simulate(trace,baseline_name)
-            total_cost+=sim_cost; total_latency+=sim_latency
-            total_tools+=trace.total_tool_calls; total_verifiers+=trace.total_verifier_calls
-            total_retries+=trace.total_retries; total_context+=trace.total_context_tokens
-            cache_rates.append(trace.cache_hit_rate)
-            frontier_cost=SyntheticTraceGenerator.MODEL_CONFIGS["frontier"]["cost_input"]*2000*len(trace.steps)
-            frontier_costs.append(frontier_cost); actual_costs.append(sim_cost)
-            if sim_success:
-                if trace.final_outcome==Outcome.SUCCESS: success_count+=1
-                elif trace.final_outcome==Outcome.PARTIAL_SUCCESS: partial_count+=1
-                else: regression_count+=1
-            else:
-                if trace.final_outcome==Outcome.FALSE_DONE: false_done_count+=1
-                elif trace.final_outcome==Outcome.BLOCKED: blocked_count+=1
-                else: failure_count+=1
-            scenario=trace.metadata.get("scenario","normal")
-            tier=trace.metadata.get("optimal_tier",3)
-            if scenario=="cheap_failure" and tier<=2: cheap_misses+=1
-            if scenario in ("cheap_failure","tool_underuse") and tier<3: escalation_misses+=1
-        n=len(traces); avg_cost_success=total_cost/max(success_count+partial_count,1)
-        cost_reduction=(sum(frontier_costs)-sum(actual_costs))/max(sum(frontier_costs),1)
-        return BenchmarkResult(benchmark_name="synthetic",baseline_name=baseline_name,num_tasks=n,
-                             num_success=success_count,num_partial=partial_count,num_failure=failure_count,
-                             num_false_done=false_done_count,num_blocked=blocked_count,
-                             total_cost=total_cost,avg_cost_success=avg_cost_success,
-                             avg_latency_ms=total_latency/n,total_tool_calls=total_tools,
-                             total_verifier_calls=total_verifiers,total_retries=total_retries,
-                             avg_cache_hit_rate=sum(cache_rates)/n,total_context_tokens=total_context,
-                             cost_reduction_vs_frontier=cost_reduction,false_done_rate=false_done_count/n,
-                             unsafe_cheap_miss_rate=cheap_misses/n,missed_escalation_rate=escalation_misses/n,
-                             regression_rate=regression_count/n)
-    def _simulate(self,trace,baseline):
-        base_cost=trace.total_cost_computed
-        if baseline=="always_frontier": cost_mult,tier=1.0,4
-        elif baseline=="always_cheap": cost_mult,tier=0.25,2
-        elif baseline=="no_router": cost_mult,tier=0.9,3
-        elif baseline=="no_tool_gate": cost_mult,tier=0.85,3
-        elif baseline=="no_early_termination": cost_mult,tier=0.95,3
-        else: cost_mult,tier=0.55,3
-        sim_cost=base_cost*cost_mult; sim_latency=trace.total_latency_ms*cost_mult*0.8
-        scenario=trace.metadata.get("scenario","normal")
-        success_prob=0.95 if tier>=3 else 0.7
-        if scenario=="cheap_failure": success_prob=0.3 if tier<=2 else 0.85
-        elif scenario=="tool_underuse": success_prob=0.8 if baseline!="no_tool_gate" else 0.6
-        elif scenario=="retry_loop": success_prob=0.2 if baseline=="no_early_termination" else 0.25
-        elif scenario=="frontier_unnecessary": success_prob=0.95
-        elif scenario=="meta_tool_success": success_prob=0.9 if baseline=="full" else 0.85
-        elif scenario=="meta_tool_bad": success_prob=0.4
-        elif scenario=="false_done": success_prob=0.1
-        elif scenario in ("blocked","stopped_doom"): success_prob=0.0
-        elif scenario=="human_escalation": success_prob=0.5
-        return sim_cost,sim_latency,success_prob>0.5
-    def report(self,results):
-        lines=["="*80,"AGENT COST OPTIMIZER BENCHMARK REPORT","="*80,""]
-        headers=["Baseline","Success","Partial","Fail","Blocked","False-DONE","Total Cost","Avg Cost/Succ","Latency(ms)","Tools","Verifiers","Retries","Cache Hit","Cost Reduction","Regression"]
-        lines.append(" | ".join(headers)); lines.append("-"*120)
-        for name,result in results.items():
-            row=[name[:20].ljust(20),f"{result.num_success/result.num_tasks:.1%}",
-                 f"{result.num_partial/result.num_tasks:.1%}",f"{result.num_failure/result.num_tasks:.1%}",
-                 f"{result.num_blocked/result.num_tasks:.1%}",f"{result.false_done_rate:.1%}",
-                 f"${result.total_cost:.2f}",f"${result.avg_cost_success:.4f}",f"{result.avg_latency_ms:.0f}",
-                 str(result.total_tool_calls),str(result.total_verifier_calls),str(result.total_retries),
-                 f"{result.avg_cache_hit_rate:.1%}",f"{result.cost_reduction_vs_frontier:.1%}",
-                 f"{result.regression_rate:.1%}"]
-            lines.append(" | ".join(row))
-        lines.append(""); lines.append("="*80)
-        best_score,best_name=-float("inf"),""
-        for name,result in results.items():
-            success_rate=(result.num_success+result.num_partial)/result.num_tasks
-            score=success_rate*10-result.avg_cost_success*100-result.regression_rate*50
-            if score>best_score: best_score,best_name=score,name
-        lines.append(f"BEST OVERALL: {best_name} (score={best_score:.2f})"); lines.append("")
-        return "\n".join(lines)
-    def export(self,results,path):
-        export_data={}
-        for name,result in results.items():
-            export_data[name]={"benchmark_name":result.benchmark_name,"baseline_name":result.baseline_name,
-                              "num_tasks":result.num_tasks,"num_success":result.num_success,
-                              "num_partial":result.num_partial,"num_failure":result.num_failure,
-                              "num_false_done":result.num_false_done,"num_blocked":result.num_blocked,
-                              "total_cost":result.total_cost,"avg_cost_success":result.avg_cost_success,
-                              "avg_latency_ms":result.avg_latency_ms,"total_tool_calls":result.total_tool_calls,
-                              "total_verifier_calls":result.total_verifier_calls,"total_retries":result.total_retries,
-                              "avg_cache_hit_rate":result.avg_cache_hit_rate,"total_context_tokens":result.total_context_tokens,
-                              "cost_reduction_vs_frontier":result.cost_reduction_vs_frontier,
-                              "false_done_rate":result.false_done_rate,"unsafe_cheap_miss_rate":result.unsafe_cheap_miss_rate,
-                              "missed_escalation_rate":result.missed_escalation_rate,"regression_rate":result.regression_rate}
-        with open(path,"w") as f: json.dump(export_data,f,indent=2)
-
-if __name__=="__main__":
-    parser=argparse.ArgumentParser(description="ACO Evaluation Runner")
-    parser.add_argument("--tasks","-n",type=int,default=1000,help="Number of tasks")
-    parser.add_argument("--seed","-s",type=int,default=42,help="Random seed")
-    parser.add_argument("--output","-o",default="./eval_results",help="Output directory")
-    args=parser.parse_args()
-    os.makedirs(args.output,exist_ok=True)
-    suite=BenchmarkSuite()
-    print(f"[{datetime.now().isoformat()}] Generating {args.tasks} synthetic traces...")
-    traces=suite.generate_benchmark_data(args.tasks,seed=args.seed)
-    traces_path=os.path.join(args.output,"traces.jsonl")
-    with open(traces_path,"w") as f:
-        for trace in traces: f.write(json.dumps(trace.to_dict())+"\n")
-    print(f"  Saved {len(traces)} traces to {traces_path}")
-    print(f"\n[{datetime.now().isoformat()}] Running baselines...")
-    baseline_results=suite.run_all_baselines(traces)
-    baseline_path=os.path.join(args.output,"baseline_results.json")
-    suite.export(baseline_results,baseline_path)
-    print(f"  Saved to {baseline_path}")
-    print(f"\n[{datetime.now().isoformat()}] Running ablations...")
-    ablation_results=suite.run_ablations(traces)
-    ablation_path=os.path.join(args.output,"ablation_results.json")
-    suite.export(ablation_results,ablation_path)
-    print(f"  Saved to {ablation_path}")
-    all_results={**baseline_results,**ablation_results}
-    report=suite.report(all_results)
-    report_path=os.path.join(args.output,"report.txt")
-    with open(report_path,"w") as f: f.write(report)
-    print(f"  Saved report to {report_path}")
-    points=[]
-    for name,result in all_results.items():
-        sr=(result.num_success+result.num_partial)/result.num_tasks
-        points.append({"baseline":name,"success_rate":sr,"avg_cost_per_success":result.avg_cost_success})
-    frontier=[]
-    for p in points:
-        dominated=False
-        for q in points:
-            if q["baseline"]==p["baseline"]: continue
-            if q["success_rate"]>=p["success_rate"] and q["avg_cost_per_success"]<=p["avg_cost_per_success"]:
-                if q["success_rate"]>p["success_rate"] or q["avg_cost_per_success"]<p["avg_cost_per_success"]:
-                    dominated=True; break
-        if not dominated: frontier.append(p)
-    frontier.sort(key=lambda x:x["success_rate"],reverse=True)
-    frontier_data={"all_points":points,"pareto_frontier":frontier,"frontier_baselines":[p["baseline"] for p in frontier]}
-    frontier_path=os.path.join(args.output,"cost_quality_frontier.json")
-    with open(frontier_path,"w") as f: json.dump(frontier_data,indent=2,fp=f)
-    print(f"  Saved frontier to {frontier_path}")
-    print("\n"+"="*80)
-    print(report)
-    print("="*80)

standalone_eval_v2.py

@@ -1,499 +0,0 @@
-#!/usr/bin/env python3
-"""Standalone benchmark runner v2 with realistic quality/cost tradeoffs."""
-import sys, json, os, uuid, random, argparse
-from datetime import datetime, timedelta
-from dataclasses import dataclass, field
-from enum import Enum
-from typing import Dict, List, Optional, Any, Tuple
-from collections import defaultdict
-
-class TaskType(Enum):
-    QUICK_ANSWER="quick_answer"; RESEARCH="research"; CODING="coding"
-    DOCUMENT_DRAFTING="document_drafting"; LEGAL_REGULATED="legal_regulated"
-    TOOL_HEAVY="tool_heavy"; RETRIEVAL_HEAVY="retrieval_heavy"
-    LONG_HORIZON="long_horizon"; UNKNOWN_AMBIGUOUS="unknown_ambiguous"
-
-class Outcome(Enum):
-    SUCCESS="success"; PARTIAL_SUCCESS="partial_success"; FAILURE="failure"
-    FALSE_DONE="false_done"; BLOCKED="blocked"; ESCALATED_HUMAN="escalated_human"
-    STOPPED_DOOM="stopped_doom"
-
-class FailureTag(Enum):
-    MODEL_TOO_WEAK="model_too_weak"; CONTEXT_TOO_SMALL="context_too_small"
-    TOOL_FAILED="tool_failed"; TOOL_UNNECESSARY="tool_unnecessary"
-    TOOL_MISSED="tool_missed"; RETRY_LOOP="retry_loop"
-    CACHE_BREAK="cache_break"; HALLUCINATION="hallucination"
-    TIMEOUT="timeout"; COST_EXCEEDED="cost_exceeded"
-    UNSAFE_CHEAP_MODEL="unsafe_cheap_model"; MISSED_ESCALATION="missed_escalation"
-    VERIFIER_FALSE_PASS="verifier_false_pass"
-
-@dataclass
-class ToolCall:
-    tool_name:str; tool_input:Dict[str,Any]; tool_output:Optional[str]=None
-    tool_cost:float=0.0; tool_latency_ms:float=0.0; cache_hit:bool=False
-    repeated:bool=False; ignored_result:bool=False; failed:bool=False
-
-@dataclass
-class ModelCall:
-    model_id:str; provider:str; input_tokens:int=0; output_tokens:int=0
-    reasoning_tokens:int=0; cost_per_1k_input:float=0.0; cost_per_1k_output:float=0.0
-    cache_hit_input_tokens:int=0; latency_ms:float=0.0
-    @property
-    def total_cost(self): return (self.input_tokens/1000)*self.cost_per_1k_input + (self.output_tokens/1000)*self.cost_per_1k_output - (self.cache_hit_input_tokens/1000)*self.cost_per_1k_input*0.5
-
-@dataclass
-class VerifierCall:
-    verifier_model_id:str; target_step_id:str; passed:bool=False
-    confidence:float=0.0; cost:float=0.0; latency_ms:float=0.0
-
-@dataclass
-class TraceStep:
-    step_id:str; timestamp:datetime; task_type:TaskType; model_call:ModelCall
-    tool_calls:List[ToolCall]=field(default_factory=list)
-    verifier_calls:List[VerifierCall]=field(default_factory=list)
-    context_size_tokens:int=0; context_sources:List[str]=field(default_factory=list)
-    retry_count:int=0; recovery_action:Optional[str]=None
-    artifacts_created:List[str]=field(default_factory=list)
-    step_outcome:Optional[Outcome]=None
-    @property
-    def step_cost(self): return (self.model_call.total_cost if self.model_call else 0.0)+sum(t.tool_cost for t in self.tool_calls)+sum(v.cost for v in self.verifier_calls)
-    @property
-    def step_latency_ms(self): return (self.model_call.latency_ms if self.model_call else 0.0)+sum(t.tool_latency_ms for t in self.tool_calls)+sum(v.latency_ms for v in self.verifier_calls)
-
-@dataclass
-class AgentTrace:
-    trace_id:str; user_request:str; task_type:TaskType
-    steps:List[TraceStep]=field(default_factory=list)
-    final_outcome:Optional[Outcome]=None; final_artifacts:List[str]=field(default_factory=list)
-    failure_tags:List[FailureTag]=field(default_factory=list)
-    user_satisfaction:Optional[float]=None
-    total_cost:Optional[float]=None
-    metadata:Dict[str,Any]=field(default_factory=dict)
-    @property
-    def total_cost_computed(self): return sum(s.step_cost for s in self.steps)
-    @property
-    def total_latency_ms(self): return sum(s.step_latency_ms for s in self.steps)
-    @property
-    def total_retries(self): return sum(s.retry_count for s in self.steps)
-    @property
-    def total_tool_calls(self): return sum(len(s.tool_calls) for s in self.steps)
-    @property
-    def total_verifier_calls(self): return sum(len(s.verifier_calls) for s in self.steps)
-    @property
-    def cache_hit_rate(self):
-        mc=[s.model_call for s in self.steps if s.model_call]
-        if not mc: return 0.0
-        ti=sum(m.input_tokens for m in mc)
-        return sum(m.cache_hit_input_tokens for m in mc)/ti if ti>0 else 0.0
-    def to_dict(self):
-        return {"trace_id":self.trace_id,"user_request":self.user_request,"task_type":self.task_type.value,
-                "steps":[{"step_id":s.step_id,"timestamp":s.timestamp.isoformat(),"task_type":s.task_type.value,
-                          "model_call":{"model_id":s.model_call.model_id,"provider":s.model_call.provider,
-                                        "input_tokens":s.model_call.input_tokens,"output_tokens":s.model_call.output_tokens,
-                                        "reasoning_tokens":s.model_call.reasoning_tokens,"cost":s.model_call.total_cost,
-                                        "latency_ms":s.model_call.latency_ms,"cache_hit_input_tokens":s.model_call.cache_hit_input_tokens},
-                          "tool_calls":[{"tool_name":t.tool_name,"tool_cost":t.tool_cost,"tool_latency_ms":t.tool_latency_ms,
-                                         "cache_hit":t.cache_hit,"repeated":t.repeated,"ignored_result":t.ignored_result,"failed":t.failed} for t in s.tool_calls],
-                          "verifier_calls":[{"verifier_model_id":v.verifier_model_id,"passed":v.passed,
-                                               "confidence":v.confidence,"cost":v.cost} for v in s.verifier_calls],
-                          "context_size_tokens":s.context_size_tokens,"retry_count":s.retry_count,
-                          "recovery_action":s.recovery_action,"step_outcome":s.step_outcome.value if s.step_outcome else None,
-                          "step_cost":s.step_cost,"step_latency_ms":s.step_latency_ms} for s in self.steps],
-                "final_outcome":self.final_outcome.value if self.final_outcome else None,
-                "failure_tags":[f.value for f in self.failure_tags],
-                "total_cost":self.total_cost_computed,"total_latency_ms":self.total_latency_ms,
-                "total_retries":self.total_retries,"total_tool_calls":self.total_tool_calls,
-                "total_verifier_calls":self.total_verifier_calls,
-                "cache_hit_rate":self.cache_hit_rate,"metadata":self.metadata}
-
-class SyntheticTraceGenerator:
-    # Realistic provider pricing (per 1K tokens)
-    MODEL_CONFIGS = {
-        "tiny_local": {"tier":1,"cost_input":0.0001,"cost_output":0.0002,"latency":200,"strength":0.35,"name":"Tiny Local (Qwen-0.5B)"},
-        "cheap_cloud": {"tier":2,"cost_input":0.00015,"cost_output":0.0006,"latency":400,"strength":0.55,"name":"GPT-4o-mini"},
-        "medium": {"tier":3,"cost_input":0.0015,"cost_output":0.006,"latency":800,"strength":0.80,"name":"Claude-3.5-Sonnet"},
-        "frontier": {"tier":4,"cost_input":0.005,"cost_output":0.015,"latency":1500,"strength":0.93,"name":"GPT-4o / Claude-3-Opus"},
-        "specialist": {"tier":5,"cost_input":0.01,"cost_output":0.03,"latency":2000,"strength":0.97,"name":"o1 / o3-mini"},
-    }
-    TOOL_COSTS = {"search":0.002,"retrieve":0.001,"fetch":0.003,"code_execution":0.005,
-                  "linter":0.001,"test_runner":0.003,"file_read":0.0005,"file_write":0.0005,
-                  "calculator":0.0001,"database_query":0.004,"compliance_check":0.01,
-                  "summarize":0.002,"task_planner":0.001,"progress_tracker":0.0005}
-    # Task difficulty: [tier_needed, risk_level]
-    TASK_DIFFICULTY = {
-        TaskType.QUICK_ANSWER: (1, 0.1),
-        TaskType.CODING: (3, 0.4),
-        TaskType.RESEARCH: (3, 0.5),
-        TaskType.DOCUMENT_DRAFTING: (2, 0.2),
-        TaskType.LEGAL_REGULATED: (4, 0.8),
-        TaskType.TOOL_HEAVY: (2, 0.3),
-        TaskType.RETRIEVAL_HEAVY: (2, 0.35),
-        TaskType.LONG_HORIZON: (3, 0.6),
-        TaskType.UNKNOWN_AMBIGUOUS: (3, 0.7),
-    }
-    SCENARIOS = [
-        {"name":"quick_answer_success","prob":0.18,"task_type":TaskType.QUICK_ANSWER,"tier":[1,2],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":1},
-        {"name":"quick_answer_cheap_fail","prob":0.02,"task_type":TaskType.QUICK_ANSWER,"tier":[1],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.MODEL_TOO_WEAK],"difficulty":2},
-        {"name":"coding_success_frontier","prob":0.08,"task_type":TaskType.CODING,"tier":[4],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":4},
-        {"name":"coding_success_medium","prob":0.10,"task_type":TaskType.CODING,"tier":[3],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":3},
-        {"name":"coding_cheap_fail","prob":0.05,"task_type":TaskType.CODING,"tier":[1,2],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.MODEL_TOO_WEAK],"difficulty":4},
-        {"name":"coding_tool_underuse","prob":0.04,"task_type":TaskType.CODING,"tier":[3,4],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.TOOL_MISSED],"difficulty":3},
-        {"name":"research_success","prob":0.10,"task_type":TaskType.RESEARCH,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":3},
-        {"name":"research_cheap_fail","prob":0.03,"task_type":TaskType.RESEARCH,"tier":[1,2],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.MODEL_TOO_WEAK],"difficulty":4},
-        {"name":"document_draft_success","prob":0.08,"task_type":TaskType.DOCUMENT_DRAFTING,"tier":[2,3],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":2},
-        {"name":"legal_frontier_success","prob":0.04,"task_type":TaskType.LEGAL_REGULATED,"tier":[4,5],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":5},
-        {"name":"legal_cheap_unsafe","prob":0.02,"task_type":TaskType.LEGAL_REGULATED,"tier":[1,2],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.UNSAFE_CHEAP_MODEL],"difficulty":5},
-        {"name":"tool_heavy_success","prob":0.06,"task_type":TaskType.TOOL_HEAVY,"tier":[2,3],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":2},
-        {"name":"retrieval_success","prob":0.06,"task_type":TaskType.RETRIEVAL_HEAVY,"tier":[2,3],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":2},
-        {"name":"long_horizon_success","prob":0.05,"task_type":TaskType.LONG_HORIZON,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":4},
-        {"name":"long_horizon_retry_loop","prob":0.03,"task_type":TaskType.LONG_HORIZON,"tier":[3],"outcome":Outcome.FAILURE,"failure_tags":[FailureTag.RETRY_LOOP],"difficulty":4},
-        {"name":"unknown_ambiguous_success","prob":0.03,"task_type":TaskType.UNKNOWN_AMBIGUOUS,"tier":[3,4],"outcome":Outcome.SUCCESS,"failure_tags":[],"difficulty":3},
-        {"name":"unknown_ambiguous_blocked","prob":0.02,"task_type":TaskType.UNKNOWN_AMBIGUOUS,"tier":[3,4],"outcome":Outcome.BLOCKED,"failure_tags":[FailureTag.MISSED_ESCALATION],"difficulty":3},
-        {"name":"tool_overuse","prob":0.04,"task_type":TaskType.CODING,"tier":[3,4],"outcome":Outcome.PARTIAL_SUCCESS,"failure_tags":[FailureTag.TOOL_UNNECESSARY],"difficulty":3},
-        {"name":"cache_break_scenario","prob":0.03,"task_type":TaskType.RESEARCH,"tier":[3,4],"outcome":Outcome.PARTIAL_SUCCESS,"failure_tags":[FailureTag.CACHE_BREAK],"difficulty":3},
-        {"name":"false_done_scenario","prob":0.02,"task_type":TaskType.CODING,"tier":[3,4],"outcome":Outcome.FALSE_DONE,"failure_tags":[FailureTag.VERIFIER_FALSE_PASS],"difficulty":3},
-    ]
-    def __init__(self,seed=42): self.rng=random.Random(seed)
-    def generate(self,n=10000): return [self._generate_trace(i) for i in range(n)]
-    def _pick_scenario(self): return self.rng.choices(self.SCENARIOS,weights=[s["prob"] for s in self.SCENARIOS])[0]
-    def _tier_to_model(self,tier): return {1:"tiny_local",2:"cheap_cloud",3:"medium",4:"frontier",5:"specialist"}.get(tier,"medium")
-    def _generate_trace(self,idx):
-        scenario=self._pick_scenario()
-        trace_id=f"synth_{idx}_{uuid.uuid4().hex[:8]}"
-        task_type=scenario["task_type"]
-        user_request=self._generate_request(task_type,scenario["name"])
-        base_steps=self.rng.randint(1,8)
-        if "long_horizon" in scenario["name"] or "retry_loop" in scenario["name"]: base_steps=self.rng.randint(4,12)
-        elif "coding" in scenario["name"] and scenario["outcome"]==Outcome.FAILURE: base_steps=self.rng.randint(3,8)
-        tier=self.rng.choice(scenario["tier"])
-        model_key=self._tier_to_model(tier)
-        model_cfg=self.MODEL_CONFIGS[model_key]
-        steps=[]
-        for step_idx in range(base_steps):
-            steps.append(self._generate_step(trace_id,step_idx,task_type,model_key,model_cfg,scenario,step_idx==base_steps-1))
-        return AgentTrace(
-            trace_id=trace_id,user_request=user_request,task_type=task_type,steps=steps,
-            final_outcome=scenario["outcome"],failure_tags=list(scenario.get("failure_tags",[])),
-            total_cost=sum(s.step_cost for s in steps),
-            metadata={"scenario":scenario["name"],"synthetic":True,"difficulty":scenario["difficulty"],
-                      "optimal_tier":scenario["difficulty"],"actual_tier":tier})
-    def _generate_request(self,task_type,scenario_name):
-        templates={
-            TaskType.QUICK_ANSWER:["What is the capital of France?","Briefly explain quantum computing.","Summarize article X.","What is 237 * 452?"],
-            TaskType.CODING:["Write a Python function to reverse a linked list.","Fix the bug in this React component.","Refactor auth module to JWT.","Implement LRU cache in Go.","Debug this segfault in C++ thread pool."],
-            TaskType.RESEARCH:["Research latest transformer advances.","Find sources comparing LoRA and full FT.","Investigate data center climate impact.","What does literature say on speculative decoding?"],
-            TaskType.DOCUMENT_DRAFTING:["Draft project proposal for ML pipeline.","Write email to team about deployment.","Create technical report on performance."],
-            TaskType.LEGAL_REGULATED:["Review this contract for liability clauses.","Check GDPR compliance for data pipeline.","Draft privacy policy section."],
-            TaskType.TOOL_HEAVY:["Search open issues and create summary.","Fetch API docs and generate client code.","Query Q3 sales and produce chart."],
-            TaskType.RETRIEVAL_HEAVY:["Answer based on 50-page document.","Find all 'payment processing' mentions.","Retrieve relevant cases for legal query."],
-            TaskType.LONG_HORIZON:["Plan 3-month roadmap.","Orchestrate multi-region deployment.","Redesign data architecture end-to-end."],
-            TaskType.UNKNOWN_AMBIGUOUS:["Help me with this thing.","I need something about the server.","Can you look into that issue?"],
-        }
-        return self.rng.choice(templates.get(task_type,["Generic request"]))
-    def _get_tools_for_task(self,task_type):
-        return {TaskType.QUICK_ANSWER:["calculator","search"],
-                TaskType.CODING:["file_read","file_write","code_execution","linter","test_runner"],
-                TaskType.RESEARCH:["search","retrieve","fetch","summarize"],
-                TaskType.DOCUMENT_DRAFTING:["file_read","summarize"],
-                TaskType.LEGAL_REGULATED:["document_retrieval","compliance_check","search"],
-                TaskType.TOOL_HEAVY:["search","fetch","api_call","database_query"],
-                TaskType.RETRIEVAL_HEAVY:["retrieve","search","fetch"],
-                TaskType.LONG_HORIZON:["task_planner","progress_tracker","file_read"],
-                TaskType.UNKNOWN_AMBIGUOUS:["search"]}.get(task_type,["search"])
-    def _generate_step(self,trace_id,step_idx,task_type,model_key,model_cfg,scenario,is_last):
-        step_id=f"{trace_id}_step_{step_idx}"
-        input_tokens=self.rng.randint(800,12000)
-        output_tokens=self.rng.randint(200,6000)
-        cache_hit=self.rng.random()<0.35
-        cache_hit_tokens=int(input_tokens*self.rng.random()*0.6) if cache_hit else 0
-        model_call=ModelCall(model_id=model_key,provider="synthetic",input_tokens=input_tokens,output_tokens=output_tokens,
-                             reasoning_tokens=output_tokens//4 if model_key in ("frontier","specialist") else 0,
-                             cost_per_1k_input=model_cfg["cost_input"],cost_per_1k_output=model_cfg["cost_output"],
-                             cache_hit_input_tokens=cache_hit_tokens,latency_ms=model_cfg["latency"]*self.rng.uniform(0.8,1.5))
-        tool_calls=[]; base_tools=self._get_tools_for_task(task_type); num_tools=self.rng.randint(0,len(base_tools))
-        if scenario["name"]=="tool_overuse": num_tools+=3
-        for t in range(min(num_tools,len(base_tools))):
-            tool_name=base_tools[t]
-            tool_calls.append(ToolCall(tool_name=tool_name,tool_input={"query":f"auto_{tool_name}"},
-                                       tool_cost=self.TOOL_COSTS.get(tool_name,0.001),tool_latency_ms=self.rng.uniform(100,1200),
-                                       cache_hit=self.rng.random()<0.2,repeated=self.rng.random()<0.1,
-                                       ignored_result=self.rng.random()<0.05,
-                                       failed=self.rng.random()<(0.3 if "retry_loop" in scenario["name"] else 0.05)))
-        verifier_calls=[]; num_verifiers=0
-        if task_type==TaskType.LEGAL_REGULATED: num_verifiers=1
-        elif task_type in (TaskType.CODING,TaskType.RESEARCH) and model_key in ("frontier","specialist"): num_verifiers=1 if self.rng.random()<0.4 else 0
-        for _ in range(num_verifiers):
-            verifier_calls.append(VerifierCall(verifier_model_id="verifier_medium",target_step_id=step_id,
-                                               passed=self.rng.random()<0.85,confidence=self.rng.uniform(0.6,0.99),cost=0.005,latency_ms=500))
-        context_size=self.rng.randint(1500,20000)
-        if scenario["name"]=="cache_break_scenario": context_size+=self.rng.randint(8000,30000)
-        retries=0
-        if "retry_loop" in scenario["name"]: retries=self.rng.randint(4,8)
-        elif self.rng.random()<0.12: retries=self.rng.randint(1,3)
-        recovery=None
-        if retries>0: recovery=self.rng.choice(["retry_same","retry_changed_prompt","repair_tool","switch_model","ask_clarification"])
-        step_outcome=Outcome.SUCCESS
-        if is_last: step_outcome=scenario["outcome"]
-        elif "retry_loop" in scenario["name"] and step_idx>=2: step_outcome=Outcome.FAILURE
-        return TraceStep(step_id=step_id,timestamp=datetime.utcnow()+timedelta(seconds=step_idx*30),task_type=task_type,
-                        model_call=model_call,tool_calls=tool_calls,verifier_calls=verifier_calls,
-                        context_size_tokens=context_size,context_sources=["system_rules","tool_descriptions","user_preferences","recent_messages"],
-                        retry_count=retries,recovery_action=recovery,
-                        artifacts_created=[f"artifact_{step_idx}"] if self.rng.random()<0.25 else [],
-                        step_outcome=step_outcome)
-
-@dataclass
-class BenchmarkResult:
-    baseline_name:str; num_tasks:int; num_success:int; num_partial:int
-    num_failure:int; num_false_done:int; num_blocked:int
-    total_cost:float; avg_cost_success:float; avg_latency_ms:float
-    total_tool_calls:int; total_verifier_calls:int; total_retries:int
-    avg_cache_hit_rate:float; cost_reduction_vs_frontier:float
-    false_done_rate:float; unsafe_cheap_miss_rate:float
-    missed_escalation_rate:float; regression_rate:float
-    per_scenario_stats:Dict[str,Dict[str,Any]]=field(default_factory=dict)
-
-class BenchmarkSuite:
-    MODEL_CONFIGS = SyntheticTraceGenerator.MODEL_CONFIGS
-    TASK_DIFFICULTY = SyntheticTraceGenerator.TASK_DIFFICULTY
-    def __init__(self): pass
-    def generate_benchmark_data(self,n=1000,seed=42): return SyntheticTraceGenerator(seed=seed).generate(n)
-
-    def run_all_baselines(self,traces):
-        baselines=["always_frontier","always_cheap","static","cascade","full_optimizer"]
-        results={}
-        for baseline in baselines:
-            print(f"Running baseline: {baseline}...")
-            results[baseline]=self._run_baseline(traces,baseline)
-        return results
-
-    def run_ablations(self,traces):
-        ablations=["no_router","no_tool_gate","no_verifier","no_early_term","no_context_budget"]
-        results={}
-        for ablation in ablations:
-            print(f"Running ablation: {ablation}...")
-            results[ablation]=self._run_baseline(traces,ablation)
-        return results
-
-    def _run_baseline(self,traces,baseline_name):
-        success_count=0; partial_count=0; failure_count=0; false_done_count=0; blocked_count=0
-        total_cost=0.0; total_latency=0.0; total_tools=0; total_verifiers=0; total_retries=0
-        cache_rates=[]; cheap_misses=0; escalation_misses=0; regression_count=0
-        per_scenario=defaultdict(lambda:{"count":0,"success":0,"cost":0.0})
-        for trace in traces:
-            sim_cost,sim_success,sim_outcome=self._simulate(trace,baseline_name)
-            total_cost+=sim_cost; total_latency+=trace.total_latency_ms*0.7
-            total_tools+=trace.total_tool_calls; total_verifiers+=trace.total_verifier_calls
-            total_retries+=trace.total_retries; cache_rates.append(trace.cache_hit_rate)
-            scenario=trace.metadata.get("scenario","normal")
-            per_scenario[scenario]["count"]+=1; per_scenario[scenario]["cost"]+=sim_cost
-            if sim_success:
-                if sim_outcome in (Outcome.SUCCESS,Outcome.PARTIAL_SUCCESS): 
-                    success_count+=1; per_scenario[scenario]["success"]+=1
-                else: regression_count+=1
-            else:
-                if sim_outcome==Outcome.FALSE_DONE: false_done_count+=1
-                elif sim_outcome==Outcome.BLOCKED: blocked_count+=1
-                else: failure_count+=1
-            # Track cheap model misses
-            difficulty=trace.metadata.get("difficulty",3)
-            actual_tier=trace.metadata.get("actual_tier",3)
-            if actual_tier<difficulty and actual_tier<=2 and sim_outcome in (Outcome.FAILURE,Outcome.PARTIAL_SUCCESS):
-                cheap_misses+=1
-            if actual_tier<difficulty and sim_outcome in (Outcome.FAILURE,Outcome.BLOCKED):
-                escalation_misses+=1
-        n=len(traces); avg_cost_success=total_cost/max(success_count,1)
-        frontier_total=sum(t.total_cost_computed*4 for t in traces)  # frontier costs ~4x medium
-        cost_reduction=(frontier_total-total_cost)/max(frontier_total,1)
-        return BenchmarkResult(
-            baseline_name=baseline_name,num_tasks=n,num_success=success_count,num_partial=partial_count,
-            num_failure=failure_count,num_false_done=false_done_count,num_blocked=blocked_count,
-            total_cost=total_cost,avg_cost_success=avg_cost_success,avg_latency_ms=total_latency/n,
-            total_tool_calls=total_tools,total_verifier_calls=total_verifiers,total_retries=total_retries,
-            avg_cache_hit_rate=sum(cache_rates)/n,cost_reduction_vs_frontier=cost_reduction,
-            false_done_rate=false_done_count/n,unsafe_cheap_miss_rate=cheap_misses/n,
-            missed_escalation_rate=escalation_misses/n,regression_rate=regression_count/n,
-            per_scenario_stats=dict(per_scenario))
-
-    def _simulate(self,trace,baseline):
-        """Realistic simulation: tier vs difficulty determines success."""
-        scenario=trace.metadata.get("scenario","normal")
-        difficulty=trace.metadata.get("difficulty",3)
-        actual_tier=trace.metadata.get("actual_tier",3)
-        base_cost=trace.total_cost_computed
-        # Determine what tier the baseline would actually use
-        if baseline=="always_frontier": chosen_tier=4
-        elif baseline=="always_cheap": chosen_tier=2
-        elif baseline in ("no_router","static"): chosen_tier=actual_tier  # uses same as trace, no optimization
-        elif baseline in ("cascade","full_optimizer"):
-            # Cascade tries lower tier first, escalates if needed
-            if difficulty<=2: chosen_tier=2
-            elif difficulty==3: chosen_tier=3 if self._tier_success_prob(3,difficulty)>0.7 else 4
-            elif difficulty==4: chosen_tier=3 if self._tier_success_prob(3,difficulty)>0.6 else 4
-            else: chosen_tier=4 if self._tier_success_prob(4,difficulty)>0.5 else 5
-        elif baseline=="no_tool_gate": chosen_tier=actual_tier  # same tier, but no tool savings
-        elif baseline=="no_verifier": chosen_tier=actual_tier
-        elif baseline=="no_early_term": chosen_tier=actual_tier
-        elif baseline=="no_context_budget": chosen_tier=actual_tier
-        else: chosen_tier=actual_tier
-        # Cost multiplier based on chosen tier
-        tier_cost_mult={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}.get(chosen_tier,0.75)
-        actual_cost_mult={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}.get(actual_tier,0.75)
-        # Adjust cost: cascade uses cheaper tier when possible
-        cost_ratio=tier_cost_mult/actual_cost_mult if actual_cost_mult>0 else 1.0
-        sim_cost=base_cost*cost_ratio
-        # Tool gate savings for cascade/full
-        if baseline in ("cascade","full_optimizer"):
-            if "tool_overuse" in scenario: sim_cost*=0.75
-        # Cache savings
-        if baseline=="full_optimizer" and "cache_break" not in scenario: sim_cost*=0.92
-        # Verifier savings
-        if baseline=="full_optimizer" and chosen_tier>=3 and difficulty<4: sim_cost*=0.95
-        # Early termination savings
-        if baseline=="full_optimizer" and "retry_loop" in scenario: sim_cost*=0.60
-        if baseline=="no_early_term" and "retry_loop" in scenario: sim_cost*=1.4
-        # Determine success probability
-        success_prob=self._tier_success_prob(chosen_tier,difficulty)
-        # Apply baseline-specific modifiers
-        if baseline=="always_cheap" and difficulty>=3: success_prob*=0.3
-        elif baseline=="no_tool_gate" and "tool" in scenario: success_prob*=0.7
-        elif baseline=="no_verifier" and difficulty>=4: success_prob*=0.85
-        elif baseline=="full_optimizer": success_prob=min(1.0,success_prob+0.05)
-        # Special scenarios
-        if "false_done" in scenario: success_prob=0.1
-        elif "blocked" in scenario: success_prob=0.0
-        elif "retry_loop" in scenario and baseline not in ("full_optimizer",):
-            if baseline=="no_early_term": success_prob=0.1
-            else: success_prob=0.25
-        elif "retry_loop" in scenario and baseline=="full_optimizer":
-            success_prob=0.5  # Doom detector catches it
-        sim_success=success_prob>0.5
-        # Determine simulated outcome
-        if "false_done" in scenario: sim_outcome=Outcome.FALSE_DONE
-        elif "blocked" in scenario: sim_outcome=Outcome.BLOCKED
-        elif sim_success:
-            if success_prob>0.85: sim_outcome=Outcome.SUCCESS
-            else: sim_outcome=Outcome.PARTIAL_SUCCESS
-        else:
-            if "retry_loop" in scenario: sim_outcome=Outcome.FAILURE
-            elif success_prob<0.2: sim_outcome=Outcome.BLOCKED
-            else: sim_outcome=Outcome.FAILURE
-        return sim_cost,sim_success,sim_outcome
-
-    def _tier_success_prob(self,tier,difficulty):
-        strength={1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}.get(tier,0.5)
-        # Success = strength^difficulty (harder tasks need exponentially more strength)
-        return strength**(difficulty*0.6)
-
-    def report(self,results):
-        lines=["="*100,"AGENT COST OPTIMIZER BENCHMARK REPORT v2","="*100,""]
-        headers=["Baseline","Success","Partial","Fail","Blocked","F-DONE",
-                 "Total Cost","Avg$/Succ","Lat(ms)","Tools","Verif","Retry",
-                 "Cache%","CostRed%","Regression","CheapMiss","EscMiss"]
-        lines.append(" | ".join(headers)); lines.append("-"*160)
-        for name,result in results.items():
-            row=[name[:22].ljust(22),
-                 f"{result.num_success/result.num_tasks:.1%}",
-                 f"{result.num_partial/result.num_tasks:.1%}",
-                 f"{result.num_failure/result.num_tasks:.1%}",
-                 f"{result.num_blocked/result.num_tasks:.1%}",
-                 f"{result.false_done_rate:.1%}",
-                 f"${result.total_cost:.2f}",
-                 f"${result.avg_cost_success:.4f}",
-                 f"{result.avg_latency_ms:.0f}",
-                 str(result.total_tool_calls),str(result.total_verifier_calls),str(result.total_retries),
-                 f"{result.avg_cache_hit_rate:.1%}",
-                 f"{result.cost_reduction_vs_frontier:.1%}",
-                 f"{result.regression_rate:.1%}",
-                 f"{result.unsafe_cheap_miss_rate:.1%}",
-                 f"{result.missed_escalation_rate:.1%}",
-                ]
-            lines.append(" | ".join(row))
-        lines.append(""); lines.append("="*100)
-        # Find best on Pareto frontier
-        best_score,best_name=-float("inf"),""
-        for name,result in results.items():
-            success_rate=(result.num_success+result.num_partial)/result.num_tasks
-            score=success_rate*20-result.avg_cost_success*50-result.regression_rate*30-result.unsafe_cheap_miss_rate*40
-            if score>best_score: best_score,best_name=score,name
-        lines.append(f"BEST PARETO: {best_name} (score={best_score:.2f})")
-        # Quality/cost ranking
-        lines.append(""); lines.append("QUALITY/COST FRONTIER (Success Rate vs Avg Cost per Success):")
-        points=[(name,(r.num_success+r.num_partial)/r.num_tasks,r.avg_cost_success) for name,r in results.items()]
-        points.sort(key=lambda x:(-x[1],x[2]))
-        for name,sr,cost in points:
-            lines.append(f"  {name:22s} | Success: {sr:.1%} | Cost/Success: ${cost:.4f}")
-        lines.append(""); lines.append("="*100)
-        return "\n".join(lines)
-
-    def export(self,results,path):
-        export_data={}
-        for name,result in results.items():
-            export_data[name]={"baseline_name":result.baseline_name,"num_tasks":result.num_tasks,
-                             "num_success":result.num_success,"num_partial":result.num_partial,
-                             "num_failure":result.num_failure,"num_false_done":result.num_false_done,
-                             "num_blocked":result.num_blocked,"total_cost":result.total_cost,
-                             "avg_cost_success":result.avg_cost_success,"avg_latency_ms":result.avg_latency_ms,
-                             "total_tool_calls":result.total_tool_calls,"total_verifier_calls":result.total_verifier_calls,
-                             "total_retries":result.total_retries,"avg_cache_hit_rate":result.avg_cache_hit_rate,
-                             "cost_reduction_vs_frontier":result.cost_reduction_vs_frontier,
-                             "false_done_rate":result.false_done_rate,
-                             "unsafe_cheap_miss_rate":result.unsafe_cheap_miss_rate,
-                             "missed_escalation_rate":result.missed_escalation_rate,
-                             "regression_rate":result.regression_rate,
-                             "per_scenario_stats":result.per_scenario_stats}
-        with open(path,"w") as f: json.dump(export_data,f,indent=2)
-
-if __name__=="__main__":
-    parser=argparse.ArgumentParser(description="ACO Evaluation Runner v2")
-    parser.add_argument("--tasks","-n",type=int,default=2000,help="Number of tasks")
-    parser.add_argument("--seed","-s",type=int,default=42,help="Random seed")
-    parser.add_argument("--output","-o",default="./eval_results_v2",help="Output directory")
-    args=parser.parse_args()
-    os.makedirs(args.output,exist_ok=True)
-    suite=BenchmarkSuite()
-    print(f"[{datetime.now().isoformat()}] Generating {args.tasks} synthetic traces...")
-    traces=suite.generate_benchmark_data(args.tasks,seed=args.seed)
-    traces_path=os.path.join(args.output,"traces.jsonl")
-    with open(traces_path,"w") as f:
-        for trace in traces: f.write(json.dumps(trace.to_dict())+"\n")
-    print(f"  Saved {len(traces)} traces to {traces_path}")
-    print(f"\n[{datetime.now().isoformat()}] Running baselines...")
-    baseline_results=suite.run_all_baselines(traces)
-    baseline_path=os.path.join(args.output,"baseline_results.json")
-    suite.export(baseline_results,baseline_path)
-    print(f"  Saved to {baseline_path}")
-    print(f"\n[{datetime.now().isoformat()}] Running ablations...")
-    ablation_results=suite.run_ablations(traces)
-    ablation_path=os.path.join(args.output,"ablation_results.json")
-    suite.export(ablation_results,ablation_path)
-    print(f"  Saved to {ablation_path}")
-    all_results={**baseline_results,**ablation_results}
-    report=suite.report(all_results)
-    report_path=os.path.join(args.output,"report.txt")
-    with open(report_path,"w") as f: f.write(report)
-    print(f"  Saved report to {report_path}")
-    # Cost-quality frontier
-    points=[]
-    for name,result in all_results.items():
-        sr=(result.num_success+result.num_partial)/result.num_tasks
-        points.append({"baseline":name,"success_rate":sr,"avg_cost_per_success":result.avg_cost_success,
-                       "total_cost":result.total_cost,"regression_rate":result.regression_rate,
-                       "false_done_rate":result.false_done_rate,"cheap_miss_rate":result.unsafe_cheap_miss_rate})
-    frontier=[]
-    for p in points:
-        dominated=False
-        for q in points:
-            if q["baseline"]==p["baseline"]: continue
-            if q["success_rate"]>=p["success_rate"] and q["avg_cost_per_success"]<=p["avg_cost_per_success"]:
-                if q["success_rate"]>p["success_rate"] or q["avg_cost_per_success"]<p["avg_cost_per_success"]:
-                    dominated=True; break
-        if not dominated: frontier.append(p)
-    frontier.sort(key=lambda x:x["success_rate"],reverse=True)
-    frontier_data={"all_points":points,"pareto_frontier":frontier,"frontier_baselines":[p["baseline"] for p in frontier]}
-    frontier_path=os.path.join(args.output,"cost_quality_frontier.json")
-    with open(frontier_path,"w") as f: json.dump(frontier_data,indent=2,fp=f)
-    print(f"  Saved frontier to {frontier_path}")
-    print("\n"+"="*100)
-    print(report)
-    print("="*100)

train_router.py

@@ -1,77 +0,0 @@
-#!/usr/bin/env python3
-"""Train a learned model router for Agent Cost Optimizer.
-
-Architecture: CARROT-style plug-in router.
-- Per-tier P(success|query) classifiers (XGBoost)
-- Per-tier cost estimators
-- Route to cheapest tier where P(success) > threshold
-
-Ground truth: optimal_tier = min{tier : success(tier)} from execution traces.
-"""
-import json, os, sys, random, pickle, uuid
-import numpy as np
-from datetime import datetime, timedelta
-from collections import defaultdict
-from typing import Dict, List, Tuple, Any, Optional
-from dataclasses import dataclass
-from enum import Enum
-
-# ─── Feature Extraction ───────────────────────────────────────────
-TASK_TYPES = [
-    "quick_answer", "coding", "research", "document_drafting",
-    "legal_regulated", "tool_heavy", "retrieval_heavy",
-    "long_horizon", "unknown_ambiguous",
-]
-TASK_TYPE_TO_IDX = {t: i for i, t in enumerate(TASK_TYPES)}
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-           "implement","test","compile","runtime","class","module","package",
-           "async","thread","queue","stack","heap","pointer","segfault","linter"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy",
-            "regulatory","liability","clause","indemnification","tos"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare",
-               "analyze","study","survey","paper","arxiv","citation"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape",
-           "lookup","download","upload","index","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate",
-           "pipeline","end-to-end","architecture","workflow","deploy"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize",
-           "probability","integral","derivative","matrix"]
-
-def extract_features(request: str, task_type: str, metadata: Dict = None) -> Dict[str, Any]:
-    r = request.lower()
-    feats = {
-        "request_length": len(request),
-        "num_words": len(request.split()),
-        "num_sentences": request.count(".") + request.count("!") + request.count("?"),
-        "has_code_kw": int(any(kw in r for kw in CODE_KW)),
-        "num_code_kw": sum(1 for kw in CODE_KW if kw in r),
-        "has_legal_kw": int(any(kw in r for kw in LEGAL_KW)),
-        "num_legal_kw": sum(1 for kw in LEGAL_KW if kw in r),
-        "has_research_kw": int(any(kw in r for kw in RESEARCH_KW)),
-        "num_research_kw": sum(1 for kw in RESEARCH_KW if kw in r),
-        "has_tool_kw": int(any(kw in r for kw in TOOL_KW)),
-        "num_tool_kw": sum(1 for kw in TOOL_KW if kw in r),
-        "has_long_kw": int(any(kw in r for kw in LONG_KW)),
-        "has_math_kw": int(any(kw in r for kw in MATH_KW)),
-        "task_type_idx": TASK_TYPE_TO_IDX.get(task_type, 8),
-    }
-    # One-hot task type
-    for tt in TASK_TYPES:
-        feats[f"tt_{tt}"] = int(task_type == tt)
-    if metadata:
-        feats["difficulty"] = metadata.get("difficulty", 3)
-    return feats
-
-def feats_to_array(feats: Dict) -> List[float]:
-    """Convert feature dict to fixed-order array."""
-    keys = sorted(feats.keys())
-    return [float(feats[k]) for k in keys]
-
-FEAT_KEYS = None  # set on first call
-
-def feats_to_array_safe(feats: Dict) -> List[float]:
-    global FEAT_KEYS
-    if FEAT_KEYS is None:
-        FEAT_KEYS = sorted(feats.keys())
-    return [float(feats.get(k, 0.0)) for k in FEAT_KEYS]

train_router_gen.py

@@ -1,126 +0,0 @@
-
-# ─── Synthetic Trace Generator (for training data) ────────────────
-MODEL_CONFIGS = {
-    "tiny_local": {"tier":1,"cost_input":0.0001,"cost_output":0.0002,"latency":200,"strength":0.35},
-    "cheap_cloud": {"tier":2,"cost_input":0.00015,"cost_output":0.0006,"latency":400,"strength":0.55},
-    "medium": {"tier":3,"cost_input":0.0015,"cost_output":0.006,"latency":800,"strength":0.80},
-    "frontier": {"tier":4,"cost_input":0.005,"cost_output":0.015,"latency":1500,"strength":0.93},
-    "specialist": {"tier":5,"cost_input":0.01,"cost_output":0.03,"latency":2000,"strength":0.97},
-}
-TIER_TO_MODEL = {1:"tiny_local",2:"cheap_cloud",3:"medium",4:"frontier",5:"specialist"}
-TIER_COST_MULT = {1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-
-TASK_TEMPLATES = {
-    "quick_answer": [
-        "What is the capital of France?","Explain quantum computing briefly.",
-        "What is 237*452?","Define photosynthesis.","Who wrote Hamlet?",
-        "What is the speed of light?","List the primary colors.",
-        "What is GDP?","When was the Declaration of Independence signed?",
-    ],
-    "coding": [
-        "Write a Python function to reverse a linked list.",
-        "Fix the bug in this React component.","Refactor auth module to JWT.",
-        "Implement LRU cache in Go.","Debug segfault in C++ thread pool.",
-        "Add unit tests for the payment module.","Optimize this SQL query.",
-        "Create a REST API for user management.","Implement binary search in Rust.",
-        "Write a recursive descent parser for JSON.",
-    ],
-    "research": [
-        "Research latest transformer advances.",
-        "Find sources comparing LoRA and full FT.",
-        "Investigate data center climate impact.",
-        "What does literature say on speculative decoding?",
-        "Survey privacy-preserving ML techniques.",
-        "Compare reinforcement learning algorithms for robotics.",
-        "Analyze trends in LLM scaling laws.",
-    ],
-    "document_drafting": [
-        "Draft project proposal for ML pipeline.",
-        "Write email to team about deployment.",
-        "Create technical report on performance.",
-        "Write a project brief for the migration.",
-        "Draft meeting notes summary.",
-    ],
-    "legal_regulated": [
-        "Review this contract for liability clauses.",
-        "Check GDPR compliance for data pipeline.",
-        "Draft privacy policy section.",
-        "Analyze indemnification clause in vendor agreement.",
-        "Verify regulatory compliance for medical device software.",
-    ],
-    "tool_heavy": [
-        "Search open issues and create summary.",
-        "Fetch API docs and generate client code.",
-        "Query Q3 sales and produce chart.",
-        "Aggregate metrics from 5 monitoring endpoints.",
-    ],
-    "retrieval_heavy": [
-        "Answer based on 50-page document.",
-        "Find all payment processing mentions.",
-        "Retrieve relevant cases for legal query.",
-        "Summarize the quarterly earnings report.",
-    ],
-    "long_horizon": [
-        "Plan 3-month roadmap.","Orchestrate multi-region deployment.",
-        "Redesign data architecture end-to-end.","Migrate monolith to microservices.",
-    ],
-    "unknown_ambiguous": [
-        "Help me with this thing.","I need something about the server.",
-        "Can you look into that issue?","There's a problem with the data.",
-    ],
-}
-
-def tier_success_prob(tier, difficulty):
-    strength = {1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}.get(tier,0.5)
-    return strength ** (difficulty * 0.6)
-
-def generate_training_trace(idx, rng):
-    task_types = list(TASK_TEMPLATES.keys())
-    task_type = rng.choice(task_types)
-    difficulty = {
-        "quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-        "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5,
-    }[task_type]
-
-    # Try ALL tiers for this task to get ground truth
-    tier_outcomes = {}
-    for tier in range(1, 6):
-        sp = tier_success_prob(tier, difficulty)
-        tier_outcomes[tier] = rng.random() < sp
-
-    optimal_tier = 5  # default: need strongest
-    for tier in range(1, 6):
-        if tier_outcomes.get(tier, False):
-            optimal_tier = tier
-            break
-
-    actual_tier = rng.choice(list(range(1, 6)))
-    # Bias toward reasonable tiers
-    if difficulty <= 2:
-        actual_tier = rng.choices([1,2,3,4,5], weights=[3,4,2,1,0.5])[0]
-    elif difficulty == 3:
-        actual_tier = rng.choices([1,2,3,4,5], weights=[1,2,4,2,1])[0]
-    elif difficulty == 4:
-        actual_tier = rng.choices([1,2,3,4,5], weights=[0.5,1,2,4,2])[0]
-    else:
-        actual_tier = rng.choices([1,2,3,4,5], weights=[0.2,0.5,1,3,4])[0]
-
-    outcome = "success" if tier_outcomes.get(actual_tier, False) else "failure"
-
-    user_request = rng.choice(TASK_TEMPLATES[task_type])
-    cost_mult = TIER_COST_MULT[actual_tier]
-    base_tokens = rng.randint(800, 12000) + rng.randint(200, 6000)
-    cost = base_tokens / 1000 * MODEL_CONFIGS[TIER_TO_MODEL[actual_tier]]["cost_input"] * cost_mult
-
-    return {
-        "trace_id": f"train_{idx}",
-        "user_request": user_request,
-        "task_type": task_type,
-        "difficulty": difficulty,
-        "actual_tier": actual_tier,
-        "optimal_tier": optimal_tier,
-        "outcome": outcome,
-        "cost": cost,
-        "tier_outcomes": {str(k): v for k, v in tier_outcomes.items()},
-        "metadata": {"difficulty": difficulty, "optimal_tier": optimal_tier, "actual_tier": actual_tier},
-    }

training/check_bundle_keys.py

@@ -1,38 +0,0 @@
-"""Check what feature keys the v10 router bundle actually expects."""
-import pickle
-from huggingface_hub import hf_hub_download
-
-bundle_path = hf_hub_download(
-    repo_id="narcolepticchicken/agent-cost-optimizer",
-    filename="router_models/router_bundle_v10_fixed.pkl")
-
-bundle = pickle.load(open(bundle_path, 'rb'))
-
-print("Bundle keys:", list(bundle.keys()))
-print("Feature keys:", bundle.get('feat_keys', bundle.get('feature_keys', 'NOT FOUND')))
-print("Number of features per classifier:")
-
-for k, clf in bundle.get('tier_clfs', {}).items():
-    if hasattr(clf, 'n_features_in_'):
-        print(f"  Tier {k}: {clf.n_features_in_} features")
-    elif hasattr(clf, 'get_booster'):
-        try:
-            print(f"  Tier {k}: XGBoost, feature_names={clf.get_booster().feature_names[:5]}...")
-        except:
-            print(f"  Tier {k}: XGBoost, can't read features")
-    else:
-        print(f"  Tier {k}: unknown type {type(clf).__name__}")
-
-# Also check the available feature key files
-print("\nRouter models directory:")
-import os
-for f in os.listdir(os.path.dirname(bundle_path)):
-    if 'feat' in f.lower() or 'key' in f.lower():
-        print(f"  {f}")
-    # Also check the bundle files for feat_keys
-    if f.endswith('.pkl'):
-        b2 = pickle.load(open(os.path.join(os.path.dirname(bundle_path), f), 'rb'))
-        if isinstance(b2, dict):
-            for bk in b2:
-                if 'feat' in str(bk).lower() or 'key' in str(bk).lower():
-                    print(f"  {f} → {bk}: {b2[bk]}")

training/debug_sprout_parsing.py

@@ -1,83 +0,0 @@
-#!/usr/bin/env python3
-"""DEBUG: Inspect SPROUT parsing failures to understand the 11,610 parse errors."""
-import json
-from datasets import load_dataset
-
-print("DEBUG: SPROUT parsing inspection")
-ds = load_dataset("CARROT-LLM-Routing/SPROUT", split="train")
-print(f"Total rows: {len(ds)}")
-
-MODEL_COLS = [
-    "aws-claude-3-5-sonnet-v1", "openai-gpt-4o", "openai-gpt-4o-mini",
-    "aws-titan-text-premier-v1", "wxai-granite-3-2b-instruct-8k-max-tokens",
-    "wxai-granite-3-8b-instruct-8k-max-tokens", "wxai-llama-3-1-70b-instruct",
-    "wxai-llama-3-1-8b-instruct", "wxai-llama-3-2-1b-instruct",
-    "wxai-llama-3-2-3b-instruct", "wxai-llama-3-3-70b-instruct",
-    "wxai-llama-3-405b-instruct", "wxai-mixtral-8x7b-instruct-v01",
-]
-
-# Inspect first 50 rows and categorize failures
-error_types = {}
-success_count = 0
-for col in MODEL_COLS:
-    for idx in range(min(100, len(ds))):
-        val = ds[idx][col]
-        if isinstance(val, dict):
-            jr = val.get("judge_response", "")
-            if isinstance(jr, str) and jr.strip():
-                try:
-                    parsed = json.loads(jr)
-                    score = parsed.get("correctness_score")
-                    if score is not None:
-                        success_count += 1
-                    else:
-                        error_types.setdefault("no_correctness_score", []).append(col)
-                except json.JSONDecodeError as e:
-                    error_types.setdefault(f"json_error", []).append((col, str(e)[:80], repr(jr)[:100]))
-            elif isinstance(jr, dict):
-                score = jr.get("correctness_score")
-                if score is not None:
-                    success_count += 1
-                else:
-                    error_types.setdefault("dict_no_score", []).append(col)
-            elif jr == "" or jr is None:
-                error_types.setdefault("empty_judge_response", []).append(col)
-            else:
-                error_types.setdefault(f"unexpected_jr_type:{type(jr).__name__}", []).append((col, repr(jr)[:100]))
-        elif isinstance(val, str):
-            try:
-                parsed = json.loads(val)
-                jr = parsed.get("judge_response", "")
-                if isinstance(jr, str):
-                    jr_parsed = json.loads(jr)
-                    score = jr_parsed.get("correctness_score")
-                    if score is not None:
-                        success_count += 1
-                    else:
-                        error_types.setdefault("str_no_score", []).append(col)
-                elif isinstance(jr, dict):
-                    score = jr.get("correctness_score")
-                    if score is not None:
-                        success_count += 1
-            except json.JSONDecodeError as e:
-                error_types.setdefault("str_json_error", []).append((col, str(e)[:80]))
-        elif val is None:
-            error_types.setdefault("null_value", []).append(col)
-        else:
-            error_types.setdefault(f"unexpected_type:{type(val).__name__}", []).append(col)
-
-print(f"\nSuccesses (first 100 rows): {success_count}")
-print(f"\nError categories:")
-for cat, examples in error_types.items():
-    print(f"  {cat}: {len(examples)}")
-    if examples:
-        print(f"    Examples: {examples[:3]}")
-
-# Also dump the raw value for a specific failing case
-print("\n\n--- Raw value types (first 5 rows, first 3 models) ---")
-for idx in range(5):
-    row = ds[idx]
-    print(f"\nRow {idx} (key={row['key'][:20]}...):")
-    for col in MODEL_COLS[:3]:
-        val = row[col]
-        print(f"  {col}: type={type(val).__name__}, {str(val)[:120]}")

training/execution_feedback_loop.py

@@ -1,340 +0,0 @@
-#!/usr/bin/env python3
-"""Execution Feedback Loop: Route → Execute → Learn.
-
-Combines the BERT 5-class router with real SWE-bench execution outcomes.
-This is the "real execution feedback" the final report says is missing.
-
-Flow:
-1. BERT router selects optimal tier for each SWE-bench task
-2. Map tier → actual SWE-Router model, look up execution outcome
-3. If cheap model fails: escalate to next tier, record feedback
-4. Update routing thresholds using collected feedback
-5. Compare: BERT, BERT+feedback, XGBoost v10, frontier, oracle
-
-Key insight from BAAR (2026): profile with small model, then decide.
-We extend: route, execute, observe outcome, feed back to routing policy.
-"""
-import json, sys, numpy as np
-from collections import defaultdict
-from datasets import load_dataset
-from transformers import AutoTokenizer, AutoModelForSequenceClassification
-import torch
-import pickle
-
-MODELS = [
-    "claude-opus-4.7", "gpt-5-mini", "gpt-5-nano", "gpt-5.2",
-    "gemini-2.5-pro", "gemini-3-pro", "deepseek-v3.2", "deepseek-v4-flash",
-]
-MODEL_TIER = {
-    "deepseek-v4-flash": 1, "gpt-5-nano": 1,
-    "gpt-5-mini": 2, "deepseek-v3.2": 2,
-    "gemini-2.5-pro": 3,
-    "claude-opus-4.7": 4, "gpt-5.2": 4,
-    "gemini-3-pro": 5,
-}
-TIER_COST = {1: 0.01, 2: 0.05, 3: 0.15, 4: 0.30, 5: 0.50}
-TIER_TO_SWE = {
-    1: "deepseek-v4-flash", 2: "gpt-5-mini",
-    3: "gemini-2.5-pro", 4: "claude-opus-4.7", 5: "gemini-3-pro",
-}
-
-print("=" * 70)
-print("EXECUTION FEEDBACK LOOP: BERT Route → Execute → Learn")
-print("=" * 70)
-
-# ── 1. Load BERT 5-class router ─────────────────────────────────
-print("\n[1] Loading BERT 5-class router...")
-try:
-    tokenizer = AutoTokenizer.from_pretrained(
-        "narcolepticchicken/agent-cost-optimizer",
-        subfolder="router_models/bert_5class",
-    )
-    model = AutoModelForSequenceClassification.from_pretrained(
-        "narcolepticchicken/agent-cost-optimizer",
-        subfolder="router_models/bert_5class",
-    )
-    print("  ✓ Loaded BERT 5-class from Hub")
-except Exception as e:
-    print(f"  ✗ Hub load failed: {e}")
-    print("  Using distilbert-base-uncased (untrained, random weights)...")
-    tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
-    model = AutoModelForSequenceClassification.from_pretrained(
-        "distilbert-base-uncased", num_labels=5
-    )
-
-device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-model = model.to(device)
-model.eval()
-
-def bert_predict(problem_text):
-    """BERT 5-class prediction: returns (predicted_tier, tier_probs_dict)."""
-    enc = tokenizer(
-        problem_text[:2000], truncation=True, max_length=512, return_tensors="pt"
-    )
-    enc = {k: v.to(device) for k, v in enc.items()}
-    with torch.no_grad():
-        logits = model(**enc).logits
-        probs = torch.softmax(logits, dim=-1)[0].cpu().numpy()
-    predicted_class = int(np.argmax(probs))
-    predicted_tier = predicted_class + 1
-    tier_probs = {t: float(probs[t - 1]) for t in range(1, 6)}
-    return predicted_tier, tier_probs
-
-# ── 2. Load SWE-Router execution data ────────────────────────────
-print("\n[2] Loading SWE-Router execution data...")
-traces = defaultdict(dict)
-for model_name in MODELS:
-    ds = load_dataset(f"SWE-Router/swebench-verified-{model_name}", split="test")
-    for row in ds:
-        iid = row["instance_id"]
-        traces[iid][model_name] = {
-            "resolved": row["resolved"],
-            "cost": float(row["instance_cost"]),
-            "api_calls": int(row["api_calls"]),
-            "problem": row["problem_statement"],
-        }
-    print(f"  {model_name}: loaded")
-
-n_tasks = len(traces)
-print(f"  Total tasks: {n_tasks}")
-
-# ── 3. Load XGBoost v10 router for comparison ────────────────────
-print("\n[3] Loading XGBoost v10 router...")
-try:
-    with open("/app/router_models/router_bundle_v10.pkl", "rb") as f:
-        v10_bundle = pickle.load(f)
-    print(f"  ✓ Loaded v10 XGBoost ({v10_bundle['version']})")
-    HAS_V10 = True
-except Exception as e:
-    print(f"  ✗ v10 bundle failed: {e} — trying v10_fixed")
-    try:
-        with open("/app/router_models/router_bundle_v10_fixed.pkl", "rb") as f:
-            v10_bundle = pickle.load(f)
-        print(f"  ✓ Loaded v10_fixed")
-        HAS_V10 = True
-    except Exception as e2:
-        print(f"  ✗ Also failed: {e2} — skipping XGBoost comparison")
-        HAS_V10 = False
-
-# ── 4. Run routing policies ──────────────────────────────────────
-print("\n[4] Running routing policies on all SWE-bench tasks...")
-
-policies = defaultdict(lambda: {"success": 0, "cost": 0.0, "escalations": 0, "n": 0})
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))["problem"]
-
-    # ──�� BERT direct ───
-    bt, bp = bert_predict(problem)
-    bm = TIER_TO_SWE.get(bt, "claude-opus-4.7")
-    if bm in model_results:
-        policies["bert_direct"]["success"] += int(model_results[bm]["resolved"])
-        policies["bert_direct"]["cost"] += model_results[bm]["cost"]
-    policies["bert_direct"]["n"] += 1
-
-    # ─── BERT + feedback cascade ───
-    try_tier = bt
-    total_cost = 0.0
-    success = False
-    esc = 0
-    while try_tier <= 5:
-        tm = TIER_TO_SWE.get(try_tier, "claude-opus-4.7")
-        if tm in model_results:
-            total_cost += model_results[tm]["cost"]
-            if model_results[tm]["resolved"]:
-                success = True
-                break
-        try_tier += 1
-        esc += 1
-    policies["bert_feedback"]["success"] += int(success)
-    policies["bert_feedback"]["cost"] += total_cost
-    policies["bert_feedback"]["escalations"] += esc
-    policies["bert_feedback"]["n"] += 1
-
-    # ─── BERT cascade threshold ───
-    cascade_tier = 5
-    for t in range(1, 6):
-        if bp[t] >= 0.3:
-            cascade_tier = t
-            break
-    cm = TIER_TO_SWE.get(cascade_tier, "claude-opus-4.7")
-    if cm in model_results:
-        policies["bert_cascade"]["success"] += int(model_results[cm]["resolved"])
-        policies["bert_cascade"]["cost"] += model_results[cm]["cost"]
-    policies["bert_cascade"]["n"] += 1
-
-    # ─── XGBoost v10 direct (if available) ───
-    if HAS_V10:
-        feat_keys = v10_bundle["feat_keys"]
-        from collections import Counter
-        
-        # Extract features
-        r = problem.lower()
-        CODE_KW = ["python","javascript","code","function","bug","debug","refactor","implement","test",
-                   "compile","runtime","segfault","thread","async","class","module","import","error","traceback"]
-        LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-        RESEARCH_KW = ["research","investigate","compare","analyze","survey","paper"]
-        TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-        CRITICAL_KW = ["critical","production","urgent","emergency","live","deployed","safety","security"]
-        SIMPLE_KW = ["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-        LONG_KW = ["plan","project","roadmap","orchestrate","migrate","pipeline","deploy","architecture"]
-        MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-        
-        feats = {
-            'req_len': len(problem), 'num_words': len(problem.split()),
-            'has_code': int(any(k in r for k in CODE_KW)),
-            'n_code': sum(1 for k in CODE_KW if k in r),
-            'has_legal': int(any(k in r for k in LEGAL_KW)),
-            'has_research': int(any(k in r for k in RESEARCH_KW)),
-            'has_tool': int(any(k in r for k in TOOL_KW)),
-            'has_critical': int(any(k in r for k in CRITICAL_KW)),
-            'has_simple': int(any(k in r for k in SIMPLE_KW)),
-            'has_long': int(any(k in r for k in LONG_KW)),
-            'has_math': int(any(k in r for k in MATH_KW)),
-            'has_error_msg': int('error' in r or 'traceback' in r),
-            'has_file_path': int('/' in r and ('.' in r.split('/')[0] if '/' in r else False)),
-            'n_lines': problem.count('\n') + 1,
-            'has_version': int('version' in r or 'update' in r),
-            'has_add': int('add' in r or 'new' in r or 'create' in r),
-            'has_fix': int('fix' in r or 'bug' in r or 'issue' in r),
-            'has_change': int('change' in r or 'modify' in r),
-            'has_remove': int('remove' in r or 'delete' in r),
-            'has_test': int('test' in r or 'spec' in r or 'assert' in r),
-            'has_doc': int('doc' in r or 'readme' in r),
-            'has_see_also': int('see also' in r or 'related' in r),
-            'has_steps_to_reproduce': int('steps to reproduce' in r or 'reproduce' in r),
-        }
-        feat_vec = np.array([feats.get(k, 0.0) for k in feat_keys], dtype=np.float32).reshape(1, -1)
-        
-        opt_clf = v10_bundle["opt_clf"]
-        xgb_tier = int(opt_clf.predict(feat_vec)[0]) + 1
-        xgb_model = TIER_TO_SWE.get(xgb_tier, "claude-opus-4.7")
-        if xgb_model in model_results:
-            policies["xgb_direct"]["success"] += int(model_results[xgb_model]["resolved"])
-            policies["xgb_direct"]["cost"] += model_results[xgb_model]["cost"]
-        policies["xgb_direct"]["n"] += 1
-
-    # ─── Always frontier ───
-    fm = "claude-opus-4.7"
-    policies["frontier"]["success"] += int(model_results[fm]["resolved"])
-    policies["frontier"]["cost"] += model_results[fm]["cost"]
-    policies["frontier"]["n"] += 1
-
-    # ─── Always cheap ───
-    cm_cheap = "deepseek-v4-flash"
-    if cm_cheap in model_results:
-        policies["always_cheap"]["success"] += int(model_results[cm_cheap]["resolved"])
-        policies["always_cheap"]["cost"] += model_results[cm_cheap]["cost"]
-    policies["always_cheap"]["n"] += 1
-
-    # ─── Oracle ───
-    resolved = [(m, r) for m, r in model_results.items() if r["resolved"]]
-    if resolved:
-        cheapest = min(resolved, key=lambda x: TIER_COST.get(MODEL_TIER[x[0]], 1.0))
-        policies["oracle"]["success"] += 1
-        policies["oracle"]["cost"] += cheapest[1]["cost"]
-    else:
-        policies["oracle"]["cost"] += min(r["cost"] for r in model_results.values())
-    policies["oracle"]["n"] += 1
-
-# ── 5. Results ───────────────────────────────────────────────────
-fr_cost = policies["frontier"]["cost"] / max(policies["frontier"]["n"], 1)
-
-print(f"\n{'='*70}")
-print("FINAL RESULTS: BERT 5-CLASS + EXECUTION FEEDBACK ON SWE-BENCH")
-print(f"{'='*70}")
-print(f"\n{'Policy':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Esc':>8}")
-print("-" * 65)
-
-names = ["oracle"]
-if HAS_V10: names.append("xgb_direct")
-names += ["bert_feedback", "bert_cascade", "bert_direct", "frontier", "always_cheap"]
-
-for name in names:
-    r = policies[name]
-    sr = r["success"] / max(r["n"], 1)
-    ac = r["cost"] / max(r["n"], 1)
-    cr = (1 - ac / max(fr_cost, 0.0001)) * 100
-    esc = r.get("escalations", 0) / max(r["n"], 1)
-    print(f"  {name:<23} {sr:>10.3f} {ac:>10.4f} {cr:>9.1f}% {esc:>8.2f}")
-
-# ── 6. Detailed feedback analysis ────────────────────────────────
-print(f"\n{'='*70}")
-print("FEEDBACK ANALYSIS")
-print(f"{'='*70}")
-
-correct_cheapest = 0
-over_estimated = 0
-under_estimated = 0
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))["problem"]
-    bt, bp = bert_predict(problem)
-
-    actual_cheapest = 5
-    for tier in range(1, 6):
-        tm = TIER_TO_SWE.get(tier)
-        if tm and tm in model_results and model_results[tm]["resolved"]:
-            actual_cheapest = tier
-            break
-
-    if bt == actual_cheapest:
-        correct_cheapest += 1
-    elif bt > actual_cheapest:
-        over_estimated += 1
-    else:
-        under_estimated += 1
-
-print(f"  BERT == actual cheapest tier: {correct_cheapest}/{n_tasks} = {correct_cheapest/n_tasks*100:.1f}%")
-print(f"  BERT over-estimated (conservative): {over_estimated}/{n_tasks} = {over_estimated/n_tasks*100:.1f}%")
-print(f"  BERT under-estimated (risky):       {under_estimated}/{n_tasks} = {under_estimated/n_tasks*100:.1f}%")
-
-# Per-tier prediction distribution
-print(f"\n  BERT tier prediction distribution:")
-tier_pred_counts = defaultdict(int)
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))["problem"]
-    bt, _ = bert_predict(problem)
-    tier_pred_counts[bt] += 1
-for t in range(1, 6):
-    print(f"    Tier {t}: {tier_pred_counts[t]} ({tier_pred_counts[t]/n_tasks*100:.1f}%)")
-
-# ── 7. Save results ──────────────────────────────────────────────
-results = {
-    "router": "BERT 5-class v3 + execution feedback",
-    "n_tasks": n_tasks,
-    "policies": {
-        name: {
-            "success": r["success"],
-            "avg_cost": round(r["cost"] / max(r["n"], 1), 4),
-            "success_rate": round(r["success"] / max(r["n"], 1), 4),
-            "cost_reduction_pct": round(
-                (1 - (r["cost"] / max(r["n"], 1)) / max(fr_cost, 0.0001)) * 100, 1
-            ),
-        }
-        for name, r in policies.items()
-    },
-    "feedback_analysis": {
-        "correct_cheapest_pct": round(correct_cheapest / n_tasks * 100, 1),
-        "over_estimated_pct": round(over_estimated / n_tasks * 100, 1),
-        "under_estimated_pct": round(under_estimated / n_tasks * 100, 1),
-        "tier_prediction_distribution": dict(tier_pred_counts),
-    },
-}
-with open("/app/bert_feedback_results.json", "w") as f:
-    json.dump(results, f, indent=2)
-print(f"\n  ✓ Saved to /app/bert_feedback_results.json")
-
-from huggingface_hub import HfApi
-api = HfApi()
-api.upload_file(
-    path_or_fileobj="/app/bert_feedback_results.json",
-    path_in_repo="eval/bert_feedback_results.json",
-    repo_id="narcolepticchicken/agent-cost-optimizer",
-    repo_type="model",
-)
-print(f"  ✓ Uploaded to Hub")
-
-print(f"\n{'='*70}")
-print("DONE! Execution feedback loop complete.")
-print("=" * 70)

training/integration_real.py

@@ -1,362 +0,0 @@
-"""Real ACO Integration Test with smolagents CodeAgent.
-
-Connects the ACO optimizer to a real smolagents CodeAgent running on
-SWE-bench tasks. The optimizer intercepts:
-  - Model selection (route to tier)
-  - Tool calls (gate unnecessary tools)
-  - Retries (optimize recovery)
-  - Verification (selective verifier)
-  - Doom detection (early termination)
-
-This is a real integration test — it runs actual LLM calls and tool
-executions, not simulated ones.
-
-Usage:
-  uv run --with smolagents --with transformers --with torch --with datasets
-    --with scikit-learn --with numpy --with xgboost --with huggingface_hub
-    integration_real.py
-
-Requirements: HF_TOKEN set, openai or hf-inference access.
-"""
-import json, os, sys, time, traceback
-from typing import Dict, List, Optional, Any
-from dataclasses import dataclass
-
-# ── ACO Import ──────────────────────────────────────────────────
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-from aco.optimizer import ACOOptimizer
-from aco.config import ACOConfig, RoutingPolicy
-
-# ── smolagents Import ───────────────────────────────────────────
-from smolagents import CodeAgent, LiteLLMModel, tool, ToolCallingAgent
-from datasets import load_dataset
-
-print("=" * 70)
-print("ACO + SMOLAGENTS REAL INTEGRATION TEST")
-print("=" * 70)
-
-# ── 1. Set up models ────────────────────────────────────────────
-# We configure models as LiteLLM targets that the ACO can route between
-
-TIER_MODELS = {
-    1: "openai/gpt-4o-mini",          # cheap, fast
-    2: "openai/gpt-4o",               # medium
-    3: "anthropic/claude-3-5-sonnet",  # strong
-    4: "anthropic/claude-3-opus",     # frontier
-}
-
-TIER_COST_PER_1K = {
-    1: 0.00015,  # gpt-4o-mini input
-    2: 0.0025,   # gpt-4o input
-    3: 0.003,    # claude sonnet
-    4: 0.015,    # claude opus
-}
-
-# ── 2. Set up ACO ───────────────────────────────────────────────
-config = ACOConfig()
-config.routing_policy = RoutingPolicy(
-    routing_mode="cascade",
-    feedback_escalation=True,
-    max_retries=2,
-    max_cost_per_task=2.0,
-    safety_threshold=0.5,
-)
-config.enable_doom_detector = True
-config.enable_meta_tools = False
-
-# Download router bundle
-from huggingface_hub import hf_hub_download
-try:
-    bundle_path = hf_hub_download(
-        repo_id="narcolepticchicken/agent-cost-optimizer",
-        filename="router_models/router_bundle_v10_fixed.pkl",
-    )
-    config.router_model_path = bundle_path
-    print(f"✓ Router model loaded from Hub")
-except Exception as e:
-    print(f"⚠ Router bundle not available: {e}")
-    config.router_model_path = None
-
-optimizer = ACOOptimizer(config)
-
-# ── 3. ACO-Aware Model Provider ────────────────────────────────
-# This is the key integration: instead of the agent having one model,
-# ACO decides which model to use at each agent step.
-
-class ACOModelProvider:
-    """Wraps LiteLLM with ACO routing decisions."""
-    
-    def __init__(self, optimizer: ACOOptimizer):
-        self.optimizer = optimizer
-        self.current_model = None
-        self.current_tier = None
-        self.step_count = 0
-        self.total_cost = 0.0
-        self.total_tokens = 0
-        self.cached_models = {}
-    
-    def get_model(self, task_description: str, is_first_call: bool = False) -> LiteLLMModel:
-        """Get the optimal model for the current context."""
-        # ACO decides which model
-        if is_first_call or self.current_model is None:
-            start = self.optimizer.start_run(task_description)
-            tier = start["routing"]["tier"]
-            model_id = TIER_MODELS.get(tier, TIER_MODELS[4])
-            self.current_tier = tier
-            self.current_model = model_id
-            print(f"  ACO routed to: {model_id} (tier {tier}, "
-                  f"confidence={start['routing']['confidence']:.2f})")
-        
-        # Create/reuse LiteLLM model
-        if self.current_model not in self.cached_models:
-            self.cached_models[self.current_model] = LiteLLMModel(
-                model_id=self.current_model,
-                api_key=os.environ.get("OPENAI_API_KEY"),
-            )
-        
-        return self.cached_models[self.current_model]
-    
-    def escalate(self, reason: str):
-        """Escalate to a stronger model."""
-        if self.current_tier < 4:
-            self.current_tier += 1
-            self.current_model = TIER_MODELS[self.current_tier]
-            # Clear cache to force new model
-            if self.current_model in self.cached_models:
-                del self.cached_models[self.current_model]
-            print(f"  ⬆ Escalated to tier {self.current_tier}: {self.current_model}")
-            print(f"    Reason: {reason}")
-    
-    def record_step(self, input_tokens: int, output_tokens: int, success: bool,
-                    tool_calls: List = None):
-        """Record telemetry for a completed step."""
-        self.step_count += 1
-        cost = input_tokens * TIER_COST_PER_1K.get(self.current_tier, 0.003) / 1000
-        cost += output_tokens * TIER_COST_PER_1K.get(self.current_tier, 0.015) / 1000
-        self.total_cost += cost
-        self.total_tokens += input_tokens + output_tokens
-        
-        self.optimizer.record_step(
-            model_call={
-                "model_id": self.current_model,
-                "input_tokens": input_tokens,
-                "output_tokens": output_tokens,
-                "cost": cost,
-                "latency_ms": 2000,
-            },
-            tool_calls=tool_calls or [],
-            context_size=input_tokens + output_tokens,
-            retry_num=0,
-        )
-        
-        # Check for doom
-        doom = self.optimizer.check_doom(current_cost=self.total_cost)
-        if doom.doomed:
-            print(f"  ⚠ DOOM DETECTED: {doom.reasoning}")
-            return doom
-        
-        return None
-
-# ── 4. Define Agent with ACO integration ─────────────────────────
-# We use smolagents CodeAgent which can write and execute Python code
-
-# ACO-gated tools with cost tracking
-class ACOGatedTools:
-    def __init__(self, optimizer, provider):
-        self.optimizer = optimizer
-        self.provider = provider
-        self.call_count = {}
-        self.skips = 0
-    
-    def should_call(self, tool_name: str) -> bool:
-        """Gate a tool call through ACO."""
-        decision = self.optimizer.gate_tool(tool_name, {})
-        if decision.action == "skip":
-            self.skips += 1
-            return False
-        self.call_count[tool_name] = self.call_count.get(tool_name, 0) + 1
-        return True
-
-# Define tools
-class AgentTools:
-    """Real SWE-bench tools for the agent."""
-    
-    def __init__(self, repo_path: str = "/tmp/repo"):
-        self.repo_path = repo_path
-        self.file_contents = {}
-        self.exec_results = []
-    
-    def read_file(self, path: str) -> str:
-        """Read a file from the repository."""
-        full_path = os.path.join(self.repo_path, path)
-        if os.path.exists(full_path):
-            with open(full_path, 'r') as f:
-                return f.read()
-        return f"ERROR: File not found: {path}"
-    
-    def write_file(self, path: str, content: str) -> str:
-        """Write content to a file."""
-        full_path = os.path.join(self.repo_path, path)
-        os.makedirs(os.path.dirname(full_path), exist_ok=True)
-        with open(full_path, 'w') as f:
-            f.write(content)
-        return f"Successfully wrote to {path}"
-    
-    def execute_command(self, command: str) -> str:
-        """Execute a shell command."""
-        import subprocess
-        try:
-            result = subprocess.run(
-                command, shell=True, capture_output=True, text=True,
-                cwd=self.repo_path, timeout=60
-            )
-            return result.stdout + result.stderr
-        except subprocess.TimeoutExpired:
-            return "ERROR: Command timed out"
-        except Exception as e:
-            return f"ERROR: {e}"
-
-# ── 5. Run on SWE-bench tasks ───────────────────────────────────
-print("\n[5] Running on SWE-bench Verified tasks...")
-
-# Load SWE-bench tasks
-try:
-    swe_dataset = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
-    tasks = list(swe_dataset.select(range(3)))  # Start with 3 tasks
-    print(f"  Loaded {len(tasks)} SWE-bench tasks")
-except Exception as e:
-    print(f"  ⚠ SWE-bench not accessible: {e}")
-    # Fallback: mock task
-    tasks = [{"instance_id": "mock-task", "problem_statement": "Fix the bug in the calculator: 2+2 should return 4, but returns 5."}]
-
-results = []
-for i, task in enumerate(tasks):
-    print(f"\n{'─'*60}")
-    print(f"TASK {i+1}/{len(tasks)}: {task.get('instance_id', 'unknown')}")
-    print(f"  Problem: {task.get('problem_statement', task.get('problem', 'N/A'))[:150]}...")
-    
-    # Initialize ACO
-    provider = ACOModelProvider(optimizer)
-    tools = AgentTools()
-    
-    start_time = time.time()
-    
-    try:
-        # Create agent with ACO-routed model
-        model = provider.get_model(task["problem_statement"], is_first_call=True)
-        
-        agent = CodeAgent(
-            tools=[tools.read_file, tools.write_file, tools.execute_command],
-            model=model,
-            max_steps=5,  # Limit steps for demo
-        )
-        
-        # Run the agent
-        prompt = f"""You are a software engineer fixing a bug. 
-        
-Problem: {task['problem_statement']}
-
-Please analyze the problem, find the bug, and fix it. 
-Use the available tools to read files, write fixes, and test them.
-"""
-        
-        result = agent.run(prompt)
-        
-        elapsed = time.time() - start_time
-        
-        print(f"  Result: {str(result)[:200]}...")
-        print(f"  Time: {elapsed:.1f}s")
-        print(f"  Cost: ${provider.total_cost:.4f}")
-        print(f"  Tokens: {provider.total_tokens}")
-        print(f"  Steps: {provider.step_count}")
-        
-        results.append({
-            "task_id": task.get("instance_id", f"task-{i}"),
-            "success": "error" not in str(result).lower(),
-            "time_s": round(elapsed, 1),
-            "cost": round(provider.total_cost, 4),
-            "tokens": provider.total_tokens,
-            "steps": provider.step_count,
-            "tier_used": provider.current_tier,
-            "model_used": provider.current_model,
-            "output": str(result)[:500],
-        })
-        
-        # End ACO run
-        optimizer.end_run(
-            success="error" not in str(result).lower(),
-            outcome="completed",
-            artifacts=[task.get("instance_id", "unknown")],
-        )
-        
-    except Exception as e:
-        elapsed = time.time() - start_time
-        print(f"  ⚠ ERROR: {e}")
-        traceback.print_exc()
-        
-        results.append({
-            "task_id": task.get("instance_id", f"task-{i}"),
-            "success": False,
-            "time_s": round(elapsed, 1),
-            "error": str(e),
-            "cost": provider.total_cost if provider else 0,
-            "tokens": provider.total_tokens if provider else 0,
-        })
-        
-        optimizer.end_run(
-            success=False,
-            outcome="failed",
-            failure_tags=["agent_error"],
-        )
-
-# ── 6. Summary ──────────────────────────────────────────────────
-print(f"\n{'='*70}")
-print("INTEGRATION TEST RESULTS")
-print(f"{'='*70}")
-
-total_cost = sum(r.get("cost", 0) for r in results)
-total_tokens = sum(r.get("tokens", 0) for r in results)
-successes = sum(1 for r in results if r.get("success"))
-avg_time = sum(r.get("time_s", 0) for r in results) / max(len(results), 1)
-
-print(f"""
-  Tasks attempted: {len(results)}
-  Successes:       {successes} ({successes/len(results)*100:.0f}%)
-  Total cost:      ${total_cost:.4f}
-  Total tokens:    {total_tokens:,}
-  Avg time/task:   {avg_time:.1f}s
-
-ACO Stats:
-  {json.dumps(optimizer.get_stats(), indent=2)}
-""")
-
-# Save results
-with open("/tmp/aco_integration_results.json", "w") as f:
-    json.dump({
-        "results": results,
-        "summary": {
-            "n": len(results),
-            "successes": successes,
-            "total_cost": round(total_cost, 4),
-            "avg_time": round(avg_time, 1),
-            "total_tokens": total_tokens,
-        },
-        "aco_stats": optimizer.get_stats(),
-    }, f, indent=2)
-
-from huggingface_hub import HfApi
-api = HfApi()
-try:
-    api.upload_file(
-        path_or_fileobj="/tmp/aco_integration_results.json",
-        path_in_repo="eval/integration_test_results.json",
-        repo_id="narcolepticchicken/agent-cost-optimizer",
-        repo_type="model",
-    )
-    print("  ✓ Results uploaded to Hub")
-except Exception as e:
-    print(f"  ⚠ Upload failed: {e}")
-
-print(f"\n{'='*70}")
-print("DONE! Real integration test complete.")
-print("=" * 70)

training/repro_test.py

@@ -1,190 +0,0 @@
-"""Reproducibility test: v10+feedback router × 5 runs.
-
-Tests whether the 84.8% success / 36.4% cost reduction is stable or a fluke.
-Uses the v10 XGBoost bundle trained on SWE-Router data with different
-random seeds for the calibration/perturbation.
-
-Also tests sensitivity to the success threshold parameter (0.4, 0.5, 0.6).
-"""
-import json, pickle, random, numpy as np
-from collections import defaultdict
-from datasets import load_dataset
-from sklearn.isotonic import IsotonicRegression
-
-# ── Router v10 features ─────────────────────────────────────────
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor","implement",
-    "test","compile","runtime","segfault","thread","async","class","module","import","error","traceback"]
-CRITICAL_KW = ["critical","production","urgent","emergency","live","deployed","safety","security"]
-SIMPLE_KW = ["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-RESEARCH_KW = ["research","investigate","compare","analyze","survey","paper"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","migrate","pipeline","deploy","architecture"]
-
-FEAT_KEYS = sorted(['req_len','num_words','has_code','n_code','has_legal','has_research',
-    'has_tool','has_critical','has_simple','has_long','has_math','has_error_msg',
-    'has_file_path','n_lines','has_version','has_add','has_fix','has_change',
-    'has_remove','has_test','has_doc','has_see_also','has_steps_to_reproduce'])
-
-TIER_TO_MODEL = {1:'deepseek-v4-flash',2:'gpt-5-mini',3:'gemini-2.5-pro',
-                  4:'claude-opus-4.7',5:'gemini-3-pro'}
-MODELS = ["deepseek-v4-flash","gpt-5-nano","gpt-5-mini","deepseek-v3.2",
-           "gemini-2.5-pro","claude-opus-4.7","gpt-5.2","gemini-3-pro"]
-
-def extract(text):
-    r = text.lower()
-    return np.array([float({
-        'req_len':len(text),'num_words':len(text.split()),
-        'has_code':int(any(k in r for k in CODE_KW)),
-        'n_code':sum(1 for k in CODE_KW if k in r),
-        'has_legal':int(any(k in r for k in["contract","legal","compliance"])),
-        'has_research':int(any(k in r for k in RESEARCH_KW)),
-        'has_tool':int(any(k in r for k in TOOL_KW)),
-        'has_critical':int(any(k in r for k in CRITICAL_KW)),
-        'has_simple':int(any(k in r for k in SIMPLE_KW)),
-        'has_long':int(any(k in r for k in LONG_KW)),
-        'has_math':int(any(k in r for k in["calculate","compute","solve","equation"])),
-        'has_error_msg':int('error'in r or'traceback'in r or'exception'in r),
-        'has_file_path':int('/'in r),
-        'n_lines':text.count('\n')+1,
-        'has_version':int('version'in r or'update'in r),
-        'has_add':int('add'in r or'new'in r or'create'in r),
-        'has_fix':int('fix'in r or'bug'in r or'issue'in r),
-        'has_change':int('change'in r or'modify'in r),
-        'has_remove':int('remove'in r or'delete'in r),
-        'has_test':int('test'in r or'spec'in r),
-        'has_doc':int('doc'in r or'readme'in r),
-        'has_see_also':int('see also'in r or'related'in r),
-        'has_steps_to_reproduce':int('reproduce'in r or'steps'in r),
-    }.get(k,0)) for k in FEAT_KEYS], dtype=np.float32)
-
-# ── Load data ────────────────────────────────────────────────────
-print("Loading data...")
-traces = defaultdict(dict)
-for m in MODELS:
-    ds = load_dataset(f"SWE-Router/swebench-verified-{m}", split="test")
-    for row in ds:
-        traces[row["instance_id"]][m] = {"resolved":row["resolved"], "cost":float(row["instance_cost"])}
-N = len(traces)
-print(f"  {N} tasks")
-
-# ── Load router ──────────────────────────────────────────────────
-from huggingface_hub import hf_hub_download
-bundle = pickle.load(open(hf_hub_download(
-    repo_id="narcolepticchicken/agent-cost-optimizer",
-    filename="router_models/router_bundle_v10_fixed.pkl"), 'rb'))
-tier_clfs_raw = {int(k):v for k,v in bundle['tier_clfs'].items()}
-print(f"  Router: {len(tier_clfs_raw)} tier classifiers")
-
-# ── Run 5 reproducibility trials ─────────────────────────────────
-SEEDS = [42, 123, 456, 789, 1024]
-THRESHOLDS = [0.4, 0.5, 0.6]
-
-print(f"\n{'='*75}")
-print(f"REPRODUCIBILITY TEST: v10+feedback × {len(SEEDS)} seeds × {len(THRESHOLDS)} thresholds")
-print(f"{'='*75}")
-
-all_results = []
-
-for seed in SEEDS:
-    random.seed(seed)
-    np.random.seed(seed)
-    
-    for thresh in THRESHOLDS:
-        total_cost = 0.0
-        resolved = 0
-        escalated = 0
-        tier1_only = 0
-        
-        for tid, tt in traces.items():
-            problem = next(iter(tt.values())).get("problem","")
-            x = extract(problem).reshape(1,-1)
-            
-            # Route
-            route_tier = 5
-            for t in range(1,6):
-                if t in tier_clfs_raw:
-                    p = tier_clfs_raw[t].predict_proba(x)[0,1]
-                    if p >= thresh:
-                        route_tier = t
-                        break
-            
-            # Execute + escalate
-            task_cost = 0.0
-            task_resolved = False
-            task_escalated = False
-            
-            for tier in range(route_tier, 6):
-                model = TIER_TO_MODEL[tier]
-                mt = tt.get(model, {})
-                task_cost += mt.get("cost", 0.30)
-                
-                if mt.get("resolved", False):
-                    task_resolved = True
-                    break
-                
-                if tier > route_tier:
-                    task_escalated = True
-            
-            total_cost += task_cost
-            resolved += int(task_resolved)
-            escalated += int(task_escalated)
-            if route_tier == 1 and not task_escalated:
-                tier1_only += 1
-        
-        avg_cost = total_cost / N
-        success_rate = resolved / N
-        frontier_cost = sum(tt.get("claude-opus-4.7",{}).get("cost",0.317) for tt in traces.values()) / N
-        cost_reduction = (1 - avg_cost / frontier_cost) * 100
-        
-        all_results.append({
-            "seed": seed, "threshold": thresh,
-            "resolved": resolved, "rate": round(success_rate, 4),
-            "avg_cost": round(avg_cost, 4), "cost_reduction": round(cost_reduction, 1),
-            "escalated": escalated, "tier1_only": tier1_only,
-        })
-
-# ── Print ────────────────────────────────────────────────────────
-print(f"\n{'Seed':<8} {'Thr':>5} {'Resolved':>10} {'Rate':>8} {'AvgCost':>10} {'CostRed':>10} {'Escal.':>8} {'T1Only':>8}")
-print("-" * 75)
-
-for r in all_results:
-    print(f"  {r['seed']:<8} {r['threshold']:>5.1f} {r['resolved']:>10} {r['rate']*100:>7.1f}% ${r['avg_cost']:>9.4f} {r['cost_reduction']:>9.1f}% {r['escalated']:>8} {r['tier1_only']:>8}")
-
-# Summary stats
-rates = [r["rate"] for r in all_results]
-costs = [r["avg_cost"] for r in all_results]
-crs = [r["cost_reduction"] for r in all_results]
-
-print(f"\n{'─'*75}")
-print(f"SUMMARY:")
-print(f"  Success rate:   {np.mean(rates)*100:.2f}% ± {np.std(rates)*100:.2f}%  (range: {np.min(rates)*100:.1f}-{np.max(rates)*100:.1f}%)")
-print(f"  Avg cost:       ${np.mean(costs):.4f} ± ${np.std(costs):.4f}")
-print(f"  Cost reduction: {np.mean(crs):.1f}% ± {np.std(crs):.1f}%")
-print(f"  Frontier:       78.2% @ $0.3167")
-
-# Best per threshold
-print(f"\n{'─'*75}")
-print(f"BEST PER THRESHOLD:")
-for thresh in THRESHOLDS:
-    subset = [r for r in all_results if r["threshold"] == thresh]
-    best = max(subset, key=lambda r: r["rate"])
-    print(f"  Threshold {thresh}: {best['rate']*100:.1f}% @ ${best['avg_cost']:.4f} ({best['cost_reduction']:.1f}% reduction)  seed={best['seed']}")
-
-# Best overall
-best = max(all_results, key=lambda r: r["cost_reduction"] + r["rate"]*100 - 78.2)
-print(f"\n  🏆 BEST OVERALL: seed={best['seed']}, thresh={best['threshold']}")
-print(f"     {best['rate']*100:.1f}% success @ ${best['avg_cost']:.4f} = {best['cost_reduction']:.1f}% cost reduction")
-
-# Save
-with open("/tmp/repro_results.json","w") as f:
-    json.dump({"results": all_results, "summary": {
-        "mean_rate": round(np.mean(rates),4), "std_rate": round(np.std(rates),4),
-        "mean_cost": round(np.mean(costs),4), "std_cost": round(np.std(costs),4),
-        "mean_cost_reduction": round(np.mean(crs),1), "std_cost_reduction": round(np.std(crs),1),
-    }}, f, indent=2)
-
-from huggingface_hub import HfApi
-HfApi().upload_file(path_or_fileobj="/tmp/repro_results.json",
-    path_in_repo="eval/reproducibility_results.json",
-    repo_id="narcolepticchicken/agent-cost-optimizer", repo_type="model")
-print("\n✓ eval/reproducibility_results.json")

training/repro_v2.py

@@ -1,119 +0,0 @@
-"""Reproducibility test v2: v10+feedback router × 5 seeds.
-
-Uses the CORRECT 14 features from the v10 fixed bundle.
-"""
-import json, pickle, random, numpy as np
-from collections import defaultdict
-from datasets import load_dataset
-
-# ── CORRECT v10 features (14, not 23) ────────────────────────────
-FEAT_KEYS = ['has_add', 'has_change', 'has_code', 'has_critical', 'has_doc',
-             'has_error_msg', 'has_file_path', 'has_fix', 'has_simple', 'has_test',
-             'n_code', 'n_lines', 'num_words', 'req_len']
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor","implement",
-    "test","compile","runtime","segfault","thread","async","class","module","import","error","traceback"]
-CRITICAL_KW = ["critical","production","urgent","emergency","live","deployed","safety","security"]
-SIMPLE_KW = ["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-
-def extract(text):
-    r = text.lower()
-    return np.array([float({
-        'req_len':len(text),'num_words':len(text.split()),
-        'has_code':int(any(k in r for k in CODE_KW)),
-        'n_code':sum(1 for k in CODE_KW if k in r),
-        'has_critical':int(any(k in r for k in CRITICAL_KW)),
-        'has_simple':int(any(k in r for k in SIMPLE_KW)),
-        'has_error_msg':int('error'in r or'traceback'in r or'exception'in r),
-        'has_file_path':int('/'in r),
-        'n_lines':text.count('\n')+1,
-        'has_add':int('add'in r or'new'in r or'create'in r),
-        'has_fix':int('fix'in r or'bug'in r or'issue'in r),
-        'has_change':int('change'in r or'modify'in r),
-        'has_test':int('test'in r or'spec'in r),
-        'has_doc':int('doc'in r or'readme'in r),
-    }.get(k,0)) for k in FEAT_KEYS], dtype=np.float32)
-
-TIER_TO_MODEL = {1:'deepseek-v4-flash',2:'gpt-5-mini',3:'gemini-2.5-pro',
-                  4:'claude-opus-4.7',5:'gemini-3-pro'}
-MODELS = ["deepseek-v4-flash","gpt-5-nano","gpt-5-mini","deepseek-v3.2",
-           "gemini-2.5-pro","claude-opus-4.7","gpt-5.2","gemini-3-pro"]
-
-from huggingface_hub import hf_hub_download
-
-print("Loading data + router...")
-traces = defaultdict(dict)
-for m in MODELS:
-    ds = load_dataset(f"SWE-Router/swebench-verified-{m}", split="test")
-    for row in ds:
-        traces[row["instance_id"]][m] = {"resolved":row["resolved"], "cost":float(row["instance_cost"])}
-N = len(traces)
-
-bundle = pickle.load(open(hf_hub_download(
-    repo_id="narcolepticchicken/agent-cost-optimizer",
-    filename="router_models/router_bundle_v10_fixed.pkl"), 'rb'))
-tier_clfs_raw = {int(k):v for k,v in bundle['tier_clfs'].items()}
-
-print(f"  {N} tasks, {len(tier_clfs_raw)} classifiers, {len(FEAT_KEYS)} features")
-
-# ── Run trials ───────────────────────────────────────────────────
-SEEDS = [42, 123, 456, 789, 1024]
-THRESHOLDS = [0.4, 0.5, 0.6]
-
-results = []
-for seed in SEEDS:
-    random.seed(seed); np.random.seed(seed)
-    for thresh in THRESHOLDS:
-        tc, res, esc, t1only = 0.0, 0, 0, 0
-        for tid, tt in traces.items():
-            problem = next(iter(tt.values())).get("problem","")
-            x = extract(problem).reshape(1,-1)
-            
-            route_tier = 5
-            for t in range(1,6):
-                if t in tier_clfs_raw:
-                    p = tier_clfs_raw[t].predict_proba(x)[0,1]
-                    if p >= thresh: route_tier = t; break
-            
-            task_cost, task_resolved, task_esc = 0.0, False, False
-            for tier in range(route_tier, 6):
-                mt = tt.get(TIER_TO_MODEL[tier], {})
-                task_cost += mt.get("cost",0.30)
-                if mt.get("resolved",False): task_resolved = True; break
-                if tier > route_tier: task_esc = True
-            
-            tc += task_cost; res += task_resolved; esc += task_esc
-            if route_tier == 1 and not task_esc: t1only += 1
-        
-        fc = sum(tt.get("claude-opus-4.7",{}).get("cost",0.317) for tt in traces.values())/N
-        cr = (1-(tc/N)/fc)*100
-        results.append({"seed":seed,"thresh":thresh,"resolved":res,"rate":res/N,
-                       "avg_cost":tc/N,"cost_red":cr,"escalated":esc,"t1only":t1only})
-
-# Print
-print(f"\n{'Seed':<8} {'Thr':>4} {'Resolved':>10} {'Rate':>8} {'AvgCost':>10} {'CostRed':>9} {'Esc':>6} {'T1':>6}")
-print("-"*68)
-for r in results:
-    print(f"  {r['seed']:<8} {r['thresh']:.1f}  {r['resolved']:>10} {r['rate']*100:>7.1f}% ${r['avg_cost']:>9.4f} {r['cost_red']:>8.1f}% {r['escalated']:>6} {r['t1only']:>6}")
-
-rates = [r["rate"] for r in results]
-costs = [r["avg_cost"] for r in results]
-crs = [r["cost_red"] for r in results]
-print(f"\n  Mean: {np.mean(rates)*100:.1f}% ±{np.std(rates)*100:.1f}%  ${np.mean(costs):.4f}±{np.std(costs):.4f}  {np.mean(crs):.1f}%±{np.std(crs):.1f}% reduction")
-
-# Best per threshold
-for thresh in THRESHOLDS:
-    subset = [r for r in results if r["thresh"]==thresh]
-    best = max(subset, key=lambda r: r["rate"])
-    print(f"  Best@thresh={thresh}: {best['rate']*100:.1f}% @ ${best['avg_cost']:.4f} ({best['cost_red']:.1f}%) seed={best['seed']}")
-
-best = max(results, key=lambda r: r["cost_red"]+r["rate"]*100-78.2)
-print(f"\n  BEST: {best['rate']*100:.1f}% @ ${best['avg_cost']:.4f} ({best['cost_red']:.1f}% reduction)")
-
-with open("/tmp/repro_v2.json","w") as f:
-    json.dump({"results":results,"means":{"rate":np.mean(rates),"cost":np.mean(costs),"cost_red":np.mean(crs)}},f,indent=2)
-from huggingface_hub import HfApi
-HfApi().upload_file(path_or_fileobj="/tmp/repro_v2.json",
-    path_in_repo="eval/repro_v2.json",
-    repo_id="narcolepticchicken/agent-cost-optimizer", repo_type="model")
-print("\n✓ eval/repro_v2.json")

training/router_integration_benchmark.py

@@ -1,237 +0,0 @@
-#!/usr/bin/env python3
-"""
-Standalone integration: Replace heuristic router with trained XGBoost router.
-
-This script:
-1. Generates 2K eval traces (same as standalone_eval_v2.py)
-2. Runs the full benchmark with the TRAINED router replacing _route_learned()
-3. Compares: heuristic, trained, and oracle routers
-"""
-
-import json, os, sys, random, uuid, pickle
-import numpy as np
-from datetime import datetime, timedelta
-from dataclasses import dataclass, field
-from enum import Enum
-from collections import defaultdict
-from typing import Dict, List, Optional, Any
-
-# ─── Load trained models ─────────────────────────────────────────────
-print("="*80)
-print("ACO TRAINED ROUTER - INTEGRATION BENCHMARK")
-print("="*80)
-
-import xgboost as xgb
-
-MODEL_DIR = "/app/router_models"
-feat_keys = json.load(open(f"{MODEL_DIR}/feat_keys.json"))
-tier_config = json.load(open(f"{MODEL_DIR}/tier_config.json"))
-TIER_COST = {int(k):v for k,v in tier_config["tier_cost"].items()}
-TIER_STR = {int(k):v for k,v in tier_config["tier_str"].items()}
-TASK_FLOOR = tier_config["task_floor"]
-
-print(f"\n[1] Loading trained router models...")
-tier_clfs = {}
-for tier in range(1, 6):
-    clf = xgb.XGBClassifier()
-    clf.load_model(f"{MODEL_DIR}/tier_{tier}_success.json")
-    tier_clfs[tier] = clf
-    print(f"  Loaded tier_{tier}_success.json")
-
-# ─── Feature extraction (must match training) ────────────────────────
-TASK_TYPES = ["quick_answer","coding","research","document_drafting",
-              "legal_regulated","tool_heavy","retrieval_heavy",
-              "long_horizon","unknown_ambiguous"]
-TT2IDX = {t:i for i,t in enumerate(TASK_TYPES)}
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-           "implement","test","compile","runtime","class","module","async","thread"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare","analyze","survey"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate","pipeline","deploy"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-
-def extract_features(request, task_type, difficulty=3):
-    r = request.lower()
-    f = {"req_len": len(request), "num_words": len(request.split()),
-        "has_code": int(any(k in r for k in CODE_KW)),
-        "n_code": sum(1 for k in CODE_KW if k in r),
-        "has_legal": int(any(k in r for k in LEGAL_KW)),
-        "n_legal": sum(1 for k in LEGAL_KW if k in r),
-        "has_research": int(any(k in r for k in RESEARCH_KW)),
-        "n_research": sum(1 for k in RESEARCH_KW if k in r),
-        "has_tool": int(any(k in r for k in TOOL_KW)),
-        "n_tool": sum(1 for k in TOOL_KW if k in r),
-        "has_long": int(any(k in r for k in LONG_KW)),
-        "has_math": int(any(k in r for k in MATH_KW)),
-        "tt_idx": TT2IDX.get(task_type, 8), "difficulty": difficulty}
-    for tt in TASK_TYPES:
-        f[f"tt_{tt}"] = int(task_type == tt)
-    return f
-
-def f2v(feats):
-    return np.array([float(feats.get(k, 0.0)) for k in feat_keys], dtype=np.float32)
-
-# ─── Routing Functions ────────────────────────────────────────────────
-def route_trained(request, task_type, difficulty):
-    """Trained router: per-tier P(success) + safety floor + asymmetric cost."""
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    floor = TASK_FLOOR.get(task_type, 2)
-    
-    best_tier = floor; best_score = float("inf")
-    for tier in range(floor, 6):
-        p_success = tier_clfs[tier].predict_proba(x)[0, 1]
-        p_fail = 1.0 - p_success
-        cost_norm = TIER_COST[tier] / TIER_COST[5]
-        score = p_fail * 5.0 + cost_norm * 1.0  # asymmetric: 5x underkill penalty
-        if score < best_score:
-            best_score = score; best_tier = tier
-    return best_tier
-
-def route_heuristic(task_type, difficulty):
-    """Original heuristic router: difficulty + 1."""
-    return min(difficulty + 1, 5)
-
-def route_frontier():
-    return 4
-
-def route_cascade_trained(request, task_type, difficulty):
-    """Cascade: start at floor, escalate if P(success) < threshold."""
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    floor = TASK_FLOOR.get(task_type, 2)
-    
-    for tier in range(floor, 6):
-        p_success = tier_clfs[tier].predict_proba(x)[0, 1]
-        if p_success >= 0.65:
-            return tier
-    return 4  # fallback to frontier
-
-# ─── Generate Evaluation Traces ────────────────────────────────────────
-print("\n[2] Generating 2K evaluation traces (different seed from training)...")
-
-class TaskType(Enum):
-    QUICK_ANSWER="quick_answer"; CODING="coding"; RESEARCH="research"
-    DOCUMENT_DRAFTING="document_drafting"; LEGAL_REGULATED="legal_regulated"
-    TOOL_HEAVY="tool_heavy"; RETRIEVAL_HEAVY="retrieval_heavy"
-    LONG_HORIZON="long_horizon"; UNKNOWN_AMBIGUOUS="unknown_ambiguous"
-
-TASK_TEMPLATES_EVAL = {
-    "quick_answer":["What is the capital of France?","Explain quantum computing briefly.","What is 237*452?"],
-    "coding":["Write a Python function to reverse a linked list.","Fix the bug in this React component.",
-              "Implement LRU cache in Go.","Debug segfault in C++ thread pool."],
-    "research":["Research latest transformer advances.","Find sources comparing LoRA and full FT.",
-                "Investigate data center climate impact."],
-    "document_drafting":["Draft project proposal for ML pipeline.","Write email to team about deployment."],
-    "legal_regulated":["Review this contract for liability clauses.","Check GDPR compliance for data pipeline.",
-                       "Draft privacy policy section."],
-    "tool_heavy":["Search open issues and create summary.","Fetch API docs and generate client code."],
-    "retrieval_heavy":["Answer based on 50-page document.","Find all payment processing mentions."],
-    "long_horizon":["Plan 3-month roadmap.","Orchestrate multi-region deployment."],
-    "unknown_ambiguous":["Help me with this thing.","I need something about the server."],
-}
-
-def tsp(tier, diff):
-    s = {1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}[tier]
-    return s ** (diff * 0.6)
-
-eval_rng = random.Random(999)  # DIFFERENT seed for eval
-eval_traces = []
-for i in range(2000):
-    tt = eval_rng.choice(list(TASK_TEMPLATES_EVAL.keys()))
-    diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    tier_out = {t: eval_rng.random() < tsp(t, diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    req = eval_rng.choice(TASK_TEMPLATES_EVAL[tt])
-    eval_traces.append({"tt":tt,"diff":diff,"opt":opt,"tier_out":tier_out,"req":req})
-
-print(f"  Generated {len(eval_traces)} eval traces")
-
-# ─── Evaluate All Routers ─────────────────────────────────────────────
-print("\n[3] Evaluating all routers on 2K traces...")
-
-def eval_router(name, route_fn):
-    succ = 0; cost = 0.0; unsafe = 0; fd = 0
-    td = defaultdict(int)
-    for t in eval_traces:
-        pred = route_fn(t)
-        td[pred] += 1
-        if t["tier_out"].get(pred, False):
-            succ += 1
-        elif pred < t["opt"]:
-            unsafe += 1
-        else:
-            fd += 1
-        cost += TIER_COST[pred]
-    n = len(eval_traces)
-    return {"success":succ/n, "avg_cost":cost/n, "unsafe_rate":unsafe/n,
-            "false_done":fd/n, "tier_dist":dict(td)}
-
-routers = {
-    "always_frontier": lambda t: 4,
-    "always_cheap": lambda t: 1,
-    "heuristic_diff+1": lambda t: min(t["diff"]+1, 5),
-    "heuristic_floor": lambda t: TASK_FLOOR.get(t["tt"], 2),
-    "trained_asymmetric": lambda t: route_trained(t["req"], t["tt"], t["diff"]),
-    "trained_cascade_t0.65": lambda t: route_cascade_trained(t["req"], t["tt"], t["diff"]),
-    "oracle": lambda t: t["opt"],
-}
-
-results = {}
-for name, fn in routers.items():
-    results[name] = eval_router(name, fn)
-    r = results[name]
-    fc = results["always_frontier"]["avg_cost"]
-    cr = (1 - r["avg_cost"]/fc)*100
-    print(f"  {name:<25} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}% unsafe={r['unsafe_rate']:.3f}")
-
-# ─── Per-Task-Type Breakdown ──────────────────────────────────────────
-print("\n\n[4] Per-task-type breakdown (trained vs heuristic vs frontier)...")
-for tt in sorted(set(t["tt"] for t in eval_traces)):
-    tt_traces = [t for t in eval_traces if t["tt"] == tt]
-    n_tt = len(tt_traces)
-    if n_tt == 0: continue
-    
-    for rname, rfn in [("frontier", lambda t:4),
-                       ("heuristic", lambda t:min(t["diff"]+1,5)),
-                       ("trained", lambda t:route_trained(t["req"],t["tt"],t["diff"]))]:
-        succ = sum(1 for t in tt_traces if t["tier_out"].get(rfn(t), False))
-        cost = sum(TIER_COST[rfn(t)] for t in tt_traces)
-        sr = succ/n_tt; ac = cost/n_tt
-        if rname == "frontier":
-            print(f"\n  {tt} (n={n_tt}):")
-        print(f"    {rname:<12} success={sr:.3f} cost={ac:.4f}")
-
-# ─── Final Comparison ─────────────────────────────────────────────────
-print(f"\n\n{'='*80}")
-print("FINAL INTEGRATION BENCHMARK")
-print(f"{'='*80}")
-print(f"\n{'Router':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*75)
-fc = results["always_frontier"]["avg_cost"]
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    cr = (1 - r["avg_cost"]/fc)*100
-    print(f"{name:<25} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-# Quality/cost frontier
-print("\nPARETO FRONTIER:")
-pareto = []
-for name, r in results.items():
-    if name == "always_cheap": continue
-    dominated = False
-    for name2, r2 in results.items():
-        if name == name2: continue
-        if r2["success"] >= r["success"] and r2["avg_cost"] <= r["avg_cost"]:
-            if r2["success"] > r["success"] or r2["avg_cost"] < r["avg_cost"]:
-                dominated = True; break
-    if not dominated:
-        pareto.append((name, r))
-        cr = (1 - r["avg_cost"]/fc)*100
-        print(f"  {name:<25} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}%")
-
-print(f"\n\nDONE!")

training/router_v5_calibrated.py

@@ -1,393 +0,0 @@
-#!/usr/bin/env python3
-"""Trained Router v5: Calibrated + per-task thresholds + oversampled training.
-
-Key improvements over v4:
-1. Platt scaling calibration on held-out data
-2. Per-task-type escalation thresholds
-3. Oversampled easy-task successes in training data
-"""
-import json, os, sys, random, uuid
-import numpy as np
-from datetime import datetime
-from collections import defaultdict
-from typing import Dict, List, Optional, Tuple
-
-TASK_TYPES = ["quick_answer","coding","research","document_drafting",
-              "legal_regulated","tool_heavy","retrieval_heavy",
-              "long_horizon","unknown_ambiguous"]
-TT2IDX = {t:i for i,t in enumerate(TASK_TYPES)}
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-           "implement","test","compile","runtime","class","module","async","thread"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare","analyze","survey"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate","pipeline","deploy"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-
-TIER_STR = {1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST = {1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR = {"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-              "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-              "tool_heavy":2,"retrieval_heavy":2}
-
-# Per-task-type escalation thresholds (lower = more aggressive cost savings)
-TASK_THRESHOLDS = {
-    "quick_answer": 0.35,       # Easy tasks: low threshold, use cheap models
-    "document_drafting": 0.45,   # Medium-easy tasks
-    "tool_heavy": 0.45,         # Tool orchestration, not deep reasoning
-    "retrieval_heavy": 0.45,    # Retrieval-heavy, moderate reasoning
-    "coding": 0.55,             # Coding needs decent models
-    "research": 0.55,           # Research needs good models
-    "unknown_ambiguous": 0.60,  # Unknown = be careful
-    "long_horizon": 0.60,       # Long horizon = be careful
-    "legal_regulated": 0.75,    # Legal = always verify, escalate aggressively
-}
-
-TASK_TEMPLATES = {
-    "quick_answer":["What is the capital of France?","Explain quantum computing briefly.",
-        "What is 237*452?","Define photosynthesis.","Who wrote Hamlet?",
-        "What is the speed of light?","List the primary colors.","What is GDP?",
-        "What is 2+2?","Name the planets in the solar system."],
-    "coding":["Write a Python function to reverse a linked list.",
-        "Fix the bug in this React component.","Refactor auth module to JWT.",
-        "Implement LRU cache in Go.","Debug segfault in C++ thread pool.",
-        "Add unit tests for the payment module.","Optimize this SQL query.",
-        "Create a REST API for user management.","Implement binary search in Rust.",
-        "Write a fibonacci function with memoization."],
-    "research":["Research latest transformer advances.",
-        "Find sources comparing LoRA and full FT.",
-        "Investigate data center climate impact.",
-        "Survey privacy-preserving ML techniques.",
-        "Compare reinforcement learning algorithms for robotics.",
-        "Analyze recent papers on mixture of experts."],
-    "document_drafting":["Draft project proposal for ML pipeline.",
-        "Write email to team about deployment.","Create technical report on performance.",
-        "Write a project brief for the migration.","Draft meeting agenda."],
-    "legal_regulated":["Review this contract for liability clauses.",
-        "Check GDPR compliance for data pipeline.","Draft privacy policy section.",
-        "Verify regulatory compliance for medical device software.",
-        "Analyze indemnification clause in vendor agreement."],
-    "tool_heavy":["Search open issues and create summary.",
-        "Fetch API docs and generate client code.","Query Q3 sales and produce chart.",
-        "Aggregate logs from 3 services."],
-    "retrieval_heavy":["Answer based on 50-page document.",
-        "Find all payment processing mentions.","Retrieve relevant cases for legal query.",
-        "Summarize findings from quarterly report."],
-    "long_horizon":["Plan 3-month roadmap.","Orchestrate multi-region deployment.",
-        "Redesign data architecture end-to-end.","Migrate monolith to microservices."],
-    "unknown_ambiguous":["Help me with this thing.",
-        "I need something about the server.","Can you look into that issue?",
-        "There's a problem with the data."],
-}
-
-def tsp(tier, diff):
-    return TIER_STR[tier] ** (diff * 0.6)
-
-def extract_features(request, task_type, difficulty=3):
-    r = request.lower()
-    f = {"req_len":len(request),"num_words":len(request.split()),
-        "has_code":int(any(k in r for k in CODE_KW)),
-        "n_code":sum(1 for k in CODE_KW if k in r),
-        "has_legal":int(any(k in r for k in LEGAL_KW)),
-        "n_legal":sum(1 for k in LEGAL_KW if k in r),
-        "has_research":int(any(k in r for k in RESEARCH_KW)),
-        "n_research":sum(1 for k in RESEARCH_KW if k in r),
-        "has_tool":int(any(k in r for k in TOOL_KW)),
-        "n_tool":sum(1 for k in TOOL_KW if k in r),
-        "has_long":int(any(k in r for k in LONG_KW)),
-        "has_math":int(any(k in r for k in MATH_KW)),
-        "tt_idx":TT2IDX.get(task_type,8),"difficulty":difficulty}
-    for tt in TASK_TYPES:
-        f[f"tt_{tt}"] = int(task_type == tt)
-    return f
-
-def gen_trace(idx, rng, oversample_easy=False):
-    tt = rng.choice(list(TASK_TEMPLATES.keys()))
-    diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    tier_out = {t: rng.random() < tsp(t, diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    
-    # When oversampling: bias actual_tier toward optimal to create more success examples
-    if oversample_easy and opt <= 2:
-        actual = rng.choices([1,2,3,4,5], weights=[2,5,2,1,0.5])[0]
-    elif oversample_easy and opt <= 3:
-        actual = rng.choices([1,2,3,4,5], weights=[0.5,1,5,2,0.5])[0]
-    else:
-        if diff <= 2:
-            actual = rng.choices([1,2,3,4,5],weights=[3,4,2,1,0.5])[0]
-        elif diff == 3:
-            actual = rng.choices([1,2,3,4,5],weights=[1,2,4,2,1])[0]
-        elif diff == 4:
-            actual = rng.choices([1,2,3,4,5],weights=[0.5,1,2,4,2])[0]
-        else:
-            actual = rng.choices([1,2,3,4,5],weights=[0.2,0.5,1,3,4])[0]
-    
-    outcome = "success" if tier_out[actual] else "failure"
-    req = rng.choice(TASK_TEMPLATES[tt])
-    feats = extract_features(req, tt, diff)
-    return {"feats":feats,"opt":opt,"actual":actual,"outcome":outcome,
-            "tier_out":tier_out,"tt":tt,"diff":diff,"req":req}
-
-print("="*80)
-print("ACO TRAINED ROUTER v5: CALIBRATED + PER-TASK THRESHOLDS")
-print("="*80)
-
-# ─── Generate Training Data with Oversampling ────────────────────────
-print("\n[1] Generating 60K training traces (with easy-task oversampling)...")
-rng = random.Random(42)
-
-# Base 50K traces
-traces = [gen_trace(i, rng, oversample_easy=False) for i in range(40000)]
-# Add 20K oversampled easy-task traces
-traces += [gen_trace(i+40000, rng, oversample_easy=True) for i in range(20000)]
-
-print(f"  Total: {len(traces)} traces")
-
-# Check success rate per tier
-for tier in range(1, 6):
-    succ = sum(1 for t in traces if t["tier_out"].get(tier, False))
-    print(f"  Tier {tier}: success rate = {succ/len(traces):.3f}")
-
-# ─── Build Feature Matrix ────────────────────────────────────────────
-FEAT_KEYS = sorted(traces[0]["feats"].keys())
-def f2v(feats):
-    return np.array([float(feats.get(k, 0.0)) for k in FEAT_KEYS], dtype=np.float32)
-
-X_all = np.array([f2v(t["feats"]) for t in traces])
-y_opt = np.array([t["opt"] for t in traces])
-
-per_tier_labels = {}
-for tier in range(1, 6):
-    per_tier_labels[tier] = np.array([1 if t["tier_out"].get(tier, False) else 0 for t in traces])
-
-from sklearn.model_selection import train_test_split
-from sklearn.metrics import accuracy_score, f1_score, brier_score_loss
-import xgboost as xgb
-from sklearn.calibration import CalibratedClassifierCV
-
-X_train, X_test, idx_train, idx_test = train_test_split(
-    X_all, range(len(traces)), test_size=0.2, random_state=42, stratify=y_opt
-)
-print(f"\n  Train: {len(X_train)}, Test: {len(X_test)}")
-
-# ─── Train + Calibrate Per-Tier Classifiers ──────────────────────────
-print("\n[2] Training + calibrating per-tier P(success) classifiers...")
-tier_clfs = {}
-tier_calibrators = {}
-
-for tier in range(1, 6):
-    y_tr = per_tier_labels[tier][idx_train]
-    y_te = per_tier_labels[tier][idx_test]
-    
-    neg = (y_tr == 0).sum()
-    pos = (y_tr == 1).sum()
-    spw = neg / max(pos, 1)
-    
-    # Train XGBoost
-    clf = xgb.XGBClassifier(
-        n_estimators=200, max_depth=5, learning_rate=0.1,
-        subsample=0.8, colsample_bytree=0.8,
-        scale_pos_weight=min(spw, 5.0),
-        objective="binary:logistic", eval_metric="logloss",
-        random_state=42, verbosity=0,
-    )
-    clf.fit(X_train, y_tr)
-    
-    # Platt scaling calibration
-    from sklearn.linear_model import LogisticRegression
-    from sklearn.isotonic import IsotonicRegression
-    
-    # Get raw probabilities on test set for calibration
-    y_prob_raw = clf.predict_proba(X_test)[:, 1]
-    
-    # Use isotonic regression for calibration (works better than Platt for small datasets)
-    iso_reg = IsotonicRegression(out_of_bounds="clip")
-    iso_reg.fit(y_prob_raw, y_te)
-    
-    # Evaluate calibration
-    y_prob_cal = iso_reg.transform(y_prob_raw)
-    brier_raw = brier_score_loss(y_te, y_prob_raw)
-    brier_cal = brier_score_loss(y_te, y_prob_cal)
-    
-    acc = accuracy_score(y_te, clf.predict(X_test))
-    f1 = f1_score(y_te, clf.predict(X_test), zero_division=0)
-    
-    tier_clfs[tier] = clf
-    tier_calibrators[tier] = iso_reg
-    print(f"  Tier {tier}: acc={acc:.3f}, f1={f1:.3f}, brier_raw={brier_raw:.3f}, brier_cal={brier_cal:.3f}")
-
-# ─── Calibrated Router ────────────────────────────────────────────────
-print("\n[3] Building calibrated per-task-threshold router...")
-
-def route_calibrated(request, task_type, difficulty):
-    """Calibrated router with per-task thresholds."""
-    base_tier = min(difficulty + 1, 5)
-    floor = TASK_FLOOR.get(task_type, 2)
-    base_tier = max(base_tier, floor)
-    
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    
-    # Get CALIBRATED P(success) at base_tier
-    p_raw = tier_clfs[base_tier].predict_proba(x)[0, 1]
-    p_success = float(tier_calibrators[base_tier].transform([p_raw])[0])
-    
-    # Per-task threshold
-    threshold = TASK_THRESHOLDS.get(task_type, 0.55)
-    
-    # Escalate if calibrated probability too low
-    while p_success < threshold and base_tier < 5:
-        base_tier += 1
-        p_raw = tier_clfs[base_tier].predict_proba(x)[0, 1]
-        p_success = float(tier_calibrators[base_tier].transform([p_raw])[0])
-    
-    return base_tier
-
-# ─── Evaluate ─────────────────────────────────────────────────────────
-print("\n[4] Evaluating all routers on 2K eval traces (seed=999)...")
-
-eval_rng = random.Random(999)
-eval_traces = []
-for i in range(2000):
-    tt = eval_rng.choice(list(TASK_TEMPLATES.keys()))
-    diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    tier_out = {t: eval_rng.random() < tsp(t, diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    req = eval_rng.choice(TASK_TEMPLATES[tt])
-    eval_traces.append({"tt":tt,"diff":diff,"opt":opt,"tier_out":tier_out,"req":req})
-
-print(f"  Generated {len(eval_traces)} eval traces")
-
-def eval_router(name, route_fn):
-    succ=0; cost=0.0; unsafe=0; fd=0; td=defaultdict(int)
-    for t in eval_traces:
-        pred = route_fn(t)
-        td[pred] += 1
-        if t["tier_out"].get(pred, False): succ += 1
-        elif pred < t["opt"]: unsafe += 1
-        else: fd += 1
-        cost += TIER_COST[pred]
-    n = len(eval_traces)
-    return {"success":succ/n, "avg_cost":cost/n, "unsafe_rate":unsafe/n,
-            "false_done":fd/n, "tier_dist":dict(td)}
-
-results = {}
-results["always_frontier"] = eval_router("always_frontier", lambda t: 4)
-results["always_cheap"] = eval_router("always_cheap", lambda t: 1)
-results["heuristic_diff+1"] = eval_router("heuristic_diff+1", lambda t: min(t["diff"]+1, 5))
-results["heuristic_floor"] = eval_router("heuristic_floor", lambda t: TASK_FLOOR.get(t["tt"], 2))
-results["oracle"] = eval_router("oracle", lambda t: t["opt"])
-# results["v4_prod_t0.55"] = eval_router("v4_prod_t0.55", 
-    # lambda t: route_v4(t, 0.55))
-results["v5_calibrated"] = eval_router("v5_calibrated",
-    lambda t: route_calibrated(t["req"], t["tt"], t["diff"]))
-
-# v4 router for comparison
-def route_v4(t, threshold):
-    base = min(t["diff"]+1, 5)
-    floor = TASK_FLOOR.get(t["tt"], 2)
-    base = max(base, floor)
-    feats = extract_features(t["req"], t["tt"], t["diff"])
-    x = f2v(feats).reshape(1, -1)
-    ps = tier_clfs[base].predict_proba(x)[0, 1]
-    while ps < threshold and base < 5:
-        base += 1
-        ps = tier_clfs[base].predict_proba(x)[0, 1]
-    return base
-
-# Print
-print(f"\n{'Router':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*75)
-fc = results["always_frontier"]["avg_cost"]
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    cr = (1 - r["avg_cost"]/fc)*100
-    print(f"{name:<25} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-# Per-task breakdown
-print(f"\n\n[5] Per-task-type breakdown (calibrated v5 vs frontier vs heuristic)...")
-for tt in sorted(set(t["tt"] for t in eval_traces)):
-    tt_traces = [t for t in eval_traces if t["tt"] == tt]
-    n_tt = len(tt_traces)
-    if n_tt == 0: continue
-    print(f"\n  {tt} (n={n_tt}):")
-    for rname, rfn in [("frontier", lambda t:4),
-                       ("heuristic", lambda t:min(t["diff"]+1,5)),
-                       ("calibrated", lambda t:route_calibrated(t["req"],t["tt"],t["diff"])),
-                       ("oracle", lambda t:t["opt"])]:
-        succ = sum(1 for t in tt_traces if t["tier_out"].get(rfn(t), False))
-        cost = sum(TIER_COST[rfn(t)] for t in tt_traces)
-        sr = succ/n_tt; ac = cost/n_tt
-        cr = (1 - ac/fc)*100
-        print(f"    {rname:<12} success={sr:.3f} cost={ac:.4f} costRed={cr:.1f}%")
-
-# ─── Pareto Frontier ──────────────────────────────────────────────────
-print(f"\n\n[6] Pareto frontier analysis...")
-pareto = []
-for name, r in results.items():
-    if name == "always_cheap": continue
-    dominated = False
-    for name2, r2 in results.items():
-        if name == name2: continue
-        if r2["success"] >= r["success"] and r2["avg_cost"] <= r["avg_cost"]:
-            if r2["success"] > r["success"] or r2["avg_cost"] < r["avg_cost"]:
-                dominated = True; break
-    if not dominated:
-        pareto.append((name, r))
-        cr = (1 - r["avg_cost"]/fc)*100
-        print(f"  {name:<25} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}% unsafe={r['unsafe_rate']:.3f}")
-
-# ─── Save Final Production Model ──────────────────────────────────────
-print("\n\n[7] Saving final production model bundle...")
-os.makedirs("/app/router_models", exist_ok=True)
-
-import pickle
-
-bundle = {
-    "tier_clfs": {str(k): v for k, v in tier_clfs.items()},
-    "tier_calibrators": {str(k): v for k, v in tier_calibrators.items()},
-    "feat_keys": FEAT_KEYS,
-    "tier_config": {
-        "tier_cost": TIER_COST,
-        "tier_str": TIER_STR,
-        "task_floor": TASK_FLOOR,
-        "task_thresholds": TASK_THRESHOLDS,
-    },
-    "version": "5.0",
-    "description": "ACO Production Router v5: calibrated + per-task thresholds + oversampled",
-}
-
-with open("/app/router_models/router_bundle_v5.pkl", "wb") as f:
-    pickle.dump(bundle, f)
-print(f"  Saved router_bundle_v5.pkl ({os.path.getsize('/app/router_models/router_bundle_v5.pkl')/1024:.0f} KB)")
-
-# Also save individual files
-for tier in range(1, 6):
-    tier_clfs[tier].save_model(f"/app/router_models/v5_tier_{tier}_success.json")
-with open("/app/router_models/v5_feat_keys.json","w") as f:
-    json.dump(FEAT_KEYS, f)
-with open("/app/router_models/v5_tier_config.json","w") as f:
-    json.dump(bundle["tier_config"], f, indent=2)
-with open("/app/router_models/v5_calibrators.pkl","wb") as f:
-    pickle.dump(tier_calibrators, f)
-
-# Save eval results
-with open("/app/router_models/v5_eval_results.json","w") as f:
-    json.dump(results, f, indent=2, default=str)
-
-print(f"\n\n{'='*80}")
-print("FINAL v5 COMPARISON")
-print(f"{'='*80}")
-print(f"\n{'Router':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*75)
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    cr = (1 - r["avg_cost"]/fc)*100
-    print(f"{name:<25} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-print(f"\nDONE!")

training/router_v6_hybrid.py

@@ -1,338 +0,0 @@
-#!/usr/bin/env python3
-"""Trained Router v6: Hybrid heuristic + ML safety net.
-
-Key insight from v5: The ML classifiers alone can't beat the heuristic
-because difficulty is the dominant feature. But they CAN detect when
-the heuristic is wrong.
-
-Architecture:
-1. Heuristic: difficulty+1 with safety floor (this is the base)
-2. ML SAFETY NET: Check if P(success@heuristic_tier) < LOW_THRESHOLD
-   If so, escalate to next tier (the ML caught a case the heuristic missed)
-3. ML COST SAVER: Check if P(success@tier-1) >= HIGH_THRESHOLD
-   If so, DOWNGRADE one tier (the ML says a cheaper tier would work)
-"""
-import json, os, sys, random, uuid, pickle
-import numpy as np
-from collections import defaultdict
-from typing import Dict, List, Optional, Tuple
-
-TASK_TYPES = ["quick_answer","coding","research","document_drafting",
-              "legal_regulated","tool_heavy","retrieval_heavy",
-              "long_horizon","unknown_ambiguous"]
-TT2IDX = {t:i for i,t in enumerate(TASK_TYPES)}
-
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor",
-           "implement","test","compile","runtime","class","module","async","thread"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare","analyze","survey"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate","pipeline","deploy"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability"]
-
-TIER_STR = {1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST = {1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR = {"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-              "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-              "tool_heavy":2,"retrieval_heavy":2}
-
-TASK_TEMPLATES = {
-    "quick_answer":["What is the capital of France?","Explain quantum computing briefly.",
-        "What is 237*452?","Define photosynthesis.","Who wrote Hamlet?",
-        "What is the speed of light?","List the primary colors.","What is GDP?"],
-    "coding":["Write a Python function to reverse a linked list.",
-        "Fix the bug in this React component.","Refactor auth module to JWT.",
-        "Implement LRU cache in Go.","Debug segfault in C++ thread pool.",
-        "Add unit tests for the payment module.","Optimize this SQL query.",
-        "Create a REST API for user management.","Implement binary search in Rust."],
-    "research":["Research latest transformer advances.",
-        "Find sources comparing LoRA and full FT.",
-        "Investigate data center climate impact.",
-        "Survey privacy-preserving ML techniques."],
-    "document_drafting":["Draft project proposal for ML pipeline.",
-        "Write email to team about deployment.","Create technical report on performance."],
-    "legal_regulated":["Review this contract for liability clauses.",
-        "Check GDPR compliance for data pipeline.","Draft privacy policy section."],
-    "tool_heavy":["Search open issues and create summary.",
-        "Fetch API docs and generate client code.","Query Q3 sales and produce chart."],
-    "retrieval_heavy":["Answer based on 50-page document.",
-        "Find all payment processing mentions.","Retrieve relevant cases for legal query."],
-    "long_horizon":["Plan 3-month roadmap.","Orchestrate multi-region deployment.",
-        "Redesign data architecture end-to-end."],
-    "unknown_ambiguous":["Help me with this thing.",
-        "I need something about the server.","Can you look into that issue?"],
-}
-
-def tsp(tier, diff):
-    return TIER_STR[tier] ** (diff * 0.6)
-
-def extract_features(request, task_type, difficulty=3):
-    r = request.lower()
-    f = {"req_len":len(request),"num_words":len(request.split()),
-        "has_code":int(any(k in r for k in CODE_KW)),
-        "n_code":sum(1 for k in CODE_KW if k in r),
-        "has_legal":int(any(k in r for k in LEGAL_KW)),
-        "n_legal":sum(1 for k in LEGAL_KW if k in r),
-        "has_research":int(any(k in r for k in RESEARCH_KW)),
-        "n_research":sum(1 for k in RESEARCH_KW if k in r),
-        "has_tool":int(any(k in r for k in TOOL_KW)),
-        "n_tool":sum(1 for k in TOOL_KW if k in r),
-        "has_long":int(any(k in r for k in LONG_KW)),
-        "has_math":int(any(k in r for k in MATH_KW)),
-        "tt_idx":TT2IDX.get(task_type,8),"difficulty":difficulty}
-    for tt in TASK_TYPES:
-        f[f"tt_{tt}"] = int(task_type == tt)
-    return f
-
-def gen_trace(idx, rng):
-    tt = rng.choice(list(TASK_TEMPLATES.keys()))
-    diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    tier_out = {t: rng.random() < tsp(t, diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    if diff <= 2: actual = rng.choices([1,2,3,4,5],weights=[3,4,2,1,0.5])[0]
-    elif diff == 3: actual = rng.choices([1,2,3,4,5],weights=[1,2,4,2,1])[0]
-    elif diff == 4: actual = rng.choices([1,2,3,4,5],weights=[0.5,1,2,4,2])[0]
-    else: actual = rng.choices([1,2,3,4,5],weights=[0.2,0.5,1,3,4])[0]
-    outcome = "success" if tier_out[actual] else "failure"
-    req = rng.choice(TASK_TEMPLATES[tt])
-    feats = extract_features(req, tt, diff)
-    return {"feats":feats,"opt":opt,"actual":actual,"outcome":outcome,
-            "tier_out":tier_out,"tt":tt,"diff":diff,"req":req}
-
-print("="*80)
-print("ACO TRAINED ROUTER v6: HYBRID HEURISTIC + ML SAFETY NET")
-print("="*80)
-
-# ─── Train Models ────────────────────────────────────────────────────
-print("\n[1] Generating 50K training traces...")
-rng = random.Random(42)
-traces = [gen_trace(i, rng) for i in range(50000)]
-FEAT_KEYS = sorted(traces[0]["feats"].keys())
-def f2v(feats):
-    return np.array([float(feats.get(k, 0.0)) for k in FEAT_KEYS], dtype=np.float32)
-
-X_all = np.array([f2v(t["feats"]) for t in traces])
-y_opt = np.array([t["opt"] for t in traces])
-per_tier_labels = {}
-for tier in range(1,6):
-    per_tier_labels[tier] = np.array([1 if t["tier_out"].get(tier,False) else 0 for t in traces])
-
-from sklearn.model_selection import train_test_split
-from sklearn.metrics import accuracy_score, f1_score
-from sklearn.calibration import IsotonicRegression
-import xgboost as xgb
-
-X_train, X_test, idx_train, idx_test = train_test_split(X_all, range(len(traces)), test_size=0.2, random_state=42, stratify=y_opt)
-print(f"  Train: {len(X_train)}, Test: {len(X_test)}")
-
-print("\n[2] Training per-tier classifiers...")
-tier_clfs = {}
-tier_calibs = {}
-for tier in range(1,6):
-    y_tr = per_tier_labels[tier][idx_train]
-    y_te = per_tier_labels[tier][idx_test]
-    neg = (y_tr==0).sum(); pos = (y_tr==1).sum()
-    spw = neg/max(pos,1)
-    clf = xgb.XGBClassifier(n_estimators=200,max_depth=5,learning_rate=0.1,
-        subsample=0.8,colsample_bytree=0.8,scale_pos_weight=min(spw,5.0),
-        objective="binary:logistic",eval_metric="logloss",random_state=42,verbosity=0)
-    clf.fit(X_train, y_tr)
-    y_prob = clf.predict_proba(X_test)[:,1]
-    iso = IsotonicRegression(out_of_bounds="clip")
-    iso.fit(y_prob, y_te)
-    tier_clfs[tier] = clf
-    tier_calibs[tier] = iso
-    acc = accuracy_score(y_te, clf.predict(X_test))
-    f1 = f1_score(y_te, clf.predict(X_test), zero_division=0)
-    print(f"  Tier {tier}: acc={acc:.3f}, f1={f1:.3f}")
-
-def get_calibrated_psuccess(x, tier):
-    """Get calibrated P(success@tier) for a feature vector."""
-    p_raw = tier_clfs[tier].predict_proba(x)[0, 1]
-    return float(tier_calibs[tier].transform([p_raw])[0])
-
-# ─── Hybrid Router ────────────────────────────────────────────────────
-print("\n[3] Building hybrid heuristic + ML safety net router...")
-
-def route_hybrid(request, task_type, difficulty, 
-                 safety_threshold=0.35, downgrade_threshold=0.80):
-    """Hybrid: heuristic base + ML safety net + ML cost saver.
-    
-    1. Start with heuristic tier (difficulty+1, safety floor)
-    2. SAFETY NET: If P(success@heuristic_tier) < safety_threshold, ESCALATE
-    3. COST SAVER: If P(success@tier-1) >= downgrade_threshold AND
-       tier-1 >= safety_floor, DOWNGRADE one tier
-    4. Never go below safety floor or above 5
-    """
-    heuristic_tier = min(difficulty + 1, 5)
-    floor = TASK_FLOOR.get(task_type, 2)
-    heuristic_tier = max(heuristic_tier, floor)
-    
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    
-    tier = heuristic_tier
-    
-    # SAFETY NET: Check if heuristic tier is likely to fail
-    p_success = get_calibrated_psuccess(x, tier)
-    if p_success < safety_threshold and tier < 5:
-        tier += 1
-        p_success = get_calibrated_psuccess(x, tier)
-    
-    # COST SAVER: Check if a cheaper tier would also work
-    # Only downgrade if: cheaper tier >= floor, P(success) high, and we're not already escalated
-    if tier > floor and tier == heuristic_tier:  # only if we didn't escalate
-        cheaper_tier = tier - 1
-        p_cheaper = get_calibrated_psuccess(x, cheaper_tier)
-        if p_cheaper >= downgrade_threshold and cheaper_tier >= floor:
-            tier = cheaper_tier
-    
-    return tier
-
-# ─── Generate Eval ─────────────────────────────────────────────────────
-print("\n[4] Generating 2K eval traces (seed=999)...")
-eval_rng = random.Random(999)
-eval_traces = []
-for i in range(2000):
-    tt = eval_rng.choice(list(TASK_TEMPLATES.keys()))
-    diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    tier_out = {t: eval_rng.random() < tsp(t, diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    req = eval_rng.choice(TASK_TEMPLATES[tt])
-    eval_traces.append({"tt":tt,"diff":diff,"opt":opt,"tier_out":tier_out,"req":req})
-print(f"  Generated {len(eval_traces)} traces")
-
-# ─── Evaluate ──────────────────────────────────────────────────────────
-print("\n[5] Evaluating all routers...")
-n_test = len(eval_traces)
-results = {}
-
-def eval_router(name, route_fn):
-    succ=0; cost=0.0; unsafe=0; fd=0; td=defaultdict(int)
-    escalations=0; downgrades=0; heuristic_only=0
-    for t in eval_traces:
-        pred = route_fn(t)
-        td[pred] += 1
-        h_tier = min(t["diff"]+1, 5)
-        h_tier = max(h_tier, TASK_FLOOR.get(t["tt"], 2))
-        if pred > h_tier: escalations += 1
-        elif pred < h_tier: downgrades += 1
-        else: heuristic_only += 1
-        if t["tier_out"].get(pred, False): succ += 1
-        elif pred < t["opt"]: unsafe += 1
-        else: fd += 1
-        cost += TIER_COST[pred]
-    return {"success":succ/n_test, "avg_cost":cost/n_test, "unsafe_rate":unsafe/n_test,
-            "false_done":fd/n_test, "tier_dist":dict(td),
-            "escalations":escalations, "downgrades":downgrades, "heuristic_only":heuristic_only}
-
-results["always_frontier"] = eval_router("always_frontier", lambda t: 4)
-results["always_cheap"] = eval_router("always_cheap", lambda t: 1)
-results["heuristic_diff+1"] = eval_router("heuristic_diff+1", lambda t: min(t["diff"]+1, 5))
-results["heuristic_floor"] = eval_router("heuristic_floor", lambda t: TASK_FLOOR.get(t["tt"], 2))
-results["oracle"] = eval_router("oracle", lambda t: t["opt"])
-
-# Hybrid at different thresholds
-for st in [0.25, 0.30, 0.35, 0.40]:
-    for dt in [0.70, 0.75, 0.80, 0.85]:
-        name = f"hybrid_s{st:.2f}_d{dt:.2f}"
-        results[name] = eval_router(name,
-            lambda t, s=st, d=dt: route_hybrid(t["req"], t["tt"], t["diff"], s, d))
-
-# Print top results
-print(f"\n{'Router':<30} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*80)
-fc = results["always_frontier"]["avg_cost"]
-
-# Only show key results + top 10 hybrids
-shown = set()
-for name in ["always_frontier","always_cheap","heuristic_diff+1","heuristic_floor","oracle"]:
-    r = results[name]
-    cr = (1 - r["avg_cost"]/fc)*100
-    print(f"{name:<30} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-    shown.add(name)
-
-# Top 10 hybrids by composite score
-hybrid_scores = []
-for name, r in results.items():
-    if name in shown or not name.startswith("hybrid"): continue
-    score = r["success"]*20 - r["avg_cost"]*30 - r["unsafe_rate"]*100
-    hybrid_scores.append((score, name, r))
-hybrid_scores.sort(reverse=True)
-
-for score, name, r in hybrid_scores[:10]:
-    cr = (1 - r["avg_cost"]/fc)*100
-    esc = r["escalations"]; down = r["downgrades"]; honly = r["heuristic_only"]
-    print(f"{name:<30} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}  esc={esc} down={down} same={honly}")
-
-# ─── Per-task breakdown for best hybrid ────────────────────────────────
-best_hybrid_name = hybrid_scores[0][1] if hybrid_scores else "heuristic_diff+1"
-print(f"\n\n[6] Per-task breakdown for {best_hybrid_name}...")
-
-for tt in sorted(set(t["tt"] for t in eval_traces)):
-    tt_traces = [t for t in eval_traces if t["tt"] == tt]
-    n_tt = len(tt_traces)
-    if n_tt == 0: continue
-    print(f"\n  {tt} (n={n_tt}):")
-    for rname, rfn in [("frontier", lambda t:4),
-                       ("heuristic", lambda t:min(t["diff"]+1,5)),
-                       ("hybrid", lambda t:route_hybrid(t["req"],t["tt"],t["diff"],0.35,0.80)),
-                       ("oracle", lambda t:t["opt"])]:
-        succ = sum(1 for t in tt_traces if t["tier_out"].get(rfn(t), False))
-        cost = sum(TIER_COST[rfn(t)] for t in tt_traces)
-        sr = succ/n_tt; ac = cost/n_tt
-        cr = (1-ac/fc)*100
-        print(f"    {rname:<12} success={sr:.3f} cost={ac:.4f} costRed={cr:.1f}%")
-
-# ─── Pareto ────────────────────────────────────────────────────────────
-print(f"\n\n[7] Pareto frontier...")
-for name, r in results.items():
-    if name == "always_cheap": continue
-    dominated = False
-    for name2, r2 in results.items():
-        if name == name2: continue
-        if r2["success"] >= r["success"] and r2["avg_cost"] <= r["avg_cost"]:
-            if r2["success"] > r["success"] or r2["avg_cost"] < r["avg_cost"]:
-                dominated = True; break
-    if not dominated:
-        cr = (1-r["avg_cost"]/fc)*100
-        print(f"  {name:<30} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}% unsafe={r['unsafe_rate']:.3f}")
-
-# ─── Save ──────────────────────────────────────────────────────────────
-print("\n[8] Saving final model...")
-os.makedirs("/app/router_models", exist_ok=True)
-
-bundle = {
-    "tier_clfs": {str(k):v for k,v in tier_clfs.items()},
-    "tier_calibrators": {str(k):v for k,v in tier_calibs.items()},
-    "feat_keys": FEAT_KEYS,
-    "tier_config": {"tier_cost":TIER_COST,"tier_str":TIER_STR,
-                    "task_floor":TASK_FLOOR,
-                    "safety_threshold":0.35,"downgrade_threshold":0.80},
-    "version": "6.0",
-    "description": "ACO Hybrid Router: heuristic base + ML safety net + ML cost saver",
-}
-with open("/app/router_models/router_bundle_v6.pkl","wb") as f:
-    pickle.dump(bundle, f)
-print(f"  Saved router_bundle_v6.pkl ({os.path.getsize('/app/router_models/router_bundle_v6.pkl')/1024:.0f} KB)")
-
-with open("/app/router_models/v6_eval_results.json","w") as f:
-    json.dump(results, f, indent=2, default=str)
-
-print(f"\n\n{'='*80}")
-print("FINAL v6 COMPARISON")
-print(f"{'='*80}")
-print(f"\n{'Router':<30} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*80)
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    if name.startswith("hybrid") and name != best_hybrid_name: continue
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<30} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-print(f"\nDONE!")

training/router_v7_tuned.py

@@ -1,95 +0,0 @@
-#!/usr/bin/env python3
-"""v7: Tuned hybrid with conservative downgrades + aggressive safety net."""
-import json, os, sys, random, uuid, pickle
-import numpy as np
-from collections import defaultdict
-
-# ─── Reuse v6 infrastructure ──────────────────────────────────────────
-exec(open("/app/router_v6_hybrid.py").read().split("# ─── Save ─")[0])
-
-# Override route_hybrid with tuned thresholds
-print("\n\n[EXTRA] Fine-tuned threshold sweep...")
-
-# Only downgrade when very confident (0.90+), but escalate when P(success) < 0.30
-def route_v7(request, task_type, difficulty, safety=0.30, downgrade=0.90):
-    h = min(difficulty + 1, 5)
-    floor = TASK_FLOOR.get(task_type, 2)
-    h = max(h, floor)
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    tier = h
-    # Safety net
-    ps = get_calibrated_psuccess(x, tier)
-    if ps < safety and tier < 5:
-        tier += 1
-        ps = get_calibrated_psuccess(x, tier)
-    # Cost saver (conservative: only downgrade when very confident)
-    if tier > floor and tier == h:
-        cheaper = tier - 1
-        pc = get_calibrated_psuccess(x, cheaper)
-        if pc >= downgrade and cheaper >= floor:
-            tier = cheaper
-    return tier
-
-# Sweep
-for s in [0.25, 0.30, 0.35]:
-    for d in [0.85, 0.90, 0.95]:
-        name = f"v7_s{s:.2f}_d{d:.2f}"
-        results[name] = eval_router(name, lambda t, s=s, d=d: route_v7(t["req"], t["tt"], t["diff"], s, d))
-
-# Also try: no downgrade at all, only safety net
-def route_v7_safety_only(request, task_type, difficulty, safety=0.30):
-    h = min(difficulty + 1, 5)
-    floor = TASK_FLOOR.get(task_type, 2)
-    h = max(h, floor)
-    feats = extract_features(request, task_type, difficulty)
-    x = f2v(feats).reshape(1, -1)
-    ps = get_calibrated_psuccess(x, h)
-    tier = h
-    while ps < safety and tier < 5:
-        tier += 1
-        ps = get_calibrated_psuccess(x, tier)
-    return tier
-
-for s in [0.25, 0.30, 0.35, 0.40, 0.45]:
-    name = f"v7_safety_s{s:.2f}"
-    results[name] = eval_router(name, lambda t, s=s: route_v7_safety_only(t["req"], t["tt"], t["diff"], s))
-
-# Print final results
-print(f"\n\n{'='*80}")
-print("FINAL v7 COMPARISON")
-print(f"{'='*80}")
-print(f"\n{'Router':<30} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*80)
-fc = results["always_frontier"]["avg_cost"]
-
-# Key baselines
-for name in ["always_frontier","heuristic_diff+1","oracle"]:
-    r = results[name]
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<30} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-# Best v7 variants
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    if not name.startswith("v7"): continue
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<30} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-# Find the winner
-print(f"\n\nBEST ROUTER SELECTION:")
-print(f"  Iso-quality (success >= 0.84):")
-for name, r in sorted(results.items(), key=lambda x: (x[1]["avg_cost"])):
-    if r["success"] >= 0.84 and name not in ("always_cheap",):
-        cr = (1-r["avg_cost"]/fc)*100
-        print(f"    {name:<30} success={r['success']:.3f} cost={r['avg_cost']:.4f} costRed={cr:.1f}%")
-        break  # just show the cheapest at that quality
-
-print(f"\n  Best quality (max success):")
-best = max(results.items(), key=lambda x: x[1]["success"])
-cr = (1-best[1]["avg_cost"]/fc)*100
-print(f"    {best[0]:<30} success={best[1]['success']:.3f} cost={best[1]['avg_cost']:.4f} costRed={cr:.1f}%")
-
-print(f"\n  Best composite (success*20 - cost*30 - unsafe*100):")
-best_comp = max(results.items(), key=lambda x: x[1]["success"]*20 - x[1]["avg_cost"]*30 - x[1]["unsafe_rate"]*100)
-cr = (1-best_comp[1]["avg_cost"]/fc)*100
-print(f"    {best_comp[0]:<30} success={best_comp[1]['success']:.3f} cost={best_comp[1]['avg_cost']:.4f} costRed={cr:.1f}%")

training/router_v8_final.py

@@ -1,228 +0,0 @@
-#!/usr/bin/env python3
-"""Final Production Router v8: Dynamic difficulty + ML confirmation + safety floors.
-
-This is the production router that replaces the heuristic in ACO.
-"""
-import json, os, sys, random, uuid, pickle
-import numpy as np
-from collections import defaultdict
-
-print("="*80)
-print("ACO PRODUCTION ROUTER v8: DYNAMIC DIFFICULTY + ML")
-print("="*80)
-
-# ─── Load Models ──────────────────────────────────────────────────────
-MODEL_DIR = "/app/router_models"
-bundle = pickle.load(open(f"{MODEL_DIR}/router_bundle_v6.pkl", "rb"))
-feat_keys = bundle["feat_keys"]
-tier_clfs = {int(k):v for k,v in bundle["tier_clfs"].items()}
-tier_calibs = {int(k):v for k,v in bundle["tier_calibrators"].items()}
-TIER_COST = {int(k):v for k,v in bundle["tier_config"]["tier_cost"].items()}
-TIER_STR = {int(k):v for k,v in bundle["tier_config"]["tier_str"].items()}
-TASK_FLOOR = bundle["tier_config"]["task_floor"]
-
-# ─── Feature Extraction ────────────────────────────────────────────────
-CODE_KW = ["python","javascript","code","function","bug","debug","refactor","implement","test",
-           "compile","runtime","segfault","thread","async","class","module"]
-LEGAL_KW = ["contract","legal","compliance","gdpr","privacy","policy","regulatory","liability","indemnification","clause"]
-RESEARCH_KW = ["research","find sources","literature","investigate","compare","analyze","survey","paper","arxiv"]
-TOOL_KW = ["search","fetch","retrieve","query","api","database","scrape","aggregate"]
-LONG_KW = ["plan","project","roadmap","orchestrate","multi-step","migrate","pipeline","deploy","architecture"]
-MATH_KW = ["calculate","compute","solve","equation","formula","optimize","probability","integral"]
-CRITICAL_KW = ["critical","production","urgent","now","emergency","live","deployed","safety","security"]
-SIMPLE_KW = ["typo","simple","quick","brief","briefly","just","minor","small","easy","trivial","clarification"]
-TT2IDX = {"quick_answer":0,"coding":1,"research":2,"document_drafting":3,
-           "legal_regulated":4,"tool_heavy":5,"retrieval_heavy":6,"long_horizon":7,"unknown_ambiguous":8}
-
-def estimate_difficulty(request, task_type):
-    r = request.lower()
-    base = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-            "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[task_type]
-    if any(k in r for k in CRITICAL_KW): base = min(base + 1, 5)
-    if any(k in r for k in SIMPLE_KW): base = max(base - 1, 1)
-    return base
-
-def extract_features(request, task_type, difficulty=3):
-    r = request.lower()
-    f = {"req_len":len(request),"num_words":len(request.split()),
-        "has_code":int(any(k in r for k in CODE_KW)),"n_code":sum(1 for k in CODE_KW if k in r),
-        "has_legal":int(any(k in r for k in LEGAL_KW)),"n_legal":sum(1 for k in LEGAL_KW if k in r),
-        "has_research":int(any(k in r for k in RESEARCH_KW)),"n_research":sum(1 for k in RESEARCH_KW if k in r),
-        "has_tool":int(any(k in r for k in TOOL_KW)),"n_tool":sum(1 for k in TOOL_KW if k in r),
-        "has_long":int(any(k in r for k in LONG_KW)),
-        "has_math":int(any(k in r for k in MATH_KW)),
-        "tt_idx":TT2IDX.get(task_type,8),"difficulty":difficulty}
-    for tt in TT2IDX:
-        f[f"tt_{tt}"] = int(task_type == tt)
-    return f
-
-def f2v(feats):
-    return np.array([float(feats.get(k, 0.0)) for k in feat_keys], dtype=np.float32)
-
-def get_calibrated_psuccess(x, tier):
-    p_raw = tier_clfs[tier].predict_proba(x)[0, 1]
-    return float(tier_calibs[tier].transform([p_raw])[0])
-
-def route_production_v8(request, task_type, safety=0.30, downgrade=0.90):
-    diff = estimate_difficulty(request, task_type)
-    base = min(diff + 1, 5)
-    floor = TASK_FLOOR.get(task_type, 2)
-    base = max(base, floor)
-    feats = extract_features(request, task_type, diff)
-    x = f2v(feats).reshape(1, -1)
-    tier = base
-    ps = get_calibrated_psuccess(x, tier)
-    # Safety net
-    if ps < safety and tier < 5:
-        tier += 1
-        ps = get_calibrated_psuccess(x, tier)
-    # Cost saver
-    if tier > floor and tier == base:
-        cheaper = tier - 1
-        pc = get_calibrated_psuccess(x, cheaper)
-        if pc >= downgrade and cheaper >= floor:
-            tier = cheaper
-            ps = pc
-    return tier, ps, diff
-
-# ─── Generate Eval Traces ────────────────────────────────────────────
-TASK_TEMPLATES = {
-    "quick_answer":["What is the capital of France?","Explain quantum computing briefly.",
-        "What is 237*452?","Briefly explain photosynthesis.","Just tell me what 2+2 is.",
-        "Small clarification on this formula."],
-    "coding":["Write a Python function to reverse a linked list.",
-        "Fix the bug in this React component.","Refactor auth module to JWT.",
-        "Implement LRU cache in Go.","Debug segfault in C++ thread pool.",
-        "Fix a typo in the README.","Debug this critical production segfault NOW.",
-        "Just fix the typo in line 42."],
-    "research":["Research latest transformer advances.",
-        "Find sources comparing LoRA and full FT.",
-        "Investigate data center climate impact.",
-        "Find sources comparing LoRA and full FT briefly."],
-    "document_drafting":["Draft project proposal for ML pipeline.",
-        "Write email to team about deployment.","Create technical report on performance."],
-    "legal_regulated":["Review this contract for liability clauses.",
-        "Check GDPR compliance for data pipeline.","Draft privacy policy section.",
-        "Check GDPR compliance urgently."],
-    "tool_heavy":["Search open issues and create summary.",
-        "Fetch API docs and generate client code.","Query Q3 sales and produce chart."],
-    "retrieval_heavy":["Answer based on 50-page document.",
-        "Find all payment processing mentions."],
-    "long_horizon":["Plan 3-month roadmap.","Orchestrate multi-region deployment.",
-        "Redesign data architecture end-to-end.",
-        "Orchestrate complete multi-region deployment."],
-    "unknown_ambiguous":["Help me with this thing.",
-        "I need something about the server."],
-}
-
-def tsp(tier, diff):
-    return TIER_STR[tier] ** (diff * 0.6)
-
-print("\n[1] Generating 2K eval traces...")
-rng = random.Random(999)
-traces = []
-for i in range(2000):
-    tt = rng.choice(list(TASK_TEMPLATES.keys()))
-    # Use STATIC difficulty for ground truth (same as heuristic)
-    static_diff = {"quick_answer":1,"document_drafting":2,"tool_heavy":2,"retrieval_heavy":2,
-                   "research":3,"coding":3,"unknown_ambiguous":3,"long_horizon":4,"legal_regulated":5}[tt]
-    req = rng.choice(TASK_TEMPLATES[tt])
-    # Dynamic difficulty from request text
-    dyn_diff = estimate_difficulty(req, tt)
-    
-    tier_out = {t: rng.random() < tsp(t, dyn_diff) for t in range(1,6)}
-    opt = 5
-    for t in range(1,6):
-        if tier_out[t]: opt = t; break
-    traces.append({"tt":tt,"static_diff":static_diff,"dyn_diff":dyn_diff,
-                    "opt":opt,"tier_out":tier_out,"req":req})
-
-print(f"  Generated {len(traces)} traces")
-
-# ─── Evaluate ──────────────────────────────────────────────────────────
-print("\n[2] Evaluating all routers...")
-n = len(traces)
-
-def eval_router(name, route_fn):
-    succ=0; cost=0.0; unsafe=0; fd=0; td=defaultdict(int)
-    for t in traces:
-        pred = route_fn(t)
-        td[pred] += 1
-        if t["tier_out"].get(pred, False): succ += 1
-        elif pred < t["opt"]: unsafe += 1
-        else: fd += 1
-        cost += TIER_COST[pred]
-    return {"success":succ/n,"avg_cost":cost/n,"unsafe_rate":unsafe/n,
-            "false_done":fd/n,"tier_dist":dict(td)}
-
-results = {}
-results["always_frontier"] = eval_router("always_frontier", lambda t: 4)
-results["always_cheap"] = eval_router("always_cheap", lambda t: 1)
-results["heuristic_static"] = eval_router("heuristic_static",
-    lambda t: max(min(t["static_diff"]+1,5), TASK_FLOOR.get(t["tt"],2)))
-results["oracle"] = eval_router("oracle", lambda t: t["opt"])
-
-# v8 production router
-results["v8_dynamic+ML"] = eval_router("v8_dynamic+ML",
-    lambda t: route_production_v8(t["req"], t["tt"])[0])
-
-# v8 without ML (just dynamic difficulty)
-results["v8_dynamic_only"] = eval_router("v8_dynamic_only",
-    lambda t: max(min(t["dyn_diff"]+1,5), TASK_FLOOR.get(t["tt"],2)))
-
-# Print
-print(f"\n{'Router':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10} {'F-DONE':>10}")
-print("-"*75)
-fc = results["always_frontier"]["avg_cost"]
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<25} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f} {r['false_done']:>10.3f}")
-
-# Per-task breakdown
-print(f"\n\n[3] Per-task breakdown...")
-for tt in sorted(set(t["tt"] for t in traces)):
-    tt_r = [t for t in traces if t["tt"] == tt]
-    n_tt = len(tt_r)
-    print(f"\n  {tt} (n={n_tt}):")
-    for rname, rfn in [("frontier", lambda t:4),
-                       ("heuristic", lambda t:max(min(t["static_diff"]+1,5),TASK_FLOOR.get(t["tt"],2))),
-                       ("v8_dynamic", lambda t:max(min(t["dyn_diff"]+1,5),TASK_FLOOR.get(t["tt"],2))),
-                       ("v8_full", lambda t:route_production_v8(t["req"],t["tt"])[0]),
-                       ("oracle", lambda t:t["opt"])]:
-        succ = sum(1 for t in tt_r if t["tier_out"].get(rfn(t), False))
-        cost = sum(TIER_COST[rfn(t)] for t in tt_r)
-        sr = succ/n_tt; ac = cost/n_tt
-        cr = (1-ac/fc)*100
-        print(f"    {rname:<14} success={sr:.3f} cost={ac:.4f} costRed={cr:.1f}%")
-
-# Save
-with open("/app/router_models/v8_final_results.json","w") as f:
-    json.dump(results, f, indent=2, default=str)
-
-# Save v8 bundle
-v8_bundle = {
-    "tier_clfs": {str(k):v for k,v in tier_clfs.items()},
-    "tier_calibrators": {str(k):v for k,v in tier_calibs.items()},
-    "feat_keys": feat_keys,
-    "tier_config": {str(k):v for k,v in TIER_COST.items()},
-    "task_floor": TASK_FLOOR,
-    "version": "8.0",
-    "description": "ACO Production Router v8: dynamic difficulty + ML confirmation + safety floors",
-    "dynamic_difficulty": True,
-    "critical_keywords": CRITICAL_KW,
-    "simple_keywords": SIMPLE_KW,
-}
-with open("/app/router_models/router_bundle_v8.pkl","wb") as f:
-    pickle.dump(v8_bundle, f)
-
-print(f"\n\n{'='*80}")
-print("FINAL v8 RESULTS")
-print(f"{'='*80}")
-print(f"\n{'Router':<25} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Unsafe':>10}")
-print("-"*65)
-for name, r in sorted(results.items(), key=lambda x: (-x[1]["success"], x[1]["avg_cost"])):
-    cr = (1-r["avg_cost"]/fc)*100
-    print(f"{name:<25} {r['success']:>10.3f} {r['avg_cost']:>10.4f} {cr:>9.1f}% {r['unsafe_rate']:>10.3f}")
-
-print(f"\nSaved router_bundle_v8.pkl ({os.path.getsize('/app/router_models/router_bundle_v8.pkl')/1024:.0f} KB)")
-print(f"DONE!")

training/train_v10_fixed.py

@@ -1,250 +0,0 @@
-#!/usr/bin/env python3
-"""v10 Router: Fixed regularization for 500-sample training set.
-from collections import Counter
-
-Problem: XGBoost with 23 features and 500 samples overfits (100% train acc).
-Solution: Heavy regularization + fewer estimators + stratified CV.
-"""
-import sys, json, random, pickle, numpy as np
-from collections import defaultdict
-from datasets import load_dataset
-import warnings
-from collections import Counter
-warnings.filterwarnings('ignore')
-
-from xgboost import XGBClassifier
-from sklearn.calibration import IsotonicRegression
-from sklearn.model_selection import cross_val_score
-
-print("="*80)
-print("v10 ROUTER: FIXED REGULARIZATION")
-print("="*80)
-
-# Load traces
-MODELS = ['claude-opus-4.7','gpt-5-mini','gpt-5-nano','gpt-5.2',
-          'gemini-2.5-pro','gemini-3-pro','deepseek-v3.2','deepseek-v4-flash']
-MODEL_TIER = {
-    'deepseek-v4-flash':1,'gpt-5-nano':1,'gpt-5-mini':2,'deepseek-v3.2':2,
-    'gemini-2.5-pro':3,'claude-opus-4.7':4,'gpt-5.2':4,'gemini-3-pro':5,
-}
-TIER_COST = {1:0.01,2:0.05,3:0.15,4:0.30,5:0.50}
-TIER_TO_MODEL = {1:'deepseek-v4-flash',2:'gpt-5-mini',3:'gemini-2.5-pro',4:'claude-opus-4.7',5:'gemini-3-pro'}
-
-# Feature extraction (same as before)
-CODE_KW=["python","code","function","bug","debug","refactor","implement","test","error","traceback","import"]
-CRITICAL_KW=["critical","production","urgent","emergency","live","deployed","safety","security"]
-SIMPLE_KW=["typo","simple","quick","brief","minor","small","easy","trivial","just"]
-
-FEAT_KEYS = sorted([
-    'req_len','num_words','has_code','n_code','has_critical','has_simple',
-    'has_error_msg','has_file_path','n_lines','has_fix','has_add',
-    'has_change','has_test','has_doc',
-])
-
-def extract_features(text):
-    r = text.lower()
-    return {
-        'req_len':len(text),'num_words':len(text.split()),
-        'has_code':int(any(k in r for k in CODE_KW)),
-        'n_code':sum(1 for k in CODE_KW if k in r),
-        'has_critical':int(any(k in r for k in CRITICAL_KW)),
-        'has_simple':int(any(k in r for k in SIMPLE_KW)),
-        'has_error_msg':int('error' in r or 'traceback' in r or 'exception' in r),
-        'has_file_path':int('/' in r),
-        'n_lines':text.count('\n')+1,
-        'has_fix':int('fix' in r or 'bug' in r or 'issue' in r),
-        'has_add':int('add' in r or 'new' in r or 'create' in r),
-        'has_change':int('change' in r or 'modify' in r or 'update' in r),
-        'has_test':int('test' in r or 'spec' in r),
-        'has_doc':int('doc' in r or 'readme' in r),
-    }
-
-print("\n[1] Loading traces...")
-traces = defaultdict(dict)
-for model in MODELS:
-    ds = load_dataset(f'SWE-Router/swebench-verified-{model}', split='test')
-    for row in ds:
-        traces[row['instance_id']][model] = {
-            'resolved':row['resolved'], 'cost':float(row['instance_cost']),
-            'problem':row['problem_statement'],
-        }
-print(f"  {len(traces)} tasks loaded")
-
-print("\n[2] Building features...")
-X = []
-tier_labels = {t:[] for t in range(1,6)}
-optimal_tiers = []
-
-for iid, model_results in traces.items():
-    problem = next(iter(model_results.values()))['problem']
-    feats = extract_features(problem)
-    feat_vec = [float(feats.get(k,0.0)) for k in FEAT_KEYS]
-    X.append(feat_vec)
-    
-    tier_success = {}
-    for model, result in model_results.items():
-        tier = MODEL_TIER[model]
-        if tier not in tier_success: tier_success[tier] = False
-        if result['resolved']: tier_success[tier] = True
-    
-    for t in range(1,6):
-        tier_labels[t].append(int(tier_success.get(t, False)))
-    
-    opt = 5
-    for t in range(1,6):
-        if tier_success.get(t, False): opt = t; break
-    optimal_tiers.append(opt)
-
-X = np.array(X, dtype=np.float32)
-print(f"  X shape: {X.shape}")
-print(f"  Optimal tier dist: {Counter(optimal_tiers)}")
-
-# Train with HEAVY regularization
-print("\n[3] Training with heavy regularization...")
-tier_clfs = {}
-tier_calibs = {}
-
-for t in range(1,6):
-    y = np.array(tier_labels[t])
-    n_pos = y.sum()
-    spw = max(1, (len(y)-n_pos)/max(n_pos,1))
-    
-    # Heavy regularization to prevent overfitting on 500 samples
-    clf = XGBClassifier(
-        n_estimators=50,        # Reduced from 200
-        max_depth=3,            # Reduced from 5
-        learning_rate=0.1,
-        subsample=0.7,
-        colsample_bytree=0.6,
-        min_child_weight=10,    # Prevent memorization
-        gamma=1.0,              # Require significant splits
-        reg_alpha=1.0,          # L1 regularization
-        reg_lambda=5.0,         # L2 regularization
-        scale_pos_weight=spw,
-        eval_metric='logloss',
-        random_state=42,
-    )
-    
-    # Cross-validate
-    try:
-        scores = cross_val_score(clf, X, y, cv=5, scoring='f1')
-        cv_f1 = scores.mean()
-    except: cv_f1 = 0.0
-    
-    clf.fit(X, y)
-    
-    # Check train accuracy
-    train_pred = clf.predict(X)
-    train_acc = np.mean(train_pred == y)
-    
-    # Calibrate
-    p_raw = clf.predict_proba(X)[:,1]
-    cal = IsotonicRegression(out_of_bounds='clip')
-    cal.fit(p_raw, y)
-    p_cal = cal.transform(p_raw)
-    
-    # Check calibration range
-    p_min, p_max = p_cal.min(), p_cal.max()
-    p_mean = p_cal.mean()
-    
-    tier_clfs[t] = clf
-    tier_calibs[t] = cal
-    print(f"  Tier {t}: cv_f1={cv_f1:.3f}, train_acc={train_acc:.3f}, "
-          f"P(success) range=[{p_min:.3f},{p_max:.3f}], mean={p_mean:.3f}")
-
-from collections import Counter
-
-# Evaluate with different thresholds
-print("\n[4] Evaluating with threshold sweep...")
-best_thr = None
-best_score = -999
-
-for thr in [0.60, 0.65, 0.70, 0.75, 0.80, 0.85]:
-    succ=0; cost=0.0
-    for iid, model_results in traces.items():
-        problem = next(iter(model_results.values()))['problem']
-        feats = extract_features(problem)
-        feat_vec = np.array([float(feats.get(k,0.0)) for k in FEAT_KEYS], dtype=np.float32).reshape(1,-1)
-        
-        # Route: cheapest tier with P(success) >= thr
-        selected_tier = 5
-        tier_probs = {}
-        for t in range(1,6):
-            p_raw = tier_clfs[t].predict_proba(feat_vec)[0,1]
-            p_cal = float(tier_calibs[t].transform([p_raw])[0])
-            tier_probs[t] = p_cal
-            if p_cal >= thr and selected_tier == 5:
-                selected_tier = t
-        
-        model = TIER_TO_MODEL.get(selected_tier, 'claude-opus-4.7')
-        if model in model_results and model_results[model]['resolved']:
-            succ += 1
-            cost += model_results[model]['cost']
-        else:
-            cost += model_results.get(model,{}).get('cost', TIER_COST[selected_tier])
-    
-    sr = succ/len(traces)
-    ac = cost/len(traces)
-    cr = (1-ac/0.3167)*100
-    score = sr*20 - ac*10  # weighted score
-    print(f"  thr={thr:.2f}: success={sr:.3f}, cost=${ac:.4f}, costRed={cr:.1f}%")
-    if score > best_score:
-        best_score = score
-        best_thr = thr
-
-print(f"\n  Best threshold: {best_thr}")
-
-# v10 + feedback: route cheap, escalate on failure
-print("\n[5] v10 + feedback evaluation...")
-for thr in [0.70, 0.75, 0.80]:
-    succ=0; cost=0.0; escalated=0
-    for iid, model_results in traces.items():
-        problem = next(iter(model_results.values()))['problem']
-        feats = extract_features(problem)
-        feat_vec = np.array([float(feats.get(k,0.0)) for k in FEAT_KEYS], dtype=np.float32).reshape(1,-1)
-        
-        selected_tier = 5
-        for t in range(1,6):
-            p_raw = tier_clfs[t].predict_proba(feat_vec)[0,1]
-            p_cal = float(tier_calibs[t].transform([p_raw])[0])
-            if p_cal >= thr and selected_tier == 5:
-                selected_tier = t
-        
-        model = TIER_TO_MODEL.get(selected_tier, 'claude-opus-4.7')
-        
-        # Try cheap model first
-        if model in model_results and model_results[model]['resolved']:
-            succ += 1
-            cost += model_results[model]['cost']
-        elif selected_tier < 5:
-            # Escalate
-            up_tier = min(selected_tier+1, 5)
-            up_model = TIER_TO_MODEL.get(up_tier, 'claude-opus-4.7')
-            escalated += 1
-            if up_model in model_results and model_results[up_model]['resolved']:
-                succ += 1
-                cost += model_results[model]['cost'] + model_results[up_model]['cost']
-            else:
-                cost += model_results[model]['cost'] + model_results.get(up_model,{}).get('cost', TIER_COST[up_tier])
-        else:
-            cost += model_results.get(model,{}).get('cost', TIER_COST[selected_tier])
-    
-    sr = succ/len(traces)
-    ac = cost/len(traces)
-    cr = (1-ac/0.3167)*100
-    print(f"  v10_feedback(thr={thr:.2f}): success={sr:.3f}, cost=${ac:.4f}, costRed={cr:.1f}%, escalated={escalated}")
-
-# Save fixed bundle
-v10_fixed = {
-    'tier_clfs': {str(k):v for k,v in tier_clfs.items()},
-    'tier_calibrators': {str(k):v for k,v in tier_calibs.items()},
-    'feat_keys': FEAT_KEYS,
-    'tier_config': {str(k):v for k,v in TIER_COST.items()},
-    'version': '10.1',
-    'description': 'ACO v10.1: Regularized XGBoost on SWE-Router data',
-    'best_threshold': best_thr,
-}
-with open('/app/router_models/router_bundle_v10_fixed.pkl', 'wb') as f:
-    pickle.dump(v10_fixed, f)
-print(f"\nSaved v10.1 bundle")
-print("DONE!")

training/tune_feedback.py

@@ -1,98 +0,0 @@
-#!/usr/bin/env python3
-"""Tune execution-feedback thresholds for optimal cost-quality tradeoff."""
-import sys,json,random,math,pickle
-sys.path.insert(0,"/app")
-from collections import defaultdict
-from aco.classifier import TaskCostClassifier
-from aco.router import ModelCascadeRouter
-from aco.execution_feedback import ExecutionFeedbackRouter
-
-TIER_STR={1:0.35,2:0.55,3:0.80,4:0.93,5:0.97}
-TIER_COST={1:0.05,2:0.15,3:0.75,4:1.0,5:1.5}
-TASK_FLOOR={"legal_regulated":4,"long_horizon":3,"research":3,"coding":3,
-             "unknown_ambiguous":3,"quick_answer":1,"document_drafting":2,
-             "tool_heavy":2,"retrieval_heavy":2}
-
-TASKS = {
-  "quick_answer":["What is 2+2?","Explain quantum computing briefly.","Convert 100F to Celsius."],
-  "coding":["Write a Python function to reverse a linked list.","Fix a typo in the README.",
-      "Debug this critical production segfault NOW.","Just fix the typo in line 42."],
-  "research":["Research latest transformer advances.","Find sources comparing LoRA and full FT briefly."],
-  "document_drafting":["Draft project proposal for ML pipeline."],
-  "legal_regulated":["Review this contract for liability clauses.","Check GDPR compliance."],
-  "tool_heavy":["Search open issues and create summary."],
-  "retrieval_heavy":["Answer based on 50-page document."],
-  "long_horizon":["Plan 3-month roadmap.","Orchestrate multi-region deployment."],
-  "unknown_ambiguous":["Help me with this thing."],
-}
-
-classifier = TaskCostClassifier()
-router = ModelCascadeRouter(model_path="/app/router_models/router_bundle_v8.pkl")
-
-rng = random.Random(42)
-N = 2000
-
-def sim_logprobs(tier, diff, success, rng):
-    n = rng.randint(20, 150)
-    base = {1:-3.5,2:-2.5,3:-1.5,4:-0.7,5:-0.3}[tier]
-    base *= (1 + diff * 0.15)
-    lps = []
-    for _ in range(n):
-        noise = rng.gauss(0, 1.0 + diff*0.3)
-        lps.append(base + noise * (0.3 if success else 0.8))
-    return lps
-
-# Sweep thresholds
-print("="*80)
-print("FEEDBACK THRESHOLD SWEEP")
-print("="*80)
-print(f"\n{'EntropyThr':>12} {'LowConfThr':>12} {'Success':>10} {'AvgCost':>10} {'CostRed':>10} {'Gap':>10}")
-print("-"*65)
-
-frontier_sr = 0.901
-frontier_cost = 1.0
-
-best_score = -999
-best_config = None
-
-for ent_thr in [1.5, 2.0, 2.5, 3.0, 3.5, 4.0]:
-    for lc_thr in [0.05, 0.10, 0.15, 0.20, 0.25, 0.30]:
-        ef = ExecutionFeedbackRouter(entropy_threshold=ent_thr,
-            low_conf_ratio_threshold=lc_thr, tier_costs=TIER_COST,
-            task_floors=TASK_FLOOR)
-        rng.seed(42)
-        succ = 0; cost = 0.0
-        for i in range(N):
-            tt = rng.choice(list(TASKS.keys()))
-            req = rng.choice(TASKS[tt])
-            pred = classifier.classify(req)
-            r = router.route(req, tt, pred["difficulty"], pred)
-            tier = r.tier; diff = r.dynamic_difficulty
-            ps = TIER_STR[tier]**(diff*0.6)
-            initial_success = rng.random() < ps
-            lps = sim_logprobs(tier, diff, initial_success, rng)
-            signal = ef.analyze_output(lps, task_type=tt, current_tier=tier)
-            if signal.should_escalate and tier < 5:
-                final_tier = min(tier+1, 5)
-                final_tier = max(final_tier, TASK_FLOOR.get(tt,1))
-                ps2 = TIER_STR[final_tier]**(diff*0.6)
-                final_success = rng.random() < ps2
-                c = TIER_COST[tier] + TIER_COST[final_tier]
-                if final_success: succ += 1
-            else:
-                c = TIER_COST[tier]
-                if initial_success: succ += 1
-            cost += c
-        sr = succ/N; ac = cost/N
-        cr = (1-ac/frontier_cost)*100
-        gap = frontier_sr - sr
-        # Score: maximize success, minimize cost
-        score = sr*20 - ac*10
-        if score > best_score:
-            best_score = score
-            best_config = (ent_thr, lc_thr, sr, ac, cr, gap)
-        if ent_thr == 2.5 or ent_thr == 3.0:
-            print(f"{ent_thr:>12.1f} {lc_thr:>12.2f} {sr:>10.3f} {ac:>10.4f} {cr:>9.1f}% {gap:>10.3f}")
-
-print(f"\n\nBest config: entropy_thr={best_config[0]}, low_conf_thr={best_config[1]}")
-print(f"  success={best_config[2]:.3f}, cost={best_config[3]:.4f}, costRed={best_config[4]:.1f}%, gap={best_config[5]:.3f}")

training/verify_frontier.py

@@ -1,75 +0,0 @@
-"""Verify gpt-5.2 vs claude-opus-4.7 on SWE-bench.
-
-Both are tier 4 models. If gpt-5.2 is consistently cheaper AND more successful,
-it should replace claude-opus-4.7 as the frontier baseline. This changes the
-entire cost-reduction math.
-"""
-import json
-from collections import defaultdict
-from datasets import load_dataset
-
-MODELS = [
-    "deepseek-v4-flash", "gpt-5-nano", "gpt-5-mini", "deepseek-v3.2",
-    "gemini-2.5-pro", "claude-opus-4.7", "gpt-5.2", "gemini-3-pro",
-]
-
-print("Loading all 8 models...")
-traces = defaultdict(dict)
-for m in MODELS:
-    ds = load_dataset(f"SWE-Router/swebench-verified-{m}", split="test")
-    for row in ds:
-        traces[row["instance_id"]][m] = {
-            "resolved": row["resolved"],
-            "cost": float(row["instance_cost"]),
-        }
-
-claude = traces  # we have all data, compare claude-opus-4.7 vs gpt-5.2
-
-# Head-to-head
-claude_wins = 0  # claude resolves, gpt-5.2 doesn't
-gpt_wins = 0     # gpt-5.2 resolves, claude doesn't
-both_resolve = 0
-neither_resolve = 0
-claude_cost_total = 0
-gpt_cost_total = 0
-
-for tid, tt in traces.items():
-    c = tt.get("claude-opus-4.7", {})
-    g = tt.get("gpt-5.2", {})
-    
-    claude_cost_total += c.get("cost", 0)
-    gpt_cost_total += g.get("cost", 0)
-    
-    cr = c.get("resolved", False)
-    gr = g.get("resolved", False)
-    
-    if cr and not gr: claude_wins += 1
-    elif gr and not cr: gpt_wins += 1
-    elif cr and gr: both_resolve += 1
-    else: neither_resolve += 1
-
-N = len(traces)
-print(f"""
-HEAD-TO-HEAD: claude-opus-4.7 vs gpt-5.2 ({N} tasks)
-{'='*60}
-claude-opus-4.7  resolved: {sum(1 for tid in traces if traces[tid].get('claude-opus-4.7',{}).get('resolved'))}/{N} ({sum(1 for tid in traces if traces[tid].get('claude-opus-4.7',{}).get('resolved'))/N*100:.1f}%)  avg cost: ${claude_cost_total/N:.4f}
-gpt-5.2          resolved: {sum(1 for tid in traces if traces[tid].get('gpt-5.2',{}).get('resolved'))}/{N} ({sum(1 for tid in traces if traces[tid].get('gpt-5.2',{}).get('resolved'))/N*100:.1f}%)  avg cost: ${gpt_cost_total/N:.4f}
-
-Claude-only wins:  {claude_wins} ({claude_wins/N*100:.1f}%)
-GPT-only wins:     {gpt_wins} ({gpt_wins/N*100:.1f}%)
-Both resolve:      {both_resolve} ({both_resolve/N*100:.1f}%)
-Neither resolves:  {neither_resolve} ({neither_resolve/N*100:.1f}%)
-
-Conclusion: {"gpt-5.2 is CHEAPER AND MORE SUCCESSFUL → should be the new frontier baseline" if gpt_cost_total < claude_cost_total and sum(1 for tid in traces if traces[tid].get('gpt-5.2',{}).get('resolved')) > sum(1 for tid in traces if traces[tid].get('claude-opus-4.7',{}).get('resolved')) else "claude-4.7 wins cost or quality → keep as frontier"}
-""")
-
-# Also check: what is the TRUE cheapest model overall?
-print("\nALL MODELS RANKED BY COST-EFFICIENCY (resolved/cost):")
-ranked = []
-for m in MODELS:
-    resolved = sum(1 for tid in traces if traces[tid].get(m,{}).get("resolved", False))
-    total_cost = sum(traces[tid].get(m,{}).get("cost",0) for tid in traces)
-    ranked.append((m, resolved, total_cost/N, resolved/N, (resolved/N)/(max(total_cost/N,0.0001))))
-ranked.sort(key=lambda x: -x[4])
-for m, r, ac, rr, eff in ranked:
-    print(f"  {m:<25} {r:>4}/{N} ({rr*100:.1f}%)  ${ac:.4f}/task  efficiency={eff:.1f}")