| --- |
| license: apache-2.0 |
| tags: |
| - routing |
| - code |
| - mlx |
| - pid |
| - cascade |
| library_name: mlx |
| --- |
| |
| # Vibe Coding Router v5 |
|
|
| A three-tier cascaded router for coding tasks that routes prompts between: |
|
|
| - **Local**: Qwen3-Coder-Next (80B/3B active MoE, on-device via MLX) |
| - **Sonnet**: Claude Sonnet 4.6 (medium-complexity cloud) |
| - **Opus**: Claude Opus 4.6 (max-capability cloud) |
|
|
| ## What's New in v5 |
|
|
| v4 suffered from **inverted routing** — simple queries went to cloud while complex ones stayed local. Root cause: length-quality anti-correlation in training data combined with PID loss reward-weight amplification. v5 fixes this with: |
|
|
| 1. **7 new complexity features** (45 handcrafted total): `is_coding_task`, `junk_score`, `scope_breadth`, `imperative_verb_density`, `noun_phrase_density`, `interaction_complexity`, `requirement_clause_count` |
| 2. **Centered complexity premium**: Adjusts training margins by `premium * (complexity_score - center)` so complex tasks push toward cloud and simple tasks push toward local |
| 3. **Junk prompt clamping**: 75 junk/greeting prompts neutralized (p_teacher=0.5, margin=0.0) |
| 4. **Reward weight cap**: PID loss reward_weight capped at 0.5 to prevent outlier margin dominance |
|
|
| ## Architecture |
|
|
| Two cascaded binary MLP routers trained with **Privileged Information Distillation (PID)**: |
|
|
| - **Router A** (local vs cloud): 77-dim → [32, 16] → 1, dropout=0.2, LayerNorm+ReLU |
| - **Router B** (sonnet vs opus): 77-dim → [128, 64] → 1, dropout=0.0, LayerNorm+ReLU |
|
|
| Features: 45 handcrafted code features + 32 PCA-reduced sentence embeddings (all-MiniLM-L6-v2). |
|
|
| ## Training |
|
|
| - **Data**: 1,644 coding prompts with real quality scores from all three models |
| - **Judge**: GPT-5.4 scoring correctness, completeness, code quality, explanation |
| - **Loss**: PID (reward-weighted CE + KL divergence), β_kl=0.02, reward_cap=0.5 |
| - **Label smoothing**: ε=0.05, cost-aware margin for Router B (cost_premium=0.03) |
| - **Complexity premium**: 2.0, centered at 0.3 |
| - **HP sweep**: 108 configurations, 3-way split (1150 train / 247 val / 247 test) |
| - **Threshold A**: 0.60 (manually tuned for routing behavior — see note below) |
| - **Threshold B**: 0.474 (calibrated on validation set) |
| |
| ### Threshold Note |
| |
| The utility-optimal Router A threshold (0.01) routes almost nothing to local because cloud quality is genuinely equal or better on nearly all prompts. The manual threshold of 0.60 trades ~1.4% utility for correct routing intuition: simple/fast tasks run locally with zero latency, while complex tasks go to cloud. |
| |
| ## Real-World Routing (28 test queries, threshold_a=0.60) |
|
|
| | Category | Local | Sonnet | Opus | |
| |----------|-------|--------|------| |
| | Simple (8) | 5 (62%) | 0 | 3 (38%) | |
| | Medium (8) | 3 (38%) | 0 | 5 (62%) | |
| | Complex (6) | 1 (17%) | 1 (17%) | 4 (67%) | |
|
|
| v4 comparison: simple→local was 0/8 (now 5/8), complex→local was 6/6 (now 1/6). |
|
|
| ## Test Set Results (calibrated thresholds) |
|
|
| | Metric | Value | |
| |--------|-------| |
| | Utility | 0.6205 | |
| | Oracle Utility | 0.7179 | |
| | Regret | 0.0973 | |
|
|
| ## Files |
|
|
| - `router_a.safetensors` — Router A weights (32×16 MLP, 13KB) |
| - `router_b.safetensors` — Router B weights (128×64 MLP, 76KB) |
| - `config.json` — Model config, thresholds, HP, training results |
| - `scaler.pkl` — StandardScaler for feature normalization |
| - `embedding_extractor.pkl` — PCA-reduced sentence-transformers extractor |
| - `sweep_results.json` — Full 108-config HP sweep results |
|
|
| ## Usage |
|
|
| ```python |
| from router.three_tier_inference import ThreeTierRouter |
| |
| router = ThreeTierRouter("models/three_tier_v5") |
| result = router.route("Write a Python function to sort a list") |
| # result.decision: "local", "sonnet", or "opus" |
| # result.p_cloud: probability of cloud routing |
| # result.p_opus: probability of opus (if routed to cloud) |
| ``` |
|
|
