coding-router v6: ONNX (int8+fp32) + PyTorch weights, tokenizer, calibration, eval results

README.md

@@ -0,0 +1,234 @@
+<!--
+This file is the Hugging Face model card. When published to
+huggingface.co/afterbuild/coding-router it becomes the repo's README.md
+(HF renders the YAML frontmatter below as model metadata).
+-->
+---
+license: apache-2.0
+language:
+- en
+library_name: onnxruntime
+pipeline_tag: text-classification
+base_model: microsoft/deberta-v3-small
+tags:
+- coding-agent
+- routing
+- multi-head-classifier
+- onnx
+- deberta-v3
+- model-router
+metrics:
+- accuracy
+- f1
+---
+
+# coding-router (spawn-router)
+
+A compact, fast, **local-first** multi-head classifier for **coding-agent task
+routing**. Given a task prompt at kickoff, it predicts stable task properties; a
+downstream policy/config then maps those properties to a model, provider, and
+execution behavior. The classifier predicts the *ontology*; your config owns the
+*orchestration*.
+
+This is the model component of **spawn** — see
+[`Afterbuild/coding-router`](https://github.com/Afterbuild/coding-router) for the
+training code and [`spawn-gateway`](https://github.com/Afterbuild) for the local
+gateway that wraps Claude Code / Codex and routes with these weights.
+
+- **Backbone:** `microsoft/deberta-v3-small` (multi-head)
+- **Checkpoint:** v6 (final text-only training run)
+- **Inference:** torch-free ONNX path, ~7 ms/prompt CPU, ~140 MB deps
+- **Input:** text only (`current_text`), 256-token max
+
+## What it predicts
+
+```
+complexity: easy | medium | hard
+  + sub-dims (0..1 regression): reasoning_depth, scope_breadth,
+    domain_knowledge, spec_completeness (inverted: low spec = harder)
+task_type:  bugfix | feature | refactor | test | design | docs | migration | exploration
+risk:       low | medium | high
+  + sub-dims (0..1 regression): security_surface, data_sensitivity,
+    production_exposure, reversal_cost
++ per-head confidences (post-hoc temperature-scaled) and overall_confidence
+```
+
+- `complexity` → capability tier (small / mid / large model)
+- `task_type` → model specialty (e.g. design → Claude, systems → GPT, docs → small)
+- `risk` → tier bumper (easy + high-risk still routes capable) and confirmation gate
+
+> **Note on the ONNX export:** the published `.onnx` graphs emit the **three
+> classification heads only** (`complexity`, `task_type`, `risk` logits). The
+> regression sub-dimensions exist in the PyTorch model (`model.pt`) but are not in
+> the ONNX outputs yet — load `model.pt` with the training code if you need them.
+
+Routing is **kickoff-only**: classify once at task start and lock the model for
+the whole task cycle (no per-turn re-routing → no context thrash).
+
+## Files
+
+| File | What |
+|---|---|
+| `tiny_router.int8.onnx` | int8-quantized graph — **recommended for serving** (~164 MB) |
+| `tiny_router.onnx` | fp32 graph (~540 MB) |
+| `model.pt` | PyTorch state dict — for fine-tuning / sub-dim outputs (~565 MB) |
+| `spm.model` + `*tokenizer*.json` | SentencePiece (DeBERTa-v2/spm) tokenizer |
+| `model_config.json` | architecture + label maps |
+| `temperature_scaling.json` | per-head calibration temperatures |
+| `*_metrics.json`, `battery_results.json` | evaluation results |
+
+## Usage (ONNX, torch-free)
+
+Needs only `onnxruntime`, `numpy`, and `sentencepiece` — no torch, no transformers.
+
+```python
+import numpy as np
+import onnxruntime as ort
+import sentencepiece as spm
+
+MODEL_DIR = "."  # dir containing tiny_router.int8.onnx, spm.model, *.json
+MAX_LEN = 256
+LABELS = {
+    "complexity_logits": ["easy", "medium", "hard"],
+    "task_type_logits": ["bugfix", "feature", "refactor", "test",
+                          "design", "docs", "migration", "exploration"],
+    "risk_logits": ["low", "medium", "high"],
+}
+TEMPS = {  # from temperature_scaling.json; output name -> head temperature
+    "complexity_logits": 0.891251,
+    "task_type_logits": 0.707946,
+    "risk_logits": 1.059254,
+}
+
+sp = spm.SentencePieceProcessor(model_file=f"{MODEL_DIR}/spm.model")
+sess = ort.InferenceSession(f"{MODEL_DIR}/tiny_router.int8.onnx",
+                            providers=["CPUExecutionProvider"])
+
+def classify(text: str) -> dict:
+    # DeBERTa-v3 spm tokenizer: [CLS]=1 + pieces (truncated) + [SEP]=2
+    pieces = sp.encode(f"Current: {text}", out_type=int)[: MAX_LEN - 2]
+    ids = [1, *pieces, 2]
+    feeds = {  # structured inputs are text-only sentinels (no interaction context)
+        "input_ids": np.array([ids], dtype=np.int64),
+        "attention_mask": np.ones((1, len(ids)), dtype=np.int64),
+        "previous_action_id": np.array([0], dtype=np.int64),    # "none"
+        "previous_outcome_id": np.array([4], dtype=np.int64),   # "unknown"
+        "log_recency_seconds": np.array([0.0], dtype=np.float32),
+        "has_interaction": np.array([0], dtype=np.int64),
+        "has_recency": np.array([0], dtype=np.int64),
+    }
+    out = {o.name: v for o, v in zip(sess.get_outputs(), sess.run(None, feeds))}
+    result = {}
+    for name, labels in LABELS.items():
+        logits = out[name][0] / TEMPS[name]
+        p = np.exp(logits - logits.max()); p /= p.sum()
+        i = int(p.argmax())
+        result[name.replace("_logits", "")] = {
+            "label": labels[i], "confidence": round(float(p[i]), 4),
+        }
+    return result
+
+print(classify("refactor JWT key rotation in prod"))
+# {'complexity': {'label': 'medium', ...}, 'task_type': {'label': 'refactor', ...},
+#  'risk': {'label': 'medium', ...}}
+```
+
+## Evaluation
+
+Two complementary measures (eval scripts in
+[`Afterbuild/coding-router`](https://github.com/Afterbuild/coding-router):
+`scripts/eval_battery.py`, `eval.py`):
+
+**Locked kickoff battery** (83 hand-labeled probes, never in training — the
+canonical cross-version benchmark):
+
+| Metric | v6 |
+|---|---|
+| Unified kickoff score | **69.5%** |
+| Exact match (all 3 heads) | 37.4% |
+| complexity | 65.1% |
+| task_type | 78.3% |
+| risk | 65.1% |
+
+**Held-out test split** (n=174, mirrors the training distribution):
+
+| Head | Accuracy | Macro F1 |
+|---|---|---|
+| complexity | 67.8% | 68.1% |
+| task_type | 86.8% | 87.2% |
+| risk | 66.7% | 62.2% |
+| **Exact match** | **39.1%** | — |
+
+Sub-dimension regression R² (PyTorch model): reasoning_depth 0.51, scope_breadth
+0.47, spec_completeness 0.34, domain_knowledge 0.30; reversal_cost 0.55,
+production_exposure 0.51, data_sensitivity 0.25, security_surface 0.19.
+
+Calibration: per-head temperature scaling fit on validation. ECE on the held-out
+split is ~0.37 at the 0.8 automation threshold — **confidence is not yet
+well-calibrated for aggressive automation**; gate on it conservatively.
+
+## Intended use
+
+- Pick a capability tier / provider for a coding task **at kickoff**, before the
+  first expensive agent call.
+- Drive a confirmation gate for high-blast-radius work (risk/security/prod).
+- Spread work across tiers to reduce rate-limit pressure.
+
+**Out of scope:** per-turn routing; non-coding prompts; high-stakes autonomous
+action without a human gate; languages other than English (trained on English).
+
+## Limitations
+
+- **Cold-start ceiling.** Effort/blast-radius isn't fully derivable from prompt
+  text — `complexity=medium` and `risk=high` are the weakest bands, especially on
+  short imperatives. Production signals (overrides, retries, session duration) are
+  the intended path past this; this checkpoint predates that loop.
+- **Synthetic-label ceiling.** Much training data is LLM-labeled; expect a
+  ~75–80% ceiling per head until real disagreement signals are mixed in.
+- **ONNX omits sub-dims** (see note above).
+
+## Training
+
+- Backbone `microsoft/deberta-v3-small`, attention pooling, head dependencies,
+  3 softmax heads + 8 regression heads; `current_text_only` feature mode.
+- 5 epochs, batch 16, encoder LR 2e-5, head LR 1e-4, weight decay 0.01, warmup
+  0.1, seed 13; post-hoc per-head temperature scaling on validation.
+- Data: v6 mixed set (train 1141 / val 174 / test 174) — a mix of synthetic
+  coding-task prompts and real coding-agent kickoff prompts. **The merged
+  training set is not distributed** (it embeds third-party trace text and
+  personal usage traces); the synthetic seed data and the full data pipeline
+  are in the code repo. See "Training data provenance" below.
+
+## Training data provenance
+
+Disclosed in full so downstream users can do their own diligence:
+
+- **Synthetic coding-task prompts** (majority of the mix) — written by Claude
+  sub-agents and hand-labeled; included in the code repo.
+- **SWE-bench problem statements** — used only as Claude-paraphrased
+  short-imperative prompts (no code, patches, or full issue text). The
+  SWE-bench benchmark code is MIT; the aggregated issue text is owned by its
+  authors and the HF dataset card carries no license tag.
+- **Public coding-agent trace datasets** (`badlogicgames/pi-mono`,
+  `armand0e/gpt-5.5-agent`, `lewtun/ml-intern-sessions`) — kickoff prompts
+  extracted and labeled. These carry `license: other` or no license; their raw
+  text is **not redistributed** here.
+- **The author's own local agent traces** — first-task prompts only; not
+  redistributed.
+- Labels and paraphrases were produced with **Anthropic Claude**; per
+  Anthropic's Commercial Terms, outputs are customer-owned. No other
+  provider's models were used for generation or labeling.
+
+The model is a non-generative classifier (three softmax heads over 256-token
+inputs); it emits logits, not text, and cannot reproduce training data.
+
+## Credits & provenance
+
+- Scaffolding began as a fork of **[tiny-router](https://github.com/UdaraJay/tiny-router)
+  by Udara Jay** (MIT); the ontology, data, heads, and serving path were rebuilt
+  for coding-agent routing.
+- Backbone: **DeBERTa-v3** (He et al.; `microsoft/deberta-v3-small`, MIT).
+- Related prior art: Vercel v0 Auto and NVIDIA's
+  prompt-task-and-complexity-classifier.
+- **License: Apache-2.0** (weights), with the training-data provenance
+  disclosed above; the training/serving code repo is MIT.

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "[MASK]": 128000
+}

battery_results.json

@@ -0,0 +1,37 @@
+{
+  "battery_size": 83,
+  "unified_score": 0.6948,
+  "exact_match": 0.3735,
+  "per_head": {
+    "complexity": 0.6506,
+    "task_type": 0.7831,
+    "risk": 0.6506
+  },
+  "per_tier": {
+    "complexity": {
+      "easy": "16/24",
+      "hard": "15/20",
+      "medium": "23/39"
+    },
+    "task_type": {
+      "bugfix": "11/14",
+      "design": "1/3",
+      "docs": "2/5",
+      "exploration": "3/4",
+      "feature": "28/29",
+      "migration": "7/11",
+      "refactor": "10/14",
+      "test": "3/3"
+    },
+    "risk": {
+      "high": "14/28",
+      "low": "26/31",
+      "medium": "14/24"
+    }
+  },
+  "per_length_exact_match": {
+    "short": "23/63",
+    "medium": "5/12",
+    "long": "3/8"
+  }
+}

metrics.json

@@ -0,0 +1,323 @@
+{
+  "per_head": {
+    "complexity": {
+      "accuracy": 0.6954,
+      "macro_f1": 0.7011,
+      "per_label": {
+        "easy": {
+          "precision": 0.7258,
+          "recall": 0.75,
+          "f1": 0.7377,
+          "support": 60
+        },
+        "medium": {
+          "precision": 0.6232,
+          "recall": 0.6418,
+          "f1": 0.6324,
+          "support": 67
+        },
+        "hard": {
+          "precision": 0.7674,
+          "recall": 0.7021,
+          "f1": 0.7333,
+          "support": 47
+        }
+      },
+      "confusion_matrix": [
+        [
+          45,
+          15,
+          0
+        ],
+        [
+          14,
+          43,
+          10
+        ],
+        [
+          3,
+          11,
+          33
+        ]
+      ]
+    },
+    "task_type": {
+      "accuracy": 0.9138,
+      "macro_f1": 0.9093,
+      "per_label": {
+        "bugfix": {
+          "precision": 0.9565,
+          "recall": 0.88,
+          "f1": 0.9167,
+          "support": 25
+        },
+        "feature": {
+          "precision": 0.9302,
+          "recall": 0.9524,
+          "f1": 0.9412,
+          "support": 42
+        },
+        "refactor": {
+          "precision": 0.8696,
+          "recall": 0.9524,
+          "f1": 0.9091,
+          "support": 21
+        },
+        "test": {
+          "precision": 1.0,
+          "recall": 0.8571,
+          "f1": 0.9231,
+          "support": 14
+        },
+        "design": {
+          "precision": 0.9375,
+          "recall": 0.9375,
+          "f1": 0.9375,
+          "support": 16
+        },
+        "docs": {
+          "precision": 0.9333,
+          "recall": 0.9333,
+          "f1": 0.9333,
+          "support": 15
+        },
+        "migration": {
+          "precision": 0.9167,
+          "recall": 1.0,
+          "f1": 0.9565,
+          "support": 22
+        },
+        "exploration": {
+          "precision": 0.7778,
+          "recall": 0.7368,
+          "f1": 0.7568,
+          "support": 19
+        }
+      },
+      "confusion_matrix": [
+        [
+          22,
+          1,
+          0,
+          0,
+          0,
+          0,
+          0,
+          2
+        ],
+        [
+          0,
+          40,
+          1,
+          0,
+          0,
+          0,
+          0,
+          1
+        ],
+        [
+          0,
+          0,
+          20,
+          0,
+          0,
+          0,
+          1,
+          0
+        ],
+        [
+          1,
+          1,
+          0,
+          12,
+          0,
+          0,
+          0,
+          0
+        ],
+        [
+          0,
+          0,
+          1,
+          0,
+          15,
+          0,
+          0,
+          0
+        ],
+        [
+          0,
+          0,
+          0,
+          0,
+          0,
+          14,
+          0,
+          1
+        ],
+        [
+          0,
+          0,
+          0,
+          0,
+          0,
+          0,
+          22,
+          0
+        ],
+        [
+          0,
+          1,
+          1,
+          0,
+          1,
+          1,
+          1,
+          14
+        ]
+      ]
+    },
+    "risk": {
+      "accuracy": 0.6954,
+      "macro_f1": 0.6554,
+      "per_label": {
+        "low": {
+          "precision": 0.8,
+          "recall": 0.8276,
+          "f1": 0.8136,
+          "support": 87
+        },
+        "medium": {
+          "precision": 0.5333,
+          "recall": 0.6038,
+          "f1": 0.5664,
+          "support": 53
+        },
+        "high": {
+          "precision": 0.7083,
+          "recall": 0.5,
+          "f1": 0.5862,
+          "support": 34
+        }
+      },
+      "confusion_matrix": [
+        [
+          72,
+          13,
+          2
+        ],
+        [
+          16,
+          32,
+          5
+        ],
+        [
+          2,
+          15,
+          17
+        ]
+      ]
+    }
+  },
+  "overall": {
+    "exact_match": 0.4655,
+    "macro_average_f1": 0.7553,
+    "automation_safe_accuracy": 0.6935,
+    "automation_safe_coverage": 0.3563,
+    "confidence_threshold": 0.8,
+    "confidence_calibration": {
+      "ece": 0.303072,
+      "bins": [
+        {
+          "range": [
+            0.5,
+            0.6
+          ],
+          "count": 5,
+          "avg_confidence": 0.586,
+          "accuracy": 0.0
+        },
+        {
+          "range": [
+            0.6,
+            0.7
+          ],
+          "count": 48,
+          "avg_confidence": 0.6575,
+          "accuracy": 0.2917
+        },
+        {
+          "range": [
+            0.7,
+            0.8
+          ],
+          "count": 59,
+          "avg_confidence": 0.7473,
+          "accuracy": 0.4068
+        },
+        {
+          "range": [
+            0.8,
+            0.9
+          ],
+          "count": 33,
+          "avg_confidence": 0.8512,
+          "accuracy": 0.5758
+        },
+        {
+          "range": [
+            0.9,
+            1.0
+          ],
+          "count": 29,
+          "avg_confidence": 0.9333,
+          "accuracy": 0.8276
+        }
+      ]
+    }
+  },
+  "temperature_scaling": {
+    "method": "per_head_temperature_scaling",
+    "per_head": {
+      "complexity": 0.891251,
+      "task_type": 0.707946,
+      "risk": 1.059254
+    }
+  },
+  "complexity_subdims": {
+    "reasoning_depth": {
+      "mae": 0.1069,
+      "r2": 0.5888
+    },
+    "spec_completeness": {
+      "mae": 0.1033,
+      "r2": 0.3667
+    },
+    "scope_breadth": {
+      "mae": 0.1076,
+      "r2": 0.5044
+    },
+    "domain_knowledge": {
+      "mae": 0.1036,
+      "r2": 0.4405
+    }
+  },
+  "risk_subdims": {
+    "security_surface": {
+      "mae": 0.1517,
+      "r2": 0.1937
+    },
+    "data_sensitivity": {
+      "mae": 0.1184,
+      "r2": 0.3094
+    },
+    "production_exposure": {
+      "mae": 0.1182,
+      "r2": 0.6323
+    },
+    "reversal_cost": {
+      "mae": 0.1045,
+      "r2": 0.6316
+    }
+  }
+}

model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:168ffa72cd2ec515f227d519ce7761a03ef3cb551dffda552ea0de9beb21c5d6
+size 565346323

model_config.json

@@ -0,0 +1,57 @@
+{
+  "encoder_name": "microsoft/deberta-v3-small",
+  "dropout": 0.1,
+  "action_vocab": [
+    "none",
+    "create",
+    "update",
+    "send",
+    "store",
+    "route",
+    "schedule",
+    "dismissed",
+    "clarify",
+    "search",
+    "notify",
+    "cancel",
+    "complete",
+    "other"
+  ],
+  "outcome_vocab": [
+    "success",
+    "pending",
+    "failed",
+    "cancelled",
+    "unknown"
+  ],
+  "label_maps": {
+    "complexity": [
+      "easy",
+      "medium",
+      "hard"
+    ],
+    "task_type": [
+      "bugfix",
+      "feature",
+      "refactor",
+      "test",
+      "design",
+      "docs",
+      "migration",
+      "exploration"
+    ],
+    "risk": [
+      "low",
+      "medium",
+      "high"
+    ]
+  },
+  "structured_hidden_dim": 32,
+  "recency_embed_dim": 8,
+  "pooling_type": "attention",
+  "use_head_dependencies": true,
+  "dependency_hidden_dim": 32,
+  "feature_mode": "current_text_only",
+  "max_length": 256,
+  "recency_max": 3600
+}

onnx_metadata.json

@@ -0,0 +1,37 @@
+{
+  "model_file": "tiny_router.onnx",
+  "feature_mode": "current_text_only",
+  "heads": [
+    "complexity",
+    "task_type",
+    "risk"
+  ],
+  "max_length": 256,
+  "label_maps": {
+    "complexity": [
+      "easy",
+      "medium",
+      "hard"
+    ],
+    "task_type": [
+      "bugfix",
+      "feature",
+      "refactor",
+      "test",
+      "design",
+      "docs",
+      "migration",
+      "exploration"
+    ],
+    "risk": [
+      "low",
+      "medium",
+      "high"
+    ]
+  },
+  "temperature_scaling": {
+    "complexity": 0.891251,
+    "task_type": 0.707946,
+    "risk": 1.059254
+  }
+}

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

temperature_scaling.json

@@ -0,0 +1,9 @@
+{
+  "method": "per_head_temperature_scaling",
+  "source_split": "validation",
+  "per_head": {
+    "complexity": 0.891251,
+    "task_type": 0.707946,
+    "risk": 1.059254
+  }
+}

test_metrics.json

@@ -0,0 +1,323 @@
+{
+  "per_head": {
+    "complexity": {
+      "accuracy": 0.6782,
+      "macro_f1": 0.6806,
+      "per_label": {
+        "easy": {
+          "precision": 0.5758,
+          "recall": 0.8636,
+          "f1": 0.6909,
+          "support": 44
+        },
+        "medium": {
+          "precision": 0.7941,
+          "recall": 0.5745,
+          "f1": 0.6667,
+          "support": 94
+        },
+        "hard": {
+          "precision": 0.65,
+          "recall": 0.7222,
+          "f1": 0.6842,
+          "support": 36
+        }
+      },
+      "confusion_matrix": [
+        [
+          38,
+          6,
+          0
+        ],
+        [
+          26,
+          54,
+          14
+        ],
+        [
+          2,
+          8,
+          26
+        ]
+      ]
+    },
+    "task_type": {
+      "accuracy": 0.8678,
+      "macro_f1": 0.8718,
+      "per_label": {
+        "bugfix": {
+          "precision": 0.875,
+          "recall": 0.84,
+          "f1": 0.8571,
+          "support": 25
+        },
+        "feature": {
+          "precision": 0.875,
+          "recall": 0.8333,
+          "f1": 0.8537,
+          "support": 42
+        },
+        "refactor": {
+          "precision": 0.84,
+          "recall": 1.0,
+          "f1": 0.913,
+          "support": 21
+        },
+        "test": {
+          "precision": 1.0,
+          "recall": 0.8571,
+          "f1": 0.9231,
+          "support": 14
+        },
+        "design": {
+          "precision": 0.8667,
+          "recall": 0.8125,
+          "f1": 0.8387,
+          "support": 16
+        },
+        "docs": {
+          "precision": 0.9286,
+          "recall": 0.8667,
+          "f1": 0.8966,
+          "support": 15
+        },
+        "migration": {
+          "precision": 0.9524,
+          "recall": 0.9091,
+          "f1": 0.9302,
+          "support": 22
+        },
+        "exploration": {
+          "precision": 0.6957,
+          "recall": 0.8421,
+          "f1": 0.7619,
+          "support": 19
+        }
+      },
+      "confusion_matrix": [
+        [
+          21,
+          0,
+          2,
+          0,
+          0,
+          0,
+          1,
+          1
+        ],
+        [
+          0,
+          35,
+          1,
+          0,
+          1,
+          0,
+          0,
+          5
+        ],
+        [
+          0,
+          0,
+          21,
+          0,
+          0,
+          0,
+          0,
+          0
+        ],
+        [
+          1,
+          1,
+          0,
+          12,
+          0,
+          0,
+          0,
+          0
+        ],
+        [
+          0,
+          1,
+          1,
+          0,
+          13,
+          1,
+          0,
+          0
+        ],
+        [
+          0,
+          1,
+          0,
+          0,
+          0,
+          13,
+          0,
+          1
+        ],
+        [
+          1,
+          1,
+          0,
+          0,
+          0,
+          0,
+          20,
+          0
+        ],
+        [
+          1,
+          1,
+          0,
+          0,
+          1,
+          0,
+          0,
+          16
+        ]
+      ]
+    },
+    "risk": {
+      "accuracy": 0.6667,
+      "macro_f1": 0.6217,
+      "per_label": {
+        "low": {
+          "precision": 0.8068,
+          "recall": 0.8353,
+          "f1": 0.8208,
+          "support": 85
+        },
+        "medium": {
+          "precision": 0.5102,
+          "recall": 0.4545,
+          "f1": 0.4808,
+          "support": 55
+        },
+        "high": {
+          "precision": 0.5405,
+          "recall": 0.5882,
+          "f1": 0.5634,
+          "support": 34
+        }
+      },
+      "confusion_matrix": [
+        [
+          71,
+          11,
+          3
+        ],
+        [
+          16,
+          25,
+          14
+        ],
+        [
+          1,
+          13,
+          20
+        ]
+      ]
+    }
+  },
+  "overall": {
+    "exact_match": 0.3908,
+    "macro_average_f1": 0.7247,
+    "automation_safe_accuracy": 0.6167,
+    "automation_safe_coverage": 0.3448,
+    "confidence_threshold": 0.8,
+    "confidence_calibration": {
+      "ece": 0.369827,
+      "bins": [
+        {
+          "range": [
+            0.5,
+            0.6
+          ],
+          "count": 2,
+          "avg_confidence": 0.5761,
+          "accuracy": 0.0
+        },
+        {
+          "range": [
+            0.6,
+            0.7
+          ],
+          "count": 52,
+          "avg_confidence": 0.6519,
+          "accuracy": 0.3077
+        },
+        {
+          "range": [
+            0.7,
+            0.8
+          ],
+          "count": 60,
+          "avg_confidence": 0.7476,
+          "accuracy": 0.25
+        },
+        {
+          "range": [
+            0.8,
+            0.9
+          ],
+          "count": 41,
+          "avg_confidence": 0.8439,
+          "accuracy": 0.5366
+        },
+        {
+          "range": [
+            0.9,
+            1.0
+          ],
+          "count": 19,
+          "avg_confidence": 0.9391,
+          "accuracy": 0.7895
+        }
+      ]
+    }
+  },
+  "temperature_scaling": {
+    "method": "per_head_temperature_scaling",
+    "per_head": {
+      "complexity": 0.891251,
+      "task_type": 0.707946,
+      "risk": 1.059254
+    }
+  },
+  "complexity_subdims": {
+    "reasoning_depth": {
+      "mae": 0.1117,
+      "r2": 0.513
+    },
+    "spec_completeness": {
+      "mae": 0.1165,
+      "r2": 0.3356
+    },
+    "scope_breadth": {
+      "mae": 0.1082,
+      "r2": 0.4687
+    },
+    "domain_knowledge": {
+      "mae": 0.1199,
+      "r2": 0.3003
+    }
+  },
+  "risk_subdims": {
+    "security_surface": {
+      "mae": 0.1627,
+      "r2": 0.1921
+    },
+    "data_sensitivity": {
+      "mae": 0.1182,
+      "r2": 0.2477
+    },
+    "production_exposure": {
+      "mae": 0.1387,
+      "r2": 0.5113
+    },
+    "reversal_cost": {
+      "mae": 0.1103,
+      "r2": 0.5463
+    }
+  }
+}

tiny_router.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d1b18a823744e68e89e0bd20cdd1a0b386442b3c08e7cce0a58992e593c13f2
+size 171752203

tiny_router.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31ebcb1174adf14fc0bc8609f6b8c5e92f2f40b5f5c96c3cffdb864d072c237b
+size 565776017

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}