Instructions to use thundercode/SatQuery with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use thundercode/SatQuery with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
04 β The Router
Parent: Architecture hub Β· Sibling chapters: 01 System overview Β· 03 Request lifecycle Β· 05 Specialists Β· 06 Evidence and confidence Β· 07 Configuration freeze
Primary sources read for this chapter (all under C:/Users/anish/satquery-ai/ unless stated):
| Source | Lines | What it establishes here |
|---|---|---|
router/encoder.py |
190 | FrozenEncoder, VERIFIED_TOKENIZER_MAX_LENGTH = 256, VERIFIED_EMBEDDING_DIM = 384, the max_length β€ 256 refusal, build_encoder, the frozen-revision contract probe |
router/adapter.py |
174 | IntentAdapter β all seven sub-modules, AdapterOutput, _init_weights, the forward shape guards, config_dict / from_config_dict / num_parameters |
router/label_space.py |
129 | TASK_CLASSES, MODALITY_CLASSES, BINARY_HEADS, TEMPORAL_TASKS, DUAL_MODALITY_TASKS, SPATIAL_TASKS, default_attributes, default_modality |
router/fallback.py |
333 | the seven term tables, the eight ordered rules, every rule confidence, self_check()'s ten curated cases, is_available() |
router/classifier.py |
475 | IntentRouter, confidence_threshold = 0.70, _predict_learned and the coherence repair, route()'s three-step order, RouterPrediction.to_trace(), save_adapter / load_adapter |
router/dataset.py |
707 | RouterExample, CURATED (70 entries), TEMPLATES (45 entries), SUBJECTS, HARD_NEGATIVE_PREFIX, RouterCorpus.validate/dedupe, split_by_group, split_leakage_report |
router/train.py |
753 | train_router, SplitMetrics, TrainingResult.gate2_passed, embed_corpus_cached, class weights, loss composition, _stub_embeddings |
artifacts/router/router_adapter_v001/metadata.json |
626 | the shipped adapter artifact: 51,725 params, corpus 576 / 54 groups, all three split metric blocks, the full 60-epoch history |
artifacts/router/threshold_sweep_val.json |
50 rows | the release-level threshold sweep: val only, n_test_examples_scored: 0, test_split_touched: false, shipped threshold 0.70, selected 0.76 |
core/planner.py |
662 | PolicyPlanner β LEXICAL_FALLBACK_DISCOUNT = 0.75, SETTLED_CONFIDENCE = 0.60, TASK_CAPABILITY, CAPABILITY_ASSETS, PlanStep / PlanRefusal / ExecutionPlan, the closed refusal list, the Β§3.5 widening rules |
core/schemas.py |
462 | Task (7 members), Modality (4), Intent + its _consistency validator |
configs/base.yaml Β§router |
β | every configured value: model, revision, max_length: 128, embedding_dim: 384, hidden_dim: 128, dropout: 0.10, confidence_threshold: 0.70, num_tasks: 6, and the six training keys |
frontend/assets/js/mission.js |
1β300 | interpret() (line 70) and chooseTask() (line 184) β the two-stage frontend router, with the four defect-fix comments verbatim |
docs/PHASE4_ROUTER_REPORT.md |
179 | the phase record: Gate 2 PASS, F4-1/F4-2/F4-3, four defects found by running, the standing caveats |
release/repo/docs/RESEARCH_NOTES.md Β§3, Β§6 |
β | the router-defect case study, the live-pass table, the two run ids, and the interpret()/chooseTask() asymmetry ruling |
release/repo/README.md Β§"Routing and the execution trace", Β§"Known limitations" rows 2 and 8 |
β | the two-stage description, the five-head table, the confidence-gate diagram, the router residuals |
1. Where the router sits, and the verb it owns
docs/ARCHITECTURE_FREEZE.md Β§5 assigns each layer exactly one verb:
Router understands; policy engine decides; specialists compute; VLM explains; evidence engine proves.
The router is the first verb. It takes a string and produces a typed reading of what the
string is asking for β nothing more. It does not choose what runs, it does not touch pixels, and
it does not answer anything. core/schemas.py:95 states the contract in the model's own docstring:
class Intent(BaseModel):
"""Output of the learned router. Advisory only β the controller decides."""
That single line is load-bearing for the whole chapter. Everything downstream of the router is
allowed β required β to overrule it. core/planner.py:13-15 restates it as the planner's reason to
exist:
The planner is the only component permitted to choose what runs. Its input is a
RouterPrediction; its output is anExecutionPlan. [β¦] The router'sIntentis an input to a decision, never the decision.
1.1 The router is one of two stages, and the split is deliberate
Routing happens twice in this system, in two languages, with two different jobs. This is the single most misread part of the architecture, so it is stated first.
| Stage | Where | Blind to asset count? | Job |
|---|---|---|---|
Reading β interpret() |
frontend/assets/js/mission.js:70 |
yes β text only | what does the question mean? |
Dispatch β chooseTask() |
frontend/assets/js/mission.js:184 |
no β reads assetCount |
what can actually be computed with the assets present? |
Reading β IntentRouter.route() |
router/classifier.py:311 |
yes | the server-side analogue: query β Intent |
Dispatch β PolicyPlanner.plan() |
core/planner.py:276 |
no β reads len(request.assets) |
the server-side analogue: Intent β ExecutionPlan |
The frontend pair and the server pair are structurally the same shape and were written independently. Β§35βΒ§38 covers the frontend pair in full; Β§39βΒ§43 covers the server pair.
1.2 The pipeline
flowchart TB
Q["query : str"] --> ENC["FrozenEncoder.encode_one<br/>MiniLM-L6-v2 @ 1110a243fdf4<br/>384-d, frozen"]
Q --> FB["lexical_route(query)<br/>deterministic, ordered rules"]
ENC --> AD["IntentAdapter<br/>5 heads"]
AD --> GATE{"task softmax max<br/>β₯ confidence_threshold<br/>(0.70)"}
GATE -->|yes| LEARNED["Intent(source='learned')"]
GATE -->|no| CMP{"fallback confidence<br/>β₯ learned confidence?"}
FB --> CMP
CMP -->|yes| FALL["Intent(source='lexical_fallback')"]
CMP -->|no| LEARNED
LEARNED --> PLAN["PolicyPlanner.plan()<br/>core/planner.py:276"]
FALL --> PLAN
PLAN --> EP["ExecutionPlan<br/>steps | refusal"]
style ENC fill:#1f6feb22,stroke:#1f6feb
style AD fill:#1f6feb22,stroke:#1f6feb
style PLAN fill:#8957e522,stroke:#8957e5
The encoder and adapter are the learned path. lexical_route is the fallback path. Both
always run when both are available; Β§23 explains exactly how route() arbitrates between them.
1.3 Status of every component in this chapter
Statuses follow DOCS_STYLE_GUIDE.md Β§2. Nothing below is upgraded.
| Component | Status | Basis |
|---|---|---|
FrozenEncoder β load, freeze, guard max_length β€ 256 |
IMPLEMENTED + VERIFIED |
router/encoder.py; the contract probe in the module docstring (2026-09-16, sentence-transformers 6.0.1) |
MiniLM revision pin 1110a243fdf4 |
VERIFIED reachable |
router/encoder.py:7; recorded again in the shipped metadata.json |
IntentAdapter β 5 heads, 51,725 params |
IMPLEMENTED + MEASURED |
router/adapter.py; metadata.json adapter.num_parameters: 51725 |
| Adapter trained on the shipped corpus | MEASURED |
metadata.json β 60 epochs, 4.92 s, artifacts/router/router_adapter_v001 |
| Gate 2 (task accuracy β₯ 0.95, all 6 classes measured) on the training-time test split | MEASURED β reported PASS at 0.975, n=80 |
metadata.json.metrics.test; docs/PHASE4_ROUTER_REPORT.md |
| Release-level threshold sweep | MEASURED β val only, n_test_examples_scored: 0 |
artifacts/router/threshold_sweep_val.json |
| Router test split touched by the release evaluation | NOT RUN |
threshold_sweep_val.json β test_split_touched: false; README.md limitation 2 |
| Lexical fallback β all eight rules | IMPLEMENTED + VERIFIED |
router/fallback.py; self_check() ten cases |
| Fallback agrees with the learned router on the curated hard negatives | VERIFIED β 0.800 hard-negative accuracy on test |
metadata.json.metrics.test.hard_negative_accuracy: 0.8 |
IntentRouter.route() two-path arbitration |
IMPLEMENTED + VERIFIED |
router/classifier.py:311-339 |
PolicyPlanner β pure, no torch, no filesystem |
IMPLEMENTED |
core/planner.py; the TYPE_CHECKING-only import of RouterPrediction |
Frontend interpret() / chooseTask() two-stage split |
IMPLEMENTED + VERIFIED live |
frontend/assets/js/mission.js; 3 passes Γ 8 cases, 24 runs |
Router-defect fix (built, \barea\b, new, chang stem) |
RESOLVED β deployed and verified live |
docs/RESEARCH_NOTES.md Β§3.3; run ids run_467ffa406f22, run_46980ba55c62 |
| Router calibration | NOT RUN β the 0.70 threshold is uncalibrated by the phase's own admission |
docs/PHASE4_ROUTER_REPORT.md Β§"Standing caveats" |
| Router accuracy as a benchmark claim | NOT RUN β the corpus is synthetic (576 examples / 54 groups) |
docs/PHASE4_ROUTER_REPORT.md; threshold_sweep_val.json β corpus_limited: true |
Router lexical residuals ("What is the new runway?", "How much built-up area was added?") |
OPEN |
README.md limitation 8 |
1.4 The three vocabularies, and why there are three
A reader who compares router/label_space.py to core/schemas.py to GET /api/capabilities finds
three different sets of task names. That is not drift. README.md:136-151 documents it as
deliberate, and it is worth restating because every later chapter depends on it.
| Vocabulary | Where | Members | Count |
|---|---|---|---|
core.schemas.Task |
core/schemas.py:35-47 |
vqa, caption, grounding, change, optical_sar, change_vqa, unsupported |
7 |
router.label_space.TASK_CLASSES |
router/label_space.py:26-33 |
vqa, caption, grounding, change, optical_sar, unsupported |
6 |
GET /api/capabilities |
live capability contract | vqa, caption, grounding, change, change_vqa, optical_sar |
6 |
The differences are each a design decision:
unsupportedis inTaskand inTASK_CLASSESbut not in the capability list.unsupportedis a routing outcome β "this is not a satellite-imagery question" β not a servable capability.core/planner.py:120says so in a comment onTASK_CAPABILITY: "UNSUPPORTEDis deliberately absent β it is a refusal, not a capability."change_vqais inTaskand in the capability list but not inTASK_CLASSES. It is reached through the change family rather than being a separate router class: the router has nochange_vqahead output, andcore/planner.py:470-491widens achange+language_outputprediction into achange_vqastep.core/schemas.py:41-46explains why the three are not interchangeable:#: R-02. Two temporally corresponding assets plus a change-oriented question #: in, a short answer out. Distinct from CHANGE, which is the change #: *detector* and returns a spatial change map with no language output, and #: distinct from VQA, which answers about ONE asset. The three are not #: interchangeable and the planner must not substitute one for another. CHANGE_VQA = "change_vqa"The reconciliation happens in one place. Because
AnalysisRequestisextra="forbid"andforce_taskis typedTask | None(core/schemas.py:412-418), any string outside theTaskenum is a 422. The frontend therefore maps the router's reading to the server enum through a single table,ROUTER_TASK_TO_SERVER(frontend/assets/js/mission.js:148-155).
router/classifier.py guards the router-side alignment at import time, so a label added to
label_space.py without a schema mapping cannot ship silently:
def _assert_schema_alignment() -> None:
missing_tasks = [t for t in TASK_CLASSES if t not in _TASK_TO_SCHEMA]
if missing_tasks:
raise RoutingError(
f"label_space TASK_CLASSES contains labels with no schema mapping: "
f"{missing_tasks}"
)
missing_mods = [m for m in MODALITY_CLASSES if m not in _MODALITY_TO_SCHEMA]
...
_assert_schema_alignment()
(router/classifier.py:62-77.) The guard runs at module import, not at first call, so the
failure is a startup failure rather than a per-request one.
Part A β The frozen encoder
2. FrozenEncoder β the contract, probed not assumed
router/encoder.py opens with the verified contract, and the phrase verified is literal: these
were probed, not read from documentation.
Verified contract (Phase 4 probe, 2026-09-16, sentence-transformers 6.0.1):
SentenceTransformer(
model_name_or_path='sentence-transformers/all-MiniLM-L6-v2',
revision='1110a243fdf4', # pinned, verified reachable
device='cpu',
)
.get_sentence_embedding_dimension() -> 384
.tokenizer.model_max_length -> 256 (finding F4-1)
.max_seq_length -> 256
params -> 22,713,216
encode(64 queries, CPU) -> 0.118 s
(router/encoder.py:3-14.)
| Property | Value | Source |
|---|---|---|
| Model | sentence-transformers/all-MiniLM-L6-v2 |
configs/base.yaml Β§router.model; metadata.json.encoder.model |
| Revision | 1110a243fdf4 |
configs/base.yaml Β§router.revision; metadata.json.encoder.revision |
| Embedding dimension | 384 | probe; VERIFIED_EMBEDDING_DIM = 384 |
| Tokenizer ceiling | 256 | probe; VERIFIED_TOKENIZER_MAX_LENGTH = 256 |
Configured max_length |
128 | configs/base.yaml Β§router.max_length; metadata.json.encoder.max_length |
| Parameters | 22,713,216 | probe; metadata.json.encoder.parameters |
| Approximate on-disk size | 90.9 MB | configs/base.yaml comment on the revision pin |
| Throughput | 0.118 s for 64 queries, CPU | probe |
| Device | cpu (config default auto β config.device_preference) |
router/encoder.py:168-182 |
2.1 The two module constants
#: The tokenizer's own ceiling, verified by probe. Truncating above this is a
#: silent no-op, so the encoder refuses rather than pretending.
VERIFIED_TOKENIZER_MAX_LENGTH = 256
#: Verified embedding dimension for all-MiniLM-L6-v2.
VERIFIED_EMBEDDING_DIM = 384
(router/encoder.py:30-35.)
3. Finding F4-1 β max_length 128 is not the model's limit
This is the most-quoted finding in the router, and it is worth stating precisely, because the distinction between the model's ceiling and our truncation is exactly the kind of thing that silently rots.
What the config said. max_length: 128.
What the probe found. The MiniLM tokenizer's own ceiling is 256
(tokenizer.model_max_length β 256, max_seq_length β 256).
Why 128 and not 256. The truncation is deliberate and sits well inside the ceiling. The
configured rationale, verbatim from configs/base.yaml:
# Finding F4-1: the MiniLM tokenizer's own ceiling is 256 (verified by probe).
# 128 is a deliberate truncation well inside that ceiling, not the model
# limit. Satellite queries are short; halving the sequence halves attention
# cost for no measurable accuracy loss. The encoder asserts this value is
# <= 256, because truncating above the ceiling is a silent no-op.
max_length: 128
Why the encoder must refuse rather than merely document. Truncating above the tokenizer's
ceiling is a silent no-op β the tokenizer will not truncate past what it can encode, so a
max_length of 512 would appear to be honoured while doing nothing at all. That is a control that
looks like it works and does not. FrozenEncoder.__init__ therefore raises:
if max_length < 1:
raise ModelLoadError(f"max_length must be >= 1, got {max_length}")
if max_length > VERIFIED_TOKENIZER_MAX_LENGTH:
raise ModelLoadError(
f"max_length={max_length} exceeds the MiniLM tokenizer ceiling of "
f"{VERIFIED_TOKENIZER_MAX_LENGTH}; truncation would be a silent no-op"
)
(router/encoder.py:54-60.)
The subtlety most readers miss. The configured value is not merely validated β it is applied.
Setting max_seq_length rewrites the tokenizer's truncation limit, so the config value is enforced
rather than documented:
# Apply the configured truncation for real (finding F4-1). Setting
# max_seq_length rewrites the tokenizer's truncation limit, so the
# config value is enforced rather than merely documented.
self._model.max_seq_length = max_length
(router/encoder.py:96-99.)
There is then a second, quieter guard: some tokenizers report a sentinel for "no limit"
(a very large integer). The class normalises that sentinel back to the verified ceiling so
tokenizer_max_length never reports a nonsense number:
tokenizer_limit = int(getattr(self._model.tokenizer, "model_max_length", 0))
if tokenizer_limit and tokenizer_limit > 10**6:
# Some tokenizers report a sentinel for "no limit".
tokenizer_limit = VERIFIED_TOKENIZER_MAX_LENGTH
self._tokenizer_max_length = tokenizer_limit
(router/encoder.py:101-105.)
Status. MEASURED β the probe is recorded with its date (2026-09-16) and its
sentence-transformers version (6.0.1) in the module docstring. The assertion is IMPLEMENTED
and its refusal path is covered by the routing test suite
(tests/routing/test_router.py, 82 tests, per docs/PHASE4_ROUTER_REPORT.md Β§Files).
4. The freeze β the whole architectural premise
The encoder is not fine-tuned. It is loaded, put in eval mode, and every parameter's gradient is switched off:
# Freeze. This is the whole architectural premise.
self._model.eval()
for param in self._model.parameters():
param.requires_grad_(False)
(router/encoder.py:91-94.)
The class docstring states what this buys:
Deliberately narrow: it encodes text and reports its dimensions. It does not train, does not expose the underlying model, and does not permit gradient flow. Everything downstream treats it as a pure function.
(router/encoder.py:40-43.)
Because the encoder is frozen, embeddings are a pure function of the query text. That single fact is what makes finding F4-2 possible (Β§8).
4.1 encode() β the inference surface
def encode(
self,
texts: Sequence[str],
batch_size: int = 64,
normalize: bool = True,
) -> np.ndarray:
"""Encode texts to a (n, embedding_dim) float32 array.
The returned array is detached: it carries no autograd history. That is
intentional β the adapter consumes it as a fixed input feature.
"""
if not texts:
return np.zeros((0, self.embedding_dim), dtype=np.float32)
cleaned = [t if isinstance(t, str) else str(t) for t in texts]
try:
embeddings = self._model.encode(
cleaned,
batch_size=batch_size,
show_progress_bar=False,
convert_to_numpy=True,
normalize_embeddings=normalize,
)
except Exception as exc: # noqa: BLE001
raise ModelLoadError(
f"encoder inference failed: {exc}", specialist="router"
) from exc
return np.asarray(embeddings, dtype=np.float32)
(router/encoder.py:127-155.) Four details are worth naming:
- Empty input returns an empty array of the right shape, not an error β
np.zeros((0, 384)). A caller batching over a filtered list does not need a special case. - Non-string inputs are coerced rather than rejected (
str(t)). The corpus isRouterExampleobjects, andRouterCorpus.texts()already yields strings; the coercion is a boundary guard. normalize_embeddings=normalizeβ the default isTrue, so the vectors are unit-length. Nothing downstream re-normalises.show_progress_bar=Falseis pinned, so training output is not polluted by a progress bar on every cache miss.
encode_one is the single-query convenience wrapper:
def encode_one(self, text: str, normalize: bool = True) -> np.ndarray:
return self.encode([text], batch_size=1, normalize=normalize)[0]
(router/encoder.py:157-158.) This is what IntentRouter._predict_learned calls.
4.2 The four read-only properties
| Property | Returns | Backing |
|---|---|---|
embedding_dim |
int |
self._model.get_sentence_embedding_dimension() β asked of the model, not read from config |
tokenizer_max_length |
int |
the normalised sentinel-aware value (Β§3) |
load_seconds |
float | None |
wall time of the SentenceTransformer(...) construction |
num_parameters |
int |
sum(p.numel() for p in self._model.parameters()) |
The embedding_dim property deliberately queries the model rather than trusting
router.embedding_dim from the config. IntentRouter.from_config then compares the two:
encoder = build_encoder(config, device=device)
if adapter is not None and adapter.input_dim != encoder.embedding_dim:
raise ModelLoadError(
f"adapter input_dim={adapter.input_dim} but the loaded encoder "
f"produces {encoder.embedding_dim}-d embeddings",
specialist="router",
)
(router/classifier.py:179-185.) A config that says 384 while the loaded encoder emits something
else fails loudly at construction rather than producing garbage at inference.
4.3 build_encoder() β config-driven construction with a device fallback
def build_encoder(config, device: str | None = None) -> FrozenEncoder:
"""Construct the encoder from the central configuration registry."""
if device is None:
configured = config.get("router.device", "auto")
if configured == "auto":
device = config.device_preference
else:
device = configured
return FrozenEncoder(
model_name=config.require("router.model"),
revision=config.require("router.revision"),
max_length=int(config.get("router.max_length", 128)),
device=device,
)
(router/encoder.py:168-182.)
Three points:
modelandrevisionuseconfig.require, notconfig.get. A missing model name or revision is a hard failure β the pin is not optional.max_lengthdefaults to 128 if the key is absent, so the encoder degrades to the configured behaviour rather than to the ceiling.device: autoresolves throughconfig.device_preference. This is how a CPU-only Space and a GPU host share one config file.configs/base.yamlsetsrouter.device: auto.
5. Finding F4-2 β the router needs no GPU
The finding, stated by the training module itself:
The encoder is FROZEN. Embeddings are therefore a pure function of the query text. So we embed the whole corpus ONCE (measured: 0.118 s / 64 queries on CPU), cache the vectors to disk, and train the 50,822-parameter adapter on the cached matrix (measured: 20 epochs / 4,096 vectors in 0.28 s).
Router training needs NO GPU. The plan's Kaggle budget ("Router | CPU/T4 | <1 h") is roughly three orders of magnitude pessimistic. Phase 4 runs and completes locally.
(router/train.py:3-12.)
docs/PHASE4_ROUTER_REPORT.md Β§F4-2 repeats it with the same numbers and adds the outcome:
"The plan's budget β 'Router | CPU/T4 | <1 h' β was roughly three orders of magnitude
pessimistic. Phase 4 ran to completion locally at zero quota cost."
Measured evidence for the claim, and its limits.
| Quantity | Value | Source | Status |
|---|---|---|---|
| Encoder throughput | 0.118 s / 64 queries, CPU | probe, router/encoder.py:14 |
MEASURED |
| Adapter training | 20 epochs / 4,096 vectors in 0.28 s | probe, router/encoder.py:22-23 |
MEASURED |
| Full 60-epoch run, 576-example corpus | 4.92 s total | metadata.json.duration_seconds: 4.92 |
MEASURED |
| GPU used | none | threshold_sweep_val.json.environment.cuda_available: false |
MEASURED |
The 0.28 s / 20 epochs figure is the probe, on a synthetic 4,096 Γ 384 matrix. The real run is the 4.92 s in the shipped artifact β a different and larger quantity (60 epochs, cache fingerprint computation, three split evaluations, artifact write). Both are true; they are not the same measurement, and this document does not merge them.
Caveat, stated because the docstring itself states it. The router/train.py docstring says
"the 50,822-parameter adapter". The shipped artifact records 51,725. Β§8 resolves this by
arithmetic; it is a documentation discrepancy in a comment, not a second model.
Part B β The five-head adapter
6. IntentAdapter β the only trainable part of the router
router/adapter.py opens with the architecture as an ASCII diagram, which is the clearest
statement of the shape anywhere in the repository:
embedding (384)
|
LayerNorm
|
Linear(384 -> hidden_dim) default hidden_dim = 128
|
GELU
|
Dropout
|
+--> task_head Linear(hidden, 6)
+--> modality_head Linear(hidden, 4)
+--> temporal_head Linear(hidden, 1) logit
+--> spatial_head Linear(hidden, 1) logit
+--> language_head Linear(hidden, 1) logit
(router/adapter.py:5-20.)
Read that diagram carefully: there is exactly one shared trunk, and the five heads all read from it. The heads are independent of one another β no head consumes another head's output. That independence is what makes the coherence-repair logic in Β§21 necessary.
6.1 The constructor, in full
def __init__(
self,
input_dim: int = 384,
hidden_dim: int = 128,
dropout: float = 0.1,
num_tasks: int = NUM_TASKS,
num_modalities: int = NUM_MODALITIES,
) -> None:
super().__init__()
if input_dim < 1 or hidden_dim < 1:
raise ValueError(
f"input_dim and hidden_dim must be positive, got {input_dim}, {hidden_dim}"
)
if not 0.0 <= dropout < 1.0:
raise ValueError(f"dropout must be in [0, 1), got {dropout}")
self.input_dim = input_dim
self.hidden_dim = hidden_dim
self.dropout_p = dropout
self.num_tasks = num_tasks
self.num_modalities = num_modalities
self.input_norm = nn.LayerNorm(input_dim)
self.trunk = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.GELU(),
nn.Dropout(dropout),
)
self.task_head = nn.Linear(hidden_dim, num_tasks)
self.modality_head = nn.Linear(hidden_dim, num_modalities)
# One logit per binary head. P(yes) = sigmoid(logit).
self.temporal_head = nn.Linear(hidden_dim, 1)
self.spatial_head = nn.Linear(hidden_dim, 1)
self.language_head = nn.Linear(hidden_dim, 1)
self._init_weights()
(router/adapter.py:64-102.)
Every sub-module, exhaustively.
| Attribute | Type | Shape / config | Role |
|---|---|---|---|
input_norm |
nn.LayerNorm |
normalised shape (input_dim,) = (384,) |
normalises the frozen embedding before the trunk |
trunk |
nn.Sequential |
Linear(384β128) β GELU β Dropout(0.1) |
the single shared representation |
task_head |
nn.Linear |
(128 β 6) |
softmax over TASK_CLASSES |
modality_head |
nn.Linear |
(128 β 4) |
softmax over MODALITY_CLASSES |
temporal_head |
nn.Linear |
(128 β 1) |
one logit; P(yes) = sigmoid(logit) |
spatial_head |
nn.Linear |
(128 β 1) |
one logit |
language_head |
nn.Linear |
(128 β 1) |
one logit |
dropout_p is stored under that name rather than dropout so it does not shadow the
nn.Dropout module convention; it is what config_dict() serialises.
Two validation guards, both at construction. input_dim and hidden_dim must be positive;
dropout must be in [0, 1) β note the half-open interval, so dropout=1.0 (which would zero
the trunk entirely) is rejected.
6.2 AdapterOutput β raw logits, no activation
@dataclass
class AdapterOutput:
"""Raw logits from all five heads. No softmax applied here."""
task_logits: torch.Tensor # (B, NUM_TASKS)
modality_logits: torch.Tensor # (B, NUM_MODALITIES)
temporal_logit: torch.Tensor # (B,)
spatial_logit: torch.Tensor # (B,)
language_logit: torch.Tensor # (B,)
def binary_logit(self, head: str) -> torch.Tensor:
if head == "temporal":
return self.temporal_logit
if head == "spatial_output":
return self.spatial_logit
if head == "language_output":
return self.language_logit
raise KeyError(f"unknown binary head: {head!r}")
(router/adapter.py:41-58.)
Three design facts:
- The two multi-class heads return
(B, C)logits; the three binary heads return(B,). Thesqueeze(-1)inforwardis what removes the trailing unit dimension. This matters for loss wiring:BCEWithLogitsLosswants(B,)targets, andnn.CrossEntropyLosswants(B, C)logits with(B,)integer targets. - No activation is applied.
softmaxandsigmoidare applied by the consumer β the loss during training, and_predict_learnedat inference. This keeps the numerically-stable*_with_logitslosses usable. binary_logit(head)keys on the label-space names β"temporal","spatial_output","language_output"β which are exactly the members ofBINARY_HEADS(router/label_space.py:63).evaluate_splitand the training loop both iteratefor head in BINARY_HEADSand call this method, so the attribute namesspatial_logitand the key"spatial_output"are reconciled in one place instead of at every call site.
6.3 Weight initialisation β and why it is not the default
def _init_weights(self) -> None:
"""Small-std init on the heads keeps the initial sigmoid near 0.5.
Without this the binary heads can start saturated, and BCE gradients
vanish before the task head has learned anything useful.
"""
for module in (self.task_head, self.modality_head,
self.temporal_head, self.spatial_head, self.language_head):
nn.init.normal_(module.weight, std=0.02)
nn.init.zeros_(module.bias)
(router/adapter.py:104-113.)
This is a deliberately non-default initialisation, and the reason is a real failure mode: a single-logit binary head whose weights start large can produce a saturated sigmoid on the first forward pass, and a saturated sigmoid has a vanishing BCE gradient. The task head would then be learning against a trunk that is receiving almost no signal from half its objectives.
Note precisely what is re-initialised: the five heads only. input_norm and trunk keep
PyTorch's defaults. The init is applied to weight (normal, Ο = 0.02) and bias (zeros).
Cross-reference. router/adapter.py:22-23 records the probe cost as "50,822 parameters".
The arithmetic in Β§8 gives 51,725. The std=0.02 figure in this chapter is read from
router/adapter.py:112; it is not the same as the 20.0 objectness weight in the grounding
head (specialists/grounding/head.py), which is a different subsystem and a different quantity.
6.4 forward() β and its two shape guards
def forward(self, embeddings: torch.Tensor) -> AdapterOutput:
"""Args:
embeddings: (B, input_dim) float tensor. Must already be detached
from the frozen encoder β the adapter does not back-propagate
into MiniLM.
"""
if embeddings.dim() != 2:
raise ValueError(
f"expected a 2-D (batch, dim) tensor, got shape {tuple(embeddings.shape)}"
)
if embeddings.shape[1] != self.input_dim:
raise ValueError(
f"expected input_dim={self.input_dim}, got {embeddings.shape[1]}"
)
hidden = self.trunk(self.input_norm(embeddings))
return AdapterOutput(
task_logits=self.task_head(hidden),
modality_logits=self.modality_head(hidden),
temporal_logit=self.temporal_head(hidden).squeeze(-1),
spatial_logit=self.spatial_head(hidden).squeeze(-1),
language_logit=self.language_head(hidden).squeeze(-1),
)
(router/adapter.py:117-140.)
The two guards are not decoration. A 1-D tensor would silently broadcast through the
Linear layers and produce a plausible-looking but wrong output; a mismatched feature width
would produce a RuntimeError deep inside nn.Linear with a shape trace that does not name the
cause. Both are caught here with a message that names the actual and expected dims.
The order of operations is fixed: normalise, then trunk. self.trunk(self.input_norm(...)) β
LayerNorm is applied to the frozen embedding, then the linear projection. Reversing them would
normalise the hidden layer instead, which is a different model.
6.5 Serialisation β the adapter carries its own architecture
def config_dict(self) -> dict[str, Any]:
return {
"input_dim": self.input_dim,
"hidden_dim": self.hidden_dim,
"dropout": self.dropout_p,
"num_tasks": self.num_tasks,
"num_modalities": self.num_modalities,
"binary_heads": list(BINARY_HEADS),
}
@classmethod
def from_config_dict(cls, payload: dict[str, Any]) -> "IntentAdapter":
return cls(
input_dim=int(payload["input_dim"]),
hidden_dim=int(payload["hidden_dim"]),
dropout=float(payload["dropout"]),
num_tasks=int(payload["num_tasks"]),
num_modalities=int(payload["num_modalities"]),
)
def num_parameters(self) -> int:
return sum(p.numel() for p in self.parameters())
(router/adapter.py:144-165.)
config_dict() is written into the checkpoint by save_adapter, and from_config_dict() is
what makes load_adapter able to reconstruct the architecture without being told it. The
checkpoint format is:
torch.save(
{
"state_dict": adapter.state_dict(),
"config": adapter.config_dict(),
},
path / ADAPTER_WEIGHTS,
)
(router/classifier.py:362-368.) load_adapter then refuses a checkpoint with no embedded
config, rather than guessing:
config = payload.get("config")
if not config:
raise ModelLoadError(
"adapter checkpoint has no embedded config; it cannot be reconstructed "
"without guessing the architecture",
specialist="router",
)
(router/classifier.py:438-444.) And it loads with strict=True, so a state dict that does not
match the reconstructed architecture is an error, not a partial load:
adapter = IntentAdapter.from_config_dict(config)
try:
adapter.load_state_dict(payload["state_dict"], strict=True)
except Exception as exc: # noqa: BLE001
raise ModelLoadError(
f"adapter state_dict does not match the reconstructed architecture: {exc}",
specialist="router",
) from exc
(router/classifier.py:446-453.)
The binary_heads key asymmetry. config_dict() emits "binary_heads", but
from_config_dict() does not read it. That is intentional and correct: the binary heads are
three fixed Linear(128, 1) layers, so their names are not a reconstructable parameter β they are
a label-space invariant. The key is written for auditability (a reader of metadata.json can
see which binary heads the artifact was built for) rather than for reconstruction. The shipped
metadata.json records it twice, under both adapter and adapter_config:
"binary_heads": ["temporal", "spatial_output", "language_output"],
"dropout": 0.1,
"hidden_dim": 128,
"input_dim": 384,
"num_modalities": 4,
"num_parameters": 51725,
"num_tasks": 6
7. The label space, exhaustively
router/label_space.py is the single source of truth for the router's output ontology. Its
docstring states why it is a module rather than a set of constants duplicated per file:
Single source of truth for the router's output ontology. Both the dataset generator and the training script import from here, so a class added in one place cannot silently desynchronise the other.
(router/label_space.py:1-6.)
7.1 The task head β six classes, in index order
TASK_CLASSES: tuple[str, ...] = (
"vqa",
"caption",
"grounding",
"change",
"optical_sar",
"unsupported",
)
TASK_TO_INDEX: dict[str, int] = {name: i for i, name in enumerate(TASK_CLASSES)}
NUM_TASKS: int = len(TASK_CLASSES)
(router/label_space.py:26-35.)
The tuple order is the index order. TASK_TO_INDEX is derived from the tuple by enumerate,
and train.py uses TASK_TO_INDEX[e.task] to build targets while evaluate_split uses
TASK_CLASSES[i] to name predictions. Reordering the tuple would silently re-map every label β
which is why the tuple is defined once and everything else is derived.
| Index | Label | Means | Configured in base.yaml? |
|---|---|---|---|
| 0 | vqa |
a free-form question about a single scene | yes, router.tasks[0] |
| 1 | caption |
a description of a single scene | yes, router.tasks[1] |
| 2 | grounding |
where β a spatial output | yes, router.tasks[2] |
| 3 | change |
what changed between two acquisitions | yes, router.tasks[3] |
| 4 | optical_sar |
joint classification from an optical + SAR pair | yes, router.tasks[4] |
| 5 | unsupported |
not a satellite-imagery question | yes, router.tasks[5] |
The config repeats the same six in the same order under router.tasks
(configs/base.yaml Β§router.tasks), so a reader of the config sees the ontology without opening
Python.
7.2 The three task families
#: Tasks that inherently require two acquisitions.
TEMPORAL_TASKS: frozenset[str] = frozenset({"change"})
#: Tasks that inherently require two modalities.
DUAL_MODALITY_TASKS: frozenset[str] = frozenset({"optical_sar"})
#: Tasks whose primary output is spatial rather than textual.
SPATIAL_TASKS: frozenset[str] = frozenset({"grounding"})
(router/label_space.py:37-44.)
Each is a frozenset with exactly one member today. They are sets rather than scalars so that
adding a second member does not require touching the consumers, and they are read by
default_attributes below.
7.3 The modality head β four classes
MODALITY_CLASSES: tuple[str, ...] = (
"optical",
"sar",
"optical_sar",
"unknown",
)
MODALITY_TO_INDEX: dict[str, int] = {name: i for i, name in enumerate(MODALITY_CLASSES)}
NUM_MODALITIES: int = len(MODALITY_CLASSES)
(router/label_space.py:50-57.) Four classes, and unknown is a first-class member rather than a
sentinel β a query with no modality marker is genuinely unknown, not "optical by default".
7.4 The binary heads β three names
BINARY_HEADS: tuple[str, ...] = ("temporal", "spatial_output", "language_output")
NUM_BINARY_HEADS: int = len(BINARY_HEADS)
(router/label_space.py:63-64.) The docstring explains the single-logit choice:
taskandmodalityuse softmax cross-entropy. The three binary heads use a single logit withBCEWithLogitsLossβ a 2-way softmax would waste a parameter and make the loss harder to weight.
(router/label_space.py:15-17.) README.md:478-479 repeats the reasoning.
The names are not the same as the head attributes. BINARY_HEADS uses spatial_output and
language_output; the module attributes are spatial_head and language_head. AdapterOutput. binary_logit() (Β§6.2) is the one place that reconciles them.
7.5 The two helper functions
def default_attributes(task: str) -> dict[str, bool]:
"""Per-task attribute defaults, used to sanity-check generated examples.
These are *defaults*, not invariants: `change` with `spatial_output=False`
is a perfectly legal request ("what changed?"). They exist so the dataset
generator and the fallback agree on what a task usually implies.
"""
return {
"temporal": task in TEMPORAL_TASKS,
"spatial_output": task in SPATIAL_TASKS,
"language_output": task != "unsupported",
}
def default_modality(task: str) -> str:
if task in DUAL_MODALITY_TASKS:
return "optical_sar"
if task == "unsupported":
return "unknown"
return "unknown"
(router/label_space.py:89-108.)
Note default_modality's third branch is redundant: task == "unsupported" already returns
"unknown", and the final return "unknown" covers it. The function is equivalent to
"optical_sar" if task in DUAL_MODALITY_TASKS else "unknown". This is a cosmetic redundancy, not a
defect β the two branches produce the same value β and it is recorded here only because a reader
diffing the branches will notice it.
Also note the docstring's own warning: these are defaults, not invariants. default_attributes
is not what RouterCorpus.validate() enforces; the validator enforces a stricter and different
set (Β§11.2).
7.6 The module's remaining surface
task_index(name), modality_index(name), is_valid_task(name) and is_valid_modality(name) are
thin lookups. task_index and modality_index raise KeyError on an unknown label, whereas
is_valid_* return a bool. router/fallback.py uses the is_valid_* pair inside self_check() to
assert that no rule ever emits a label outside the space.
8. Parameter accounting β 51,725, and why a comment says 50,822
The shipped artifact records adapter.num_parameters: 51725
(artifacts/router/router_adapter_v001/metadata.json), and docs/PHASE4_ROUTER_REPORT.md:37
agrees: "51,725 adapter params on a frozen 22,713,216-param encoder". Two other places say
50,822: the router/adapter.py module docstring (router/adapter.py:22) and the
router/train.py docstring (router/train.py:8), plus configs/base.yaml's comment on F4-2.
The arithmetic is unambiguous, and it settles the question:
| Sub-module | Parameter count | Working |
|---|---|---|
input_norm (LayerNorm(384)) |
768 | 384 Γ 2 (weight + bias) |
trunk[0] (Linear(384 β 128)) |
49,280 | 384 Γ 128 weights + 128 bias |
task_head (Linear(128 β 6)) |
774 | 128 Γ 6 weights + 6 bias |
modality_head (Linear(128 β 4)) |
516 | 128 Γ 4 weights + 4 bias |
temporal_head (Linear(128 β 1)) |
129 | 128 weights + 1 bias |
spatial_head (Linear(128 β 1)) |
129 | 128 weights + 1 bias |
language_head (Linear(128 β 1)) |
129 | 128 weights + 1 bias |
| Total | 51,725 | 768 + 49,280 + 774 + 516 + 129 + 129 + 129 |
The nn.GELU() and nn.Dropout() layers in the trunk contribute zero parameters, which is why
the trunk's total (49,280) equals its Linear layer alone.
Status. The 51,725 figure is MEASURED β it is what num_parameters() returns for the shipped
weights, and it is what the artifact records. The 50,822 figure is a stale comment; it is
reported here rather than quietly corrected, because the docs are public and a reader who greps
the repository will find both. The README's F4-2 paragraph still carries 50,822
(README.md:508). Treating this as an OPEN documentation discrepancy is the honest
description: the code is right, three comments are stale, and no behaviour depends on either
number.
9. The loss β three objectives, weighted
The training loss composes three terms. The docstring names them:
task cross-entropy, class-weighted (classes are imbalanced)
modality cross-entropy
binary x3 BCE-with-logits, one logit per head
Weights live in config under `router.training`.
(router/train.py:22-28.)
The implementation, verbatim from the inner loop:
loss_task = ce(out.task_logits, train_tensors["task"][idx])
loss_mod = ce_mod(out.modality_logits, train_tensors["modality"][idx])
binary_targets = train_tensors["binary"][idx]
loss_bin = (
bce(out.temporal_logit, binary_targets[:, 0])
+ bce(out.spatial_logit, binary_targets[:, 1])
+ bce(out.language_logit, binary_targets[:, 2])
) / len(BINARY_HEADS)
loss = (
task_loss_weight * loss_task
+ modality_loss_weight * loss_mod
+ binary_loss_weight * loss_bin
)
(router/train.py:606-620.)
Three details that matter.
- The binary term is the mean over the three heads, not the sum. The division by
len(BINARY_HEADS)keeps the term on the same scale as the two cross-entropies, sobinary_loss_weightmeans the same thing regardless of how many binary heads exist. - The binary targets are read by column index, in
BINARY_HEADSorder.binary_targets[:, 0]istemporal,[:, 1]isspatial_output,[:, 2]islanguage_outputβ the order comes from_to_tensors, which builds the matrix by iteratingBINARY_HEADS(router/train.py:301-305). The two orders are the same tuple, so they cannot drift. - Only the task cross-entropy is class-weighted.
cecarriesweight=class_weights;ce_modis an unweightednn.CrossEntropyLoss()andbceis a plainBCEWithLogitsLoss()(router/train.py:582-584). The class imbalance that motivated weighting is a task-class imbalance; the modality and binary targets are not weighted.
Gradient clipping. Every step clips the global norm to 1.0:
optimizer.zero_grad(set_to_none=True)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
(router/train.py:622-625.) set_to_none=True frees the gradient buffers rather than zeroing
them, which is both faster and stricter β a parameter that receives no gradient ends with None,
not a zero tensor.
The configured weights (configs/base.yaml Β§router.training; metadata.json.hyperparameters):
| Key | Value | Effect |
|---|---|---|
task_loss_weight |
1.0 |
the primary objective, full weight |
modality_loss_weight |
0.3 |
auxiliary; the modality label is often unknown |
binary_loss_weight |
0.5 |
auxiliary; three heads, mean-reduced |
The ordering β task 1.0 > binary 0.5 > modality 0.3 β encodes the priority: the task label is what the controller keys off; the binary heads describe aspects of the request; the modality label is the least informative head for this corpus.
Part C β The corpus
10. RouterExample β one labelled query, with a leakage boundary
@dataclass
class RouterExample:
"""One labelled query.
`group` is the leakage boundary: all examples sharing a group must land in
the same split. `source` distinguishes hand-written from generated material
so the two can be reported separately.
"""
text: str
task: str
modality: str = "unknown"
temporal: bool = False
spatial_output: bool = False
language_output: bool = True
group: str = ""
source: str = "template"
def __post_init__(self) -> None:
if self.task not in TASK_TO_INDEX:
raise ValueError(f"unknown task label: {self.task!r}")
if self.modality not in MODALITY_TO_INDEX:
raise ValueError(f"unknown modality label: {self.modality!r}")
if not self.text.strip():
raise ValueError("example text must not be empty")
if not self.group:
raise ValueError(f"example {self.text!r} has no group tag")
(router/dataset.py:87-113.)
| Field | Type | Default | Meaning | Constraint |
|---|---|---|---|---|
text |
str |
β | the query as a user would type it | must be non-empty after strip() |
task |
str |
β | one of TASK_CLASSES |
validated against TASK_TO_INDEX |
modality |
str |
"unknown" |
one of MODALITY_CLASSES |
validated against MODALITY_TO_INDEX |
temporal |
bool |
False |
needs two acquisitions | β |
spatial_output |
bool |
False |
the answer must contain coordinates | β |
language_output |
bool |
True |
the answer must contain prose | β |
group |
str |
"" |
the leakage boundary | must be non-empty β __post_init__ refuses a blank group |
source |
str |
"template" |
"curated" or "template" |
not validated (a free string) |
The group requirement is the load-bearing one. A blank group would make split_by_group
unable to guarantee that a near-duplicate pair travels together, so it is a construction-time
error rather than a split-time surprise.
label_tuple() is the training-time projection of an example into tensors:
def label_tuple(self) -> tuple[int, int, tuple[float, ...]]:
"""(task_index, modality_index, (temporal, spatial, language)) as floats."""
binaries = tuple(
1.0 if getattr(self, head) else 0.0 for head in BINARY_HEADS
)
return TASK_TO_INDEX[self.task], MODALITY_TO_INDEX[self.modality], binaries
(router/dataset.py:115-120.) Note getattr(self, head) β the binary head names in
BINARY_HEADS (temporal, spatial_output, language_output) are exactly the dataclass field
names, so no mapping table is needed. to_dict() serialises all eight fields
(router/dataset.py:122-132).
11. RouterCorpus β the container and its four checks
@dataclass
class RouterCorpus:
"""A labelled, group-tagged corpus ready for splitting and caching."""
examples: list[RouterExample] = field(default_factory=list)
(router/dataset.py:389-393.)
| Method | Returns | What it does |
|---|---|---|
__len__ |
int |
number of examples |
__iter__ |
iterator | iterates examples |
texts() |
list[str] |
[e.text for e in self.examples] β the encoder's input |
groups() |
set[str] |
the distinct group ids |
task_counts() |
dict[str, int] |
initialised over all six TASK_CLASSES, so an absent class reports 0 rather than being missing |
source_counts() |
dict[str, int] |
counts by source (only keys that occur) |
positives(head) |
int |
sum(1 for e in self.examples if getattr(e, head)) |
dedupe() |
(RouterCorpus, list[str]) |
removes exact duplicate texts; returns conflicts |
validate() |
list[str] |
structural checks; empty list means clean |
task_counts() initialising every class to zero is deliberate. A missing key and a zero count
are different things, and a validator that iterates counts[name] would raise KeyError on an
absent class instead of reporting it. Because the dict is pre-seeded, validate() can report
"task class 'vqa' has no examples" instead of crashing.
11.1 dedupe() β and why a duplicate is a conflict, not a silent merge
def dedupe(self) -> tuple["RouterCorpus", list[str]]:
"""Remove exact duplicate texts, keeping the first occurrence.
A query appearing under two different labels is a labelling conflict and
is reported rather than silently resolved.
"""
seen: dict[str, str] = {}
kept: list[RouterExample] = []
conflicts: list[str] = []
for example in self.examples:
key = example.text.strip().lower()
if key in seen:
if seen[key] != example.task:
conflicts.append(
f"{example.text!r}: {seen[key]} vs {example.task}"
)
continue
seen[key] = example.task
kept.append(example)
return RouterCorpus(kept), conflicts
(router/dataset.py:422-443.)
Three behaviours to name:
The comparison key is
text.strip().lower()β case- and whitespace-insensitive. So"What changed?"and"what changed? "are the same example.The first occurrence wins. The corpus is assembled with
CURATEDfirst (router/dataset.py:525-526), so a hand-checked curated example always beats a generated template that happens to produce the same text.Same text, different task β a reported conflict.
train_routerturns any conflict into a hard failure:corpus, conflicts = corpus.dedupe() if conflicts: raise RoutingError( "router corpus contains labelling conflicts (same text, two tasks):\n - " + "\n - ".join(conflicts) )(
router/train.py:488-493.) This is why the hard-negative families are legal: they contain the same text under different groups only when the texts differ ("Describe the water body."vs"Show me the water body."). A genuine same-text-two-labels pair would abort training.
11.2 validate() β five structural checks
def validate(self) -> list[str]:
"""Structural sanity checks. Returns a list of problems (empty = good)."""
(router/dataset.py:445-446.) The checks, in order:
(a) The corpus is non-empty. An empty corpus returns immediately with a single problem.
(b) Groups may span tasks only if they are hard-negative families.
spanning = {
group: sorted({e.task for e in examples})
for group, examples in _group_index(self).items()
if len({e.task for e in examples}) > 1
}
for group, tasks in spanning.items():
if not group.startswith(HARD_NEGATIVE_PREFIX):
problems.append(
f"group {group!r} spans multiple tasks {tasks} but is not a "
f"hard-negative family (expected prefix "
f"{HARD_NEGATIVE_PREFIX!r}); either split it or rename it"
)
(router/dataset.py:465-476.) The comment above it is the clearest statement of the design:
A group is a LEAKAGE boundary, not a label constraint: every example in a group goes to the same split, whatever its label. That is exactly what the hard-negative families need β
hn_desc_vs_showcontains both "Describe the water body." (caption) and "Show me the water body." (grounding), and splitting those two apart would destroy the whole point of the pair.So a group MAY span tasks. What must never happen is two examples with the same TEXT carrying different labels; that is a labelling conflict and
dedupe()reports it.
(router/dataset.py:453-464.)
(c) Every task class is represented.
counts = self.task_counts()
for name in TASK_CLASSES:
if counts[name] == 0:
problems.append(f"task class {name!r} has no examples")
(router/dataset.py:478-482.)
(d) An unsupported example must never claim a capability.
for e in self.examples:
if e.task == "unsupported" and (e.temporal or e.spatial_output or e.language_output):
problems.append(
f"unsupported example {e.text!r} claims a capability"
)
(router/dataset.py:484-489.) This is the strict rule β and it is stricter than
default_attributes, which sets language_output: task != "unsupported" (also false for
unsupported). The two agree; validate() is the enforcement.
(e) Temporal / spatial / modality coherence.
for e in self.examples:
if e.task == "change" and not e.temporal:
problems.append(f"change example {e.text!r} is not marked temporal")
if e.task == "grounding" and not e.spatial_output:
problems.append(
f"grounding example {e.text!r} is not marked spatial_output"
)
if e.task == "optical_sar" and e.modality != "optical_sar":
problems.append(
f"optical_sar example {e.text!r} has modality {e.modality!r}"
)
(router/dataset.py:491-502.)
Note the asymmetry between (e) and default_attributes: a change example must be temporal,
and a grounding example must be spatial_output β but a change example may be
spatial_output=True or False (both appear in CURATED, Β§12). The validator enforces the
necessary attribute, not the full default.
12. CURATED β the 70 hand-written examples
The corpus docstring states the design of the two sources:
- A hand-written CURATED set. Short, natural, exactly the phrasings a user types. This is the part that has to be right; everything else is volume.
- Template-generated examples from a shared template table. Each (task, template) pair carries a GROUP id.
(router/dataset.py:3-9.)
CURATED is a tuple[RouterExample, ...] of 70 declared entries across six task families plus
three hard-negative families. Every entry is a RouterExample with all eight fields positional:
(text, task, modality, temporal, spatial_output, language_output, group, source).
12.1 The curated examples by family
Caption β 8 examples, group cur_cap, all source="curated":
| Text | Modality | temporal | spatial | language |
|---|---|---|---|---|
Describe this image. |
unknown |
F | F | T |
Describe this satellite image. |
optical |
F | F | T |
Give me a description of the scene. |
unknown |
F | F | T |
What do you see in this image? |
unknown |
F | F | T |
Write a caption for this image. |
unknown |
F | F | T |
Summarize what is visible here. |
unknown |
F | F | T |
Describe the land cover in this scene. |
optical |
F | F | T |
Tell me about this image in detail. |
unknown |
F | F | T |
VQA β 12 examples, group cur_vqa, all modality="optical":
What land cover is visible? Β· How many buildings are in this image? Β·
Is there water in this scene? Β· What is the dominant land cover? Β·
Are there any roads visible? Β· What type of vegetation is present? Β·
Is this an urban or rural area? Β· Does this image contain a river? Β·
How many vehicles can you count? Β· What is the weather like in this image? Β·
Is this image from an agricultural area? Β· What season does this image show?
All twelve: temporal=F, spatial_output=F, language_output=T.
Grounding β 10 examples, group cur_grd, all modality="optical", all spatial_output=True:
| Text |
|---|
Show me the water body. |
Locate the buildings. |
Where is the road? |
Highlight the vegetation. |
Find the bridge in this image. |
Point out the harbor. |
Draw a box around the river. |
Can you show me where the water body is? |
Mark the location of the airport. |
Where are the trees located? |
Change β 12 examples, group cur_chg, all modality="optical", all temporal=True. Four of
the twelve carry spatial_output=True:
| Text | spatial_output |
|---|---|
What changed between these images? |
F |
What changed? |
F |
Has the urban area increased? |
F |
Compare the before and after images. |
F |
Describe the changes. |
F |
Did any buildings appear? |
F |
Where did the change happen? |
T |
Show me the changed regions. |
T |
What changed and show me where? |
T |
Highlight the areas that changed. |
T |
Locate where new construction appeared. |
T |
How much forest was lost between the two dates? |
F |
This table is the single most useful object in the corpus, because it is the concrete evidence that
change does not imply spatial_output. The docstring for default_attributes says exactly
this: "change with spatial_output=False is a perfectly legal request ('what changed?')"
(router/label_space.py:92-94).
Optical-SAR β 8 examples, group cur_osr, all modality="optical_sar", all
temporal=F, spatial_output=F, language_output=T:
Compare the optical and radar images. Β· Compare optical and SAR to identify built-up regions. Β·
Use both the radar and optical images. Β· Jointly analyse the optical and SAR pair. Β·
What can the radar tell us that the optical cannot? Β· Fuse the optical and SAR data. Β·
Analyse the co-registered optical and radar pair. Β·
Show built-up areas using radar and optical together.
Note that Compare optical and SAR to identify built-up regions. and
Show built-up areas using radar and optical together. both contain built-up β and both are
labelled optical_sar, not change. These two curated examples are the corpus-side record of the
very bug the router later exhibited (Β§44βΒ§47): the word "built-up" is land-cover vocabulary, and the
modality pair is what makes the request a fusion request.
Unsupported β 6 examples, group cur_uns, all modality="unknown", all three booleans
False:
Book me a flight to Delhi. Β· What is the weather tomorrow? Β·
Write me a poem about satellites. Β· Who won the cricket match? Β·
Send an email to my supervisor. Β· What is the capital of France?
12.2 The three hard-negative families
The corpus docstring names the pairs that matter, and the reason they are first-class:
Hard negatives are first-class here, not an afterthought. The pairs that matter:
"describe the water body" -> caption (describe = language) "show me the water body" -> grounding (show = spatial) "what changed" -> change, spatial_output=False "where did the change happen"-> change, spatial_output=True "compare these two images" -> change (two images, temporal) "compare optical and radar" -> optical_sar (two modalities)Every one of those is a single-token difference in the input and a different label on the output. A router that has not seen such pairs will fail them.
(router/dataset.py:18-31.)
hn_desc_vs_show β 6 examples. The describe/show contrast, three subjects:
| Text | Task | spatial_output |
|---|---|---|
Describe the water body. |
caption |
F |
Show me the water body. |
grounding |
T |
Describe the road. |
caption |
F |
Show me the road. |
grounding |
T |
Describe the buildings. |
caption |
F |
Show me the buildings. |
grounding |
T |
hn_what_vs_where β 4 examples. The what/where contrast on the same task:
| Text | Task | spatial_output |
|---|---|---|
What changed? |
change |
F |
Where did the change happen? |
change |
T |
Describe the changes. |
change |
F |
Show me where the changes are. |
change |
T |
hn_temporal_vs_modality β 4 examples. The temporal/modality contrast β the hardest family,
because the surface forms are near-identical and the semantics differ:
| Text | Task | temporal | modality |
|---|---|---|---|
Compare these two images. |
change |
T | optical |
Compare optical and radar. |
optical_sar |
F | optical_sar |
What is different between these two dates? |
change |
T | optical |
What is different between the optical and SAR views? |
optical_sar |
F | optical_sar |
The HARD_NEGATIVE_PREFIX constant is what makes these three families special:
#: Groups whose name starts with this are hard-negative FAMILIES: sets of
#: deliberately confusable queries that must travel together through any split.
#: `hn_desc_vs_show` holds both "Describe the water body." (caption) and
#: "Show me the water body." (grounding) β splitting those apart would remove
#: the only signal that teaches the router the difference.
HARD_NEGATIVE_PREFIX = "hn_"
(router/dataset.py:53-58.)
The prefix does three jobs at once: validate() uses it to permit a group to span tasks;
split_by_group uses it to force the family into the test split; and evaluate_split uses it
to compute the hard_negative_accuracy metric. One string, three consumers, one definition.
12.3 Declared versus recorded counts
CURATED declares 70 entries. metadata.json.corpus.by_source records 66 curated and
510 template, totalling 576 across 54 groups. The difference is dedupe(), which runs
before the split (Β§11.1) and removes exact duplicate texts while keeping the first occurrence.
Four of the duplicates are within CURATED, and they are visible in the source: the same text
appears once in a task family and once in a hard-negative family, deliberately, so the family is
self-contained.
| Duplicated text | First occurrence (family) | Second occurrence (family) |
|---|---|---|
Show me the water body. |
cur_grd (grounding) |
hn_desc_vs_show (grounding) |
What changed? |
cur_chg (change) |
hn_what_vs_where (change) |
Where did the change happen? |
cur_chg (change) |
hn_what_vs_where (change) |
Describe the changes. |
cur_chg (change) |
hn_what_vs_where (change) |
A further set is cross-source: a curated example whose text a template reproduces exactly, such as
Show me the water body. (curated) versus template t_grd_a = "Show me the {}." with the subject
water body. Because CURATED is extended into the corpus first
(router/dataset.py:525-526), the curated entry survives and the generated duplicate is dropped.
Status. The recorded counts (66 / 510 / 576 / 54) are MEASURED β they are in the shipped
artifact. The mechanism of the reduction (exact-duplicate removal by dedupe()) is
IMPLEMENTED and verified by the source. The exact per-text enumeration of every dropped
duplicate is UNKNOWN β not established from the available evidence: reconstructing it requires
running build_corpus() + dedupe() and diffing, which this document did not do. The four
within-CURATED duplicates above are visible by inspection of the tuple and are therefore stated;
the cross-source ones are stated as a mechanism with one example, not as a complete list.
13. TEMPLATES and SUBJECTS β volume with a group tag
The template table is a tuple of 7-tuples:
#: (group_id, task, template, modality, temporal, spatial, language)
#: `{}` is replaced by a subject from the matching subject list.
TEMPLATES: tuple[tuple[str, str, str, str, bool, bool, bool], ...] = (
(router/dataset.py:233-235.)
TEMPLATES holds 45 entries. Expanded against SUBJECTS, each template yields one example per
subject, and every example from a given template shares that template's group id β which is the
whole point (Β§14).
13.1 Every template, by family
Caption β 8 templates (router/dataset.py:237-241, 302-304):
| Group | Template | modality | temporal | spatial | language |
|---|---|---|---|---|---|
t_cap_a |
Describe the {}. |
optical | F | F | T |
t_cap_b |
Give a detailed description of the {}. |
optical | F | F | T |
t_cap_c |
What does the {} look like? |
optical | F | F | T |
t_cap_d |
Write a caption describing the {}. |
optical | F | F | T |
t_cap_e |
Summarize the {} visible in this image. |
optical | F | F | T |
t_cap_f |
Provide an overview of the {}. |
optical | F | F | T |
t_cap_g |
Explain what is happening in the {}. |
optical | F | F | T |
t_cap_h |
Give me a short report on the {}. |
optical | F | F | T |
VQA β 5 templates (router/dataset.py:244-248):
| Group | Template |
|---|---|
t_vqa_a |
Is the {} present in this image? |
t_vqa_b |
How many {} are visible? |
t_vqa_c |
What kind of {} is shown here? |
t_vqa_d |
Can you tell if this contains {}? |
t_vqa_e |
Are there any {} in the scene? |
All five: optical, F, F, T.
Grounding β 8 templates (router/dataset.py:251-256, 307, 317):
| Group | Template |
|---|---|
t_grd_a |
Show me the {}. |
t_grd_b |
Locate the {}. |
t_grd_c |
Where is the {}? |
t_grd_d |
Highlight the {}. |
t_grd_e |
Draw a box around the {}. |
t_grd_f |
Point out the {} on the map. |
t_grd_g |
Give me the coordinates of the {}. |
t_grd_h |
Trace the outline of the {}. |
All eight: optical, F, T, T.
Change β 10 templates (router/dataset.py:259-264, 295-298):
| Group | Template | spatial |
|---|---|---|
t_chg_a |
What happened to the {} between the two images? |
F |
t_chg_b |
Has the {} changed? |
F |
t_chg_c |
Describe how the {} changed over time. |
F |
t_chg_d |
Where did the {} change? |
T |
t_chg_e |
Show the regions where the {} changed. |
T |
t_chg_f |
How much did the {} increase or decrease? |
F |
t_chg_g |
Compare the two acquisitions of the {}. |
F |
t_chg_h |
What is different about the {} between the two dates? |
F |
t_chg_i |
Did the {} expand or shrink? |
F |
t_chg_j |
Point out where the {} was modified. |
T |
All ten: optical, temporal=T, language=T.
Optical-SAR β 4 templates (router/dataset.py:267-270):
| Group | Template |
|---|---|
t_osr_a |
Compare the optical and radar views of the {}. |
t_osr_b |
Use both sensors to identify the {}. |
t_osr_c |
How does the {} appear differently in SAR and optical? |
t_osr_d |
Analyse the {} using the co-registered optical and SAR pair. |
All four: optical_sar, F, F, T.
Unsupported β 10 templates (router/dataset.py:282-291):
| Group | Template |
|---|---|
t_uns_a |
Tell me a joke about {}. |
t_uns_b |
What is the population of {}? |
t_uns_c |
Recommend a restaurant near {}. |
t_uns_d |
Book me a flight to {}. |
t_uns_e |
Who won the match in {}? |
t_uns_f |
Translate this sentence into {}. |
t_uns_g |
Write a poem about {}. |
t_uns_h |
What is the weather in {} tomorrow? |
t_uns_i |
Send an email about {}. |
t_uns_j |
Summarize this news article about {}. |
All ten: unknown, F, F, F.
13.2 The unsupported template-count comment is a defect record
The comment above the unsupported block is one of the most valuable passages in the router
source, because it explains a measured failure and the fix that followed:
# Volume matters here more than anywhere else. Group-level splitting keeps
# every template on one side of the boundary, so a class with only four
# templates ends up with two of them available for training β and a template
# the model never saw cannot be learned, only guessed at. With four templates
# the "Recommend a restaurant near X" phrasing fell entirely into test and
# the router, having never seen it, read "near <scene noun>" as a grounding
# request. That is correct behaviour on insufficient data, so the fix is more
# distinct phrasings, not a looser gate.
(router/dataset.py:272-281.)
The generalisable rule is stated in the same block:
Template COUNT is what buys group-level trainability; subject count only buys volume.
(router/dataset.py:354-355.) This is the second half of the unsupported defect in
docs/PHASE4_ROUTER_REPORT.md Β§2 β the same edit pass that grew unsupported from 4 to 10
templates also grew its subject list to 20, taking the class to 30.4 % of the corpus and
forcing its inverse-frequency class weight down to 0.548, which destabilised the surrounding
classes. The fix was to hold at 10 templates Γ 10 subjects β 14 %.
13.3 SUBJECTS β ten to fifteen nouns per family
SUBJECTS: dict[str, tuple[str, ...]] = {
"caption": (
"scene", "image", "landscape", "area", "region",
"terrain", "land cover", "cityscape", "coastline", "farmland",
),
"vqa": (
"water body", "road", "building", "forest", "river",
"bridge", "harbor", "vehicle", "tree", "field",
"airport", "railway", "dam", "lake", "industrial area",
),
"grounding": (
"water body", "building", "road", "river", "forest",
"bridge", "harbor", "lake", "airport", "dam",
"railway line", "parking lot", "swimming pool", "stadium", "quarry",
),
"change": (
"urban area", "forest", "water body", "built-up area", "vegetation",
"farmland", "road network", "construction site", "coastline", "wetland",
),
"optical_sar": (
"built-up area", "water body", "forest", "agricultural field", "urban region",
"flooded area", "road network", "industrial zone", "coastline", "wetland",
),
"unsupported": (
"the moon", "the ocean", "galaxies", "the ISS", "satellites",
"Delhi", "Mumbai", "cricket", "French", "the Himalayas",
),
}
(router/dataset.py:320-360.)
| Family | Subject count | Template count | Instances (count Γ count) |
|---|---|---|---|
caption |
10 | 8 | 80 |
vqa |
15 | 5 | 75 |
grounding |
15 | 8 | 120 |
change |
10 | 10 | 100 |
optical_sar |
10 | 4 | 40 |
unsupported |
10 | 10 | 100 |
| Total | β | 45 | 515 |
The unsupported subject list is the one with a stated design, and it mixes two kinds of noun:
- satellite-imagery nouns used in a NON-imagery frame (the moon, the ocean, the ISS) β the valuable half. "Recommend a restaurant near the ocean" contains a scene noun but is not an imagery request, so it teaches that the TEMPLATE dominates the subject.
- unambiguous out-of-domain entities (Delhi, cricket, French)
(router/dataset.py:344-351.)
That first kind is a deliberate hard negative inside the unsupported class: it isolates the template as the label-bearing feature, so the router cannot learn "contains a scene noun β imagery request".
The arithmetic gap. 70 declared curated + 515 template instances = 585, but the recorded
corpus is 576 (66 + 510). The 9-item reduction is dedupe() (Β§12.3). The recorded
per-family by_task totals are:
| Task | Recorded count | Declared total before dedupe |
|---|---|---|
caption |
91 | 8 + 80 = 88 |
change |
115 | 12 + 100 = 112 |
grounding |
128 | 10 + 120 = 130 |
optical_sar |
50 | 8 + 40 = 48 |
unsupported |
105 | 6 + 100 = 106 |
vqa |
87 | 12 + 75 = 87 |
| Total | 576 | 571 |
Note that the recorded totals exceed the declared totals for four families β which is the
opposite of what pure deduplication would produce. This means the declared TEMPLATES/SUBJECTS
tables as read here do not fully account for the shipped corpus: the shipped
metadata.json was produced on 2026-09-16, and the corpus source has clearly been edited since
(the unsupported comment records an expansion from 4 to 10 templates, and caption gained
t_cap_f/g/h). The per-family provenance of the shipped 576-example corpus is therefore
UNKNOWN β not established from the available evidence. What is established: the shipped
artifact's own counts (66 curated, 510 template, 576 total, 54 groups, and the six by_task
counts), the current declared table (70 curated, 45 templates), and the fact that train_router
runs dedupe() before splitting. A reader who needs the exact shipped corpus should regenerate it
with build_corpus(seed=42) against the 2026-09-16 revision of router/dataset.py.
This is stated rather than smoothed over deliberately. The style guide's rule 6 is that an honest gap beats a confident invention, and the tempting move here β asserting "dedupe removed 9 duplicates" β is contradicted by the arithmetic.
13.4 _generate_templates() β one group per template
def _generate_templates() -> list[RouterExample]:
"""Expand the template table into examples, one group per template."""
out: list[RouterExample] = []
for group, task, template, modality, temporal, spatial, language in TEMPLATES:
subjects = SUBJECTS.get(task, SUBJECTS["caption"])
for subject in subjects:
out.append(
RouterExample(
text=template.format(subject),
task=task,
modality=modality,
temporal=temporal,
spatial_output=spatial,
language_output=language,
group=group,
source="template",
)
)
return out
(router/dataset.py:363-381.)
Two details: the group is the template id, shared by every subject expansion (this is what
makes the group a leakage boundary at the template level, not the example level); and the
subject list falls back to SUBJECTS["caption"] for an unknown task, so a new task added to
TEMPLATES without a subject list still generates rather than crashing.
13.5 build_corpus() β assembly and the ablation switches
def build_corpus(
n_template_repeats: int = 1,
seed: int = 42,
include_curated: bool = True,
include_templates: bool = True,
) -> RouterCorpus:
"""Assemble the full router corpus.
Args:
n_template_repeats: duplicate the template block N times with shuffled
subject ordering. Volume without new phrasings; keep at 1 unless the
adapter is clearly underfitting.
seed: controls subject shuffling.
include_curated / include_templates: ablation switches.
"""
rng = random.Random(seed)
examples: list[RouterExample] = []
if include_curated:
examples.extend(CURATED)
if include_templates:
for _ in range(max(1, n_template_repeats)):
block = _generate_templates()
rng.shuffle(block)
examples.extend(block)
return RouterCorpus(examples)
(router/dataset.py:507-534.)
Three things to note. CURATED is extended first, which is what makes the curated entry
win a dedupe collision. The shuffle is applied to the template block only, so curated order is
deterministic. And n_template_repeats repeats the same phrasings with shuffled subject order
β the docstring is explicit that this buys volume, not new phrasings: "Volume without new
phrasings; keep at 1 unless the adapter is clearly underfitting." The shipped run used the default
n_template_repeats=1.
14. Finding F4-3 β splits must be by group, never by example
The dataset docstring states the finding with its worked example:
The group tag is the important part (finding F4-3).
"show me the water body"and"show me the road"come from the same template and differ by one token. Splitting them across train/val makes validation trivially easy and gives a fake accuracy number. Splitting by group keeps every template on exactly one side β the router-level analogue of scene-level splitting, and the same class of bug Gate 1 exists to catch.
(router/dataset.py:11-16.)
router/train.py restates it and adds the enforcement:
Groups are template ids and curated families. Splitting by example would put "show me the water body" in train and "show me the road" in val β same template, one token apart β and report a fake accuracy.
split_by_groupmirrorsevaluation.leakage.assign_splits_by_scenedeliberately, and the train function REFUSES to proceed if the split report is not clean.
(router/train.py:14-20.)
The refusal is real code:
report = split_leakage_report(split)
if not report["clean"]:
raise RoutingError(
"router split failed the group-leakage audit; refusing to train: "
f"{report['groups_across_splits']}"
)
(router/train.py:502-507.)
14.1 split_by_group() β the signature and the two guarantees
def split_by_group(
corpus: RouterCorpus,
train_ratio: float = 0.75,
val_ratio: float = 0.15,
seed: int = 42,
hard_negatives_to_test: bool = True,
) -> dict[str, list[RouterExample]]:
"""Split by GROUP, never by example (finding F4-3).
Mirrors `evaluation.leakage.assign_splits_by_scene` deliberately: the failure
mode is identical, so the guard should look identical too. Groups are sorted
before shuffling so the result does not depend on corpus order.
Two guarantees beyond a naive ratio split:
1. **Every task class reaches train.** A random group split can strand a
class entirely in val/test, and then its head can never learn it. Groups
are walked in shuffled order and any group that covers a not-yet-covered
task is pulled into train first, before the ratio is topped up.
2. **Hard-negative families are held out.** They are the cases the router
most needs measured, so by default they go to test rather than training
on the exact pairs being scored.
"""
(router/dataset.py:537-566.)
Input validation, before any work:
if not 0.0 < train_ratio < 1.0:
raise ValueError(f"train_ratio must be in (0,1), got {train_ratio}")
if not 0.0 <= val_ratio < 1.0:
raise ValueError(f"val_ratio must be in [0,1), got {val_ratio}")
if train_ratio + val_ratio >= 1.0:
raise ValueError(
f"train_ratio + val_ratio must be < 1, got {train_ratio} + {val_ratio}"
)
(router/dataset.py:567-574.) Note the asymmetry: train_ratio is strictly interior
(0, 1) β a 100 %-train split is meaningless β while val_ratio may be 0.
Forced test groups:
groups = sorted(by_group)
forced_test = {
g for g in groups
if hard_negatives_to_test and g.startswith(HARD_NEGATIVE_PREFIX)
}
pool = [g for g in groups if g not in forced_test]
if not pool:
raise ValueError(
"every group is a hard-negative family; there is nothing to train on"
)
(router/dataset.py:580-591.)
Stratification by dominant task β and the comment that records the defect:
# Stratify by dominant task ----------------------------------------
# Group-level splitting is what prevents leakage. It is NOT sufficient on
# its own: a naive group split can strand an entire task class in one
# partition. That is not hypothetical β with 38 groups and a flat 75/15
# allocation, `vqa` (the most important mandatory task) and `unsupported`
# both ended up with ZERO test examples, so the acceptance gate could not
# measure them and their recall printed as a misleading 0.000.
#
# Stratifying by task fixes that WITHOUT weakening leakage safety: a group
# still travels to exactly one split, whatever its labels. Only the choice
# of which split changes.
test_ratio = 1.0 - train_ratio - val_ratio
strata: dict[str, list[str]] = {}
for group in pool:
strata.setdefault(_dominant_task(by_group[group]), []).append(group)
(router/dataset.py:593-608.)
_dominant_task is the stratum chooser, and its tie-break is deterministic:
def _dominant_task(examples: list["RouterExample"]) -> str:
"""Most common task in a group, ties broken deterministically.
Used only to choose which stratum a group belongs to when balancing splits.
Hard-negative families deliberately span tasks, so they need a stratum too;
the alphabetically-first task is as good a tie-break as any.
"""
counts: dict[str, int] = {}
for e in examples:
counts[e.task] = counts.get(e.task, 0) + 1
return sorted(counts.items(), key=lambda kv: (-kv[1], kv[0]))[0][0]
(router/dataset.py:74-84.) The sort key (-count, name) means: highest count first, and on a tie,
alphabetically-first name. Deterministic and reproducible.
The per-stratum allocation, with its three degenerate cases:
for task in sorted(strata):
group_list = list(strata[task])
rng.shuffle(group_list)
n = len(group_list)
if n == 1:
# Cannot be spread without losing the class from training. Training
# must win: an unlearnable class is worse than an unmeasured one.
assignment[group_list[0]] = "train"
continue
if n == 2:
# One to train (learnable), one to test (measurable). Validation
# coverage for this task comes from the corpus at large.
assignment[group_list[0]] = "train"
assignment[group_list[1]] = "test"
continue
n_test = max(1, round(n * test_ratio))
n_val = max(1, round(n * val_ratio))
# Keep at least one group for training, whatever the ratios ask for.
while n_test + n_val >= n:
if n_test > 1:
n_test -= 1
elif n_val > 1:
n_val -= 1
else:
break
n_train = n - n_test - n_val
for i, group in enumerate(group_list):
if i < n_train:
assignment[group] = "train"
elif i < n_train + n_val:
assignment[group] = "val"
else:
assignment[group] = "test"
(router/dataset.py:613-649.)
The n == 1 rule is the explicit priority statement: an unlearnable class is worse than an
unmeasured one. A single-group stratum goes to train, so Gate 2's "every class measured"
condition can still fail β which is the correct failure, because an unmeasured class is at least
visible in unmeasured_test_tasks (Β§18.1).
The shuffle is per stratum, not global, so adding a group to one task family does not perturb the assignment of another.
The output is emitted in sorted group order, not shuffle order:
# Emit in sorted group order. The seed chooses *which* group goes where; it
# must not control the order of the returned lists, or downstream code
# taking records[:N] would silently depend on the seed.
out: dict[str, list[RouterExample]] = {"train": [], "val": [], "test": []}
for group in groups:
out[assignment[group]].extend(by_group[group])
return out
(router/dataset.py:660-666.) This is a subtle but important guarantee: the seed chooses the
partition, but not the ordering within it. A caller who slices split["train"][:100] gets the
same 100 examples regardless of seed.
14.2 split_leakage_report() β the audit
def split_leakage_report(split: dict[str, list[RouterExample]]) -> dict[str, object]:
"""Verify no group crosses a split boundary. Returns a report dict."""
group_splits: dict[str, set[str]] = {}
for name, examples in split.items():
for e in examples:
group_splits.setdefault(e.group, set()).add(name)
offenders = {g: sorted(s) for g, s in group_splits.items() if len(s) > 1}
return {
"clean": not offenders,
"groups_across_splits": offenders,
"sizes": {name: len(examples) for name, examples in split.items()},
"task_counts": {
name: _count_tasks(examples) for name, examples in split.items()
},
"unique_groups": {name: len({e.group for e in examples})
for name, examples in split.items()},
}
(router/dataset.py:669-687.)
The report has five keys, and the shipped artifact records all five
(metadata.json.split):
"clean": true,
"groups_across_splits": {},
"sizes": { "test": 80, "train": 410, "val": 86 },
"task_counts": {
"test": { "caption": 13, "change": 13, "grounding": 17, "optical_sar": 12, "unsupported": 10, "vqa": 15 },
"train": { "caption": 70, "change": 82, "grounding": 97, "optical_sar": 28, "unsupported": 76, "vqa": 57 },
"val": { "caption": 8, "change": 20, "grounding": 14, "optical_sar": 10, "unsupported": 19, "vqa": 15 }
},
"unique_groups": { "test": 9, "train": 37, "val": 8 }
Every one of the six task classes has non-zero test support. That is the stratification
working, and it is the precondition for gate2_passed (Β§18.2) being able to report anything at all.
| Split | Examples | Unique groups | Notes |
|---|---|---|---|
| train | 410 | 37 | 71.2 % of 576 |
| val | 86 | 8 | 14.9 % |
| test | 80 | 9 | 13.9 % β includes all three hn_* families |
| total | 576 | 54 | β |
The group counts (37 + 8 + 9 = 54) sum exactly to the corpus's 54 groups, which is the audit's
clean: true in arithmetic form: every group appears in exactly one split.
The test split's 9 groups include the 3 hard-negative families, so the 80 test examples
contain all 14 curated hn_* examples. That is why hard_negative_accuracy can be computed at all
(Β§16.4) and why it is the most honest generalisation number in the artifact.
Part D β Training
15. train_router() β the full signature
def train_router(
corpus: RouterCorpus | None = None,
encoder=None,
epochs: int = 60,
batch_size: int = 64,
learning_rate: float = 1e-3,
weight_decay: float = 0.01,
task_loss_weight: float = 1.0,
modality_loss_weight: float = 0.3,
binary_loss_weight: float = 0.5,
hidden_dim: int = 128,
dropout: float = 0.1,
seed: int = 42,
device: str = "cpu",
artifact_dir: str | Path | None = None,
cache_dir: str | Path | None = None,
config_hash: str | None = None,
n_template_repeats: int = 1,
val_ratio: float = 0.15,
hard_negatives_to_test: bool = True,
verbose: bool = True,
) -> TrainingResult:
(router/train.py:444-465.)
| Parameter | Default | Shipped value | Source of the shipped value |
|---|---|---|---|
corpus |
None β build_corpus(...) |
built from source | metadata.json.corpus |
encoder |
None β stub |
the frozen MiniLM encoder | metadata.json.encoder_type: "frozen_sentence_transformer" |
epochs |
60 | 60 | metadata.json.hyperparameters.epochs |
batch_size |
64 | 64 | metadata.json.hyperparameters.batch_size |
learning_rate |
1e-3 |
0.001 | metadata.json.hyperparameters.learning_rate |
weight_decay |
0.01 | 0.01 | metadata.json.hyperparameters.weight_decay |
task_loss_weight |
1.0 | 1.0 | metadata.json.hyperparameters.task_loss_weight |
modality_loss_weight |
0.3 | 0.3 | metadata.json.hyperparameters.modality_loss_weight |
binary_loss_weight |
0.5 | 0.5 | metadata.json.hyperparameters.binary_loss_weight |
hidden_dim |
128 | 128 | metadata.json.adapter.hidden_dim |
dropout |
0.1 | 0.1 | metadata.json.adapter.dropout |
seed |
42 | 42 | metadata.json.seed |
device |
"cpu" |
cpu |
threshold_sweep_val.json.environment.requested_device |
artifact_dir |
None β artifacts/router/router_adapter_v001 |
that path | metadata.json location |
cache_dir |
None β artifacts/router/cache |
that path | artifacts/router/cache/ exists |
config_hash |
None |
615478910dc266bf |
metadata.json.config_hash |
n_template_repeats |
1 | 1 (default) | metadata.json.corpus.total: 576 is consistent with 1 |
val_ratio |
0.15 | 0.15 | configs/base.yaml Β§router.training.val_ratio |
hard_negatives_to_test |
True |
true | configs/base.yaml Β§router.training.hard_negatives_to_test |
verbose |
True |
β | prints every 10 epochs |
Two config_hash values exist and they are different numbers for different objects. The
adapter artifact records config_hash: "615478910dc266bf", while the threshold sweep and the
release-wide frozen config hash is 78f1e3700da15aa1. threshold_sweep_val.json records both,
under different keys β adapter_config_hash: "615478910dc266bf" and
config_hash: "78f1e3700da15aa1". So the two are not in conflict: one hashes the adapter's
architecture config, the other the frozen system configuration. The style guide's
"Frozen config hash 78f1e3700da15aa1" refers to the latter.
15.1 The five-stage body, in order
started = time.time()
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
(router/train.py:473-476.) All three RNGs are seeded, not just torch's β because the shuffle
uses random.Random(seed) (Β§15.3), the numpy path is used by the stub embeddings, and torch seeds
the weight init.
Stage 1 β corpus validation and dedupe. validate() problems abort with a RoutingError
naming every problem; dedupe() conflicts abort with the conflict list (Β§11.1).
Stage 2 β split, with the leakage guard proven to have fired. split_by_group(...) then
split_leakage_report(...), and a non-clean report aborts. Then a second guard:
train_ex, val_ex, test_ex = split["train"], split["val"], split["test"]
if not train_ex or not val_ex or not test_ex:
raise RoutingError(
f"split produced an empty partition: train={len(train_ex)} "
f"val={len(val_ex)} test={len(test_ex)}"
)
(router/train.py:509-514.) The shipped split is 410 / 86 / 80 β none empty.
Stage 3 β embeddings. Either the cached frozen-encoder path or the stub path (Β§17).
Stage 4 β model, class weights, optimizer, losses, the epoch loop.
Stage 5 β final evaluation, metadata assembly, artifact write, return.
15.2 The embedding cache and its fingerprint
def _corpus_fingerprint(corpus: RouterCorpus, encoder_id: str, max_length: int) -> str:
payload = {
"version": CORPUS_CACHE_VERSION,
"encoder": encoder_id,
"max_length": max_length,
"texts": sorted(e.text for e in corpus),
}
blob = json.dumps(payload, sort_keys=True).encode()
return hashlib.sha256(blob).hexdigest()
(router/train.py:221-229.) CORPUS_CACHE_VERSION = "v1" (router/train.py:71).
The cache key includes the sorted text set, not just a count β so changing one query
invalidates the cache. embed_corpus_cached explains why this matters:
The cache key includes the encoder id AND the corpus text set, so changing either invalidates it. A stale cache silently training on the wrong vectors would be worse than no cache.
(router/train.py:239-243.)
encoder_id = f"{encoder.model_name}@{encoder.revision}"
fingerprint = _corpus_fingerprint(corpus, encoder_id, encoder.max_length)
vectors_file = cache_path / f"embeddings_{fingerprint[:16]}.npy"
meta_file = cache_path / f"embeddings_{fingerprint[:16]}.json"
if vectors_file.exists() and meta_file.exists() and not force:
try:
cached = np.load(vectors_file)
meta = json.loads(meta_file.read_text(encoding="utf-8"))
if cached.shape[0] == len(corpus) and meta.get("fingerprint") == fingerprint:
return cached.astype(np.float32)
except Exception: # noqa: BLE001 - a bad cache is a miss, not an error
pass
(router/train.py:248-260.)
Three robustness details, each deliberate:
- A corrupt cache is a cache miss, not an error. The
except Exception: passis commented as such β a truncated.npyor malformed JSON causes a re-embed, not a crash. - The row count is checked and the fingerprint is checked. Either alone would be weaker: the fingerprint catches a changed corpus, the shape check catches a truncated file whose metadata still parses.
- The
.npyand.jsonare keyed by the same 16-hex prefix, so they cannot be mismatched.
The sidecar metadata records the provenance of the cache:
meta_file.write_text(
json.dumps(
{
"fingerprint": fingerprint,
"encoder": encoder_id,
"max_length": encoder.max_length,
"count": int(vectors.shape[0]),
"dim": int(vectors.shape[1]),
"seconds": round(time.time() - started, 3),
},
indent=2,
sort_keys=True,
),
encoding="utf-8",
)
(router/train.py:267-281.)
The row-index bookkeeping is by text, not by position:
# index bookkeeping: embeddings follow corpus order -----------------
text_to_row = {e.text: i for i, e in enumerate(corpus)}
all_ex = corpus.examples
def rows_for(examples: list[RouterExample]) -> np.ndarray:
idx = [text_to_row[e.text] for e in examples]
return embeddings[np.asarray(idx, dtype=np.int64)]
(router/train.py:547-557.) Embeddings are computed over corpus.texts() in corpus order, and the
split lists are subsets of the same objects; the text_to_row map is what re-associates a split
example with its vector. A subtle consequence: because the map is keyed on text, a corpus that
survived dedupe() is required β two examples with the same text would map to one row. dedupe()
has already guaranteed that.
15.3 Class weights, computed on the train split only
# Class weights: inverse frequency on the TRAIN split only. Computing them
# on the full corpus would leak val/test label distribution into training.
train_task_counts = np.zeros(NUM_TASKS, dtype=np.float64)
for e in train_ex:
train_task_counts[TASK_TO_INDEX[e.task]] += 1
counts = np.maximum(train_task_counts, 1.0)
class_weights = torch.tensor(
(counts.sum() / (NUM_TASKS * counts)), dtype=torch.float32, device=device
)
(router/train.py:568-576.)
The formula is the standard "balanced" weighting: w_c = N / (C Β· n_c), so a class with the mean
count gets weight 1.0, a rarer class gets more, and the weights average to 1.0 across classes.
np.maximum(train_task_counts, 1.0) floors an absent class at 1 to avoid a division by zero β
though validate() has already guaranteed every class has examples.
Computing on the train split only is a leakage control, and the comment says so. Using the full corpus would let the val/test label distribution influence the loss weights.
The shipped weights are not recorded in the artifact. metadata.json.hyperparameters carries
the loss weights but not the derived class_weights. The values are therefore UNKNOWN β not established from the available evidence; they are computable from
metadata.json.split.task_counts.train, and this document does not compute them.
15.4 Optimizer and scheduler
optimizer = torch.optim.AdamW(
model.parameters(), lr=learning_rate, weight_decay=weight_decay
)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=max(1, epochs))
ce = nn.CrossEntropyLoss(weight=class_weights)
ce_mod = nn.CrossEntropyLoss()
bce = nn.BCEWithLogitsLoss()
(router/train.py:578-584.)
AdamW with lr=0.001, weight_decay=0.01 β decoupled weight decay, which is the "W" in the
name and the reason it is preferred over Adam(weight_decay=...).
CosineAnnealingLR with T_max = epochs (60). The shipped history confirms the cosine schedule
exactly: the learning rate starts at 0.0009993147673772868 (epoch 1), passes exactly 0.00075 at
epoch 20, exactly 0.0005 at epoch 30, exactly 0.00025 at epoch 40, and reaches exactly 0.0 at
epoch 60 (metadata.json.history). Those four round numbers at 20/30/40/60 are the signature of
T_max=60 with no warm restarts.
Best-state selection. Every epoch, the model is evaluated on val and the best val task accuracy is snapshotted:
if val_metrics.task_accuracy > best_val:
best_val = val_metrics.task_accuracy
best_state = {k: v.detach().clone() for k, v in model.state_dict().items()}
...
if best_state is not None:
model.load_state_dict(best_state)
(router/train.py:643-656.) The shipped history shows the best val task accuracy first reached at
epoch 8 (0.9651162790697675) and never exceeded β every epoch from 8 to 60 reports exactly that
value. So the shipped adapter's weights are the epoch-8 snapshot, and the remaining 52 epochs
did not improve the selected metric. This is MEASURED, directly from the history array.
Note the strict >: a tie does not replace the snapshot, so the earliest epoch achieving the
best value wins. With a flat plateau from epoch 8 onward, that is epoch 8.
15.5 The epoch loop, and the shuffle
rng = random.Random(seed)
for epoch in range(epochs):
model.train()
order = list(range(n_train))
rng.shuffle(order)
epoch_loss = 0.0
batches = 0
for start in range(0, n_train, batch_size):
idx = torch.tensor(order[start:start + batch_size], dtype=torch.long, device=device)
out = model(train_tensors["x"][idx])
...
scheduler.step()
val_metrics = evaluate_split(model, val_ex, val_x, "val", device)
history.append(
{
"epoch": float(epoch + 1),
"loss": epoch_loss / max(1, batches),
"val_task_accuracy": val_metrics.task_accuracy,
"val_combined_accuracy": val_metrics.combined_accuracy,
"lr": float(optimizer.param_groups[0]["lr"]),
}
)
(router/train.py:592-641.)
Four details:
- The shuffle is a
random.Random(seed)instance, not the global RNG. Becausetrain_routeralso seeds the globalrandomat the top, both are deterministic β but the local instance means the shuffle stream is independent of anything else that touchesrandomduring the run. scheduler.step()is called once per epoch, after the batch loop, before the val evaluation. The history'slrfield is therefore the rate used for the next epoch's first batch.historyis appended every epoch, and the shipped artifact contains 60 rows β one per epoch, matchingepochs: 60.val_metricsis recomputed inside the loop and again after the best-state restore. The post-restore recomputation is what produces the finalmetrics.valblock, which is why the recordedval.task_accuracy(0.9651) matches the epoch-8 history value exactly.
16. Metrics β SplitMetrics and evaluate_split
@dataclass
class SplitMetrics:
"""Per-split evaluation results."""
name: str
n: int
task_accuracy: float
modality_accuracy: float
binary_accuracy: dict[str, float]
combined_accuracy: float
macro_f1_task: float
per_task_recall: dict[str, float] = field(default_factory=dict)
#: How many test examples each class actually had. A recall without
#: support is a number that means nothing; they are reported together.
task_support: dict[str, int] = field(default_factory=dict)
hard_negative_accuracy: float | None = None
(router/train.py:79-94.)
| Field | Meaning | Rounding on serialisation |
|---|---|---|
name |
"train" / "val" / "test" |
β |
n |
examples in the split | β |
task_accuracy |
argmax over task_logits == target |
4 dp |
modality_accuracy |
argmax over modality_logits == target |
4 dp |
binary_accuracy |
per head: sigmoid(logit) >= 0.5 == target |
4 dp each |
combined_accuracy |
every head correct for the example to count | 4 dp |
macro_f1_task |
macro-averaged F1 over the six task classes | 4 dp |
per_task_recall |
recall per class that has support > 0 | 4 dp each |
task_support |
example count per class, all six keys | β |
hard_negative_accuracy |
accuracy over hn_* examples, None if none |
4 dp or null |
16.1 The support-alongside-recall decision
def _recall_per_class(
y_true: list[int], y_pred: list[int], n_classes: int
) -> tuple[list[float], list[int]]:
"""Per-class recall AND per-class support.
Support is returned alongside recall because a class with zero test
examples has a recall of 0.0 that means nothing. Returning them together
makes that impossible to misread: `vqa 0.000` without support is exactly
the kind of number that gets quoted as a failure when it is an absence.
"""
(router/train.py:332-341.) This is the code-level fix for defect 1 of
docs/PHASE4_ROUTER_REPORT.md Β§"Four real defects" β the phase where vqa landed with 72 training
examples and 0 test examples, so its recall printed as 0.000 and the gate reported "BELOW
TARGET (0.909)" over a model never asked about vqa at all.
Note the deliberate asymmetry in the two fields: per_task_recall omits a zero-support class
(if support[i] > 0), while task_support includes all six keys. So a class with no test
examples appears in task_support as 0 and is absent from per_task_recall β which is exactly
what the shipped artifact shows for no class, because stratification gave all six test support.
16.2 The local macro-F1
def _macro_f1(y_true: list[int], y_pred: list[int], n_classes: int) -> float:
"""Macro-averaged F1. Implemented locally to avoid a sklearn dependency in
the training path (sklearn is installed for dev, not required at runtime)."""
if not y_true:
return 0.0
f1s: list[float] = []
for c in range(n_classes):
tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
precision = tp / (tp + fp) if (tp + fp) else 0.0
recall = tp / (tp + fn) if (tp + fn) else 0.0
f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) else 0.0
f1s.append(f1)
return sum(f1s) / len(f1s)
(router/train.py:315-329.)
The denominator is n_classes = 6, always β including a class with zero predictions and zero
support, which contributes 0.0. This is the same denominator convention that makes the
optical-SAR macro-F1 (0.434161) low by construction with 5 absent classes
(README.md:827-829) β a shared convention across the project, worth noting because it means a
macro score here is not comparable to a macro score computed over present classes only.
16.3 evaluate_split() β and the combined_accuracy definition
@torch.no_grad()
def evaluate_split(
model: IntentAdapter,
examples: list[RouterExample],
embeddings: np.ndarray,
name: str,
device: str,
batch_size: int = 128,
) -> SplitMetrics:
(router/train.py:352-360.)
The empty-split short-circuit is explicit:
if not examples:
return SplitMetrics(
name=name, n=0, task_accuracy=0.0, modality_accuracy=0.0,
binary_accuracy={h: 0.0 for h in BINARY_HEADS},
combined_accuracy=0.0, macro_f1_task=0.0,
task_support={task: 0 for task in TASK_CLASSES},
)
(router/train.py:363-369.) Note task_support is populated with six zeros, so an empty split
still produces a complete support dict rather than a missing one.
Batched inference at 128, larger than the training batch of 64 β evaluation has no gradient and
no optimizer state, so it can afford larger batches. @torch.no_grad() decorates the whole
function.
combined_accuracy is an AND over all five heads:
# Combined: every head must be right for an example to count.
combined_hits = 0
for i in range(n):
ok = task_pred[i] == task_true[i] and modality_pred[i] == modality_true[i]
if ok:
for j, head in enumerate(BINARY_HEADS):
if int(tensors["binary"][i, j].item()) != binary_pred[head][i]:
ok = False
break
combined_hits += int(ok)
(router/train.py:401-410.) This is the strictest of the reported metrics, and it is why the
shipped test combined_accuracy (0.9625) is lower than task_accuracy (0.975): five examples got
the task right but at least one binary head wrong. On val the gap is much wider β 0.7907 combined
against 0.9651 task β because the val modality_accuracy is only 0.8256, so most of the
combined failures are modality errors, not task errors.
16.4 The hard-negative metric
# Hard negatives: examples whose group starts with "hn_" (curated) β these
# are the one-token-difference pairs the router exists to get right.
hn_examples = [e for e in examples if e.group.startswith(HARD_NEGATIVE_PREFIX)]
hn_acc: float | None = None
if hn_examples:
hn_idx = [i for i, e in enumerate(examples) if e.group.startswith(HARD_NEGATIVE_PREFIX)]
hits = sum(1 for i in hn_idx if task_pred[i] == task_true[i])
hn_acc = hits / len(hn_idx)
(router/train.py:414-421.)
Three properties: it is None when the split has no hn_* examples (which is why train and
val both record null in the shipped metadata β the families are forced to test); it measures
task accuracy only, not the binary heads; and it is computed over the hn_* examples in
whatever split is being evaluated.
| Split | hard_negative_accuracy |
Why |
|---|---|---|
| train | null |
no hn_* groups β they are forced to test |
| val | null |
same |
| test | 0.8 | 14 hn_* examples, 11 correct |
The test value of 0.8 means 2 of the 14 curated hard-negative examples were mis-classified at the
task level. The artifact does not record which two. That is UNKNOWN β not established from the available evidence, and it is the single most useful thing a future run could add, because those
two examples are precisely the router's remaining frontier.
17. The stub path β tests only, never deployed
def _stub_embeddings(corpus: RouterCorpus, dim: int = 384) -> np.ndarray:
"""Deterministic bag-of-hashes pseudo-embedding.
Exists so `train_router` can be exercised in unit tests without downloading
a 90 MB model. It is NOT a semantic encoder: two paraphrases get unrelated
vectors. Any artifact trained with it is marked `encoder_type: "stub"` and
must never be deployed.
"""
out = np.zeros((len(corpus), dim), dtype=np.float32)
for row, example in enumerate(corpus):
for token in example.text.lower().split():
h = int(hashlib.sha256(token.encode()).hexdigest()[:8], 16)
out[row, h % dim] += 1.0
norms = np.linalg.norm(out, axis=1, keepdims=True)
norms[norms == 0] = 1.0
return out / norms
(router/train.py:728-743.)
The mechanism: each token is hashed to a bucket in [0, 384) and incremented, then the row is
L2-normalised β a hashing-trick bag-of-words with no semantic structure. Two paraphrases get
unrelated vectors, which is exactly why it is unusable as a router and adequate as a test
fixture.
Three guards around it: the encoder_type is recorded as "stub"; encoder_meta records
model: "stub", revision: "none", max_length: 0, parameters: 0; and the docstring's
"must never be deployed" is repeated in docs/PHASE4_ROUTER_REPORT.md Β§"Standing caveats". The
shipped artifact records encoder_type: "frozen_sentence_transformer" β so the shipped adapter is
not a stub artifact.
A subtlety in the stub path's corpus hash: when encoder is None, corpus_hash is computed
with max_length=0 and the encoder id "stub@none":
corpus_hash = _corpus_fingerprint(
corpus, encoder_meta["model"] + "@" + str(encoder_meta["revision"]), 0
)
(router/train.py:543-545.) The shipped artifact's corpus.hash is
8054810736ef97c3db873b2d7073948773a8982f1d16411833d39e15e1871e83, computed with max_length=0
regardless of the encoder used β a small inconsistency worth naming, since it means the recorded
corpus hash is not the same value the embedding cache keys on (which uses the real
max_length=128). Both are deterministic; they are simply different hashes over different payloads.
18. TrainingResult β and the gate
@dataclass
class TrainingResult:
"""Everything the caller needs to decide the next step."""
artifact_dir: Path
metadata: dict[str, Any]
train_metrics: SplitMetrics
val_metrics: SplitMetrics
test_metrics: SplitMetrics
split_report: dict[str, Any]
corpus_hash: str
duration_seconds: float
history: list[dict[str, float]] = field(default_factory=list)
(router/train.py:114-126.)
18.1 unmeasured_test_tasks β the visibility property
@property
def unmeasured_test_tasks(self) -> list[str]:
"""Task classes with no test examples. The gate cannot speak for these."""
return [
task for task in TASK_CLASSES
if self.test_metrics.task_support.get(task, 0) == 0
]
(router/train.py:128-134.) A class absent from the test set is named, not silently scored as
zero. The shipped result has an empty list β all six classes have test support (13 / 13 / 17 / 12 /
10 / 15).
18.2 gate2_passed β the two-part condition
@property
def gate2_passed(self) -> bool:
"""Gate 2 passes only if EVERY task class was measured AND cleared 0.95.
Without the first condition a class can be absent from the test set and
the gate reports a pass over a model that was never asked about it.
"""
return not self.unmeasured_test_tasks and self.test_metrics.task_accuracy >= 0.95
(router/train.py:136-143.)
Two conditions, ANDed, and the first is the novel one. A naive gate would check only
task_accuracy >= 0.95; that gate passed at 0.909 over five of six classes in the defective run.
The two-part gate cannot pass while any class is unmeasured, which is what forces the stratification
fix rather than allowing it to be skipped.
The gate's own threshold is 0.95, and it is a hardcoded literal in the property β not a config
key. The shipped test accuracy is 0.975, so gate2_passed is True, and
docs/PHASE4_ROUTER_REPORT.md records the verdict block verbatim:
GATE 2 ROUTER ACCEPTANCE: task accuracy >= 0.95, all classes measured
classes with test support : 6/6
-> PASS (0.975)
18.3 summary() β the printed report
TrainingResult.summary() (router/train.py:145-213) builds a fixed-width text report with five
blocks: a header with corpus / groups / hash / encoder / adapter params / epochs / duration; the
split sizes and the leakage-clean flag; a metrics table with one row per split; the hard-negative
accuracy when present; the per-task recall table with an n/a row and the explicit
(no test examples) marker for a zero-support class; and the gate verdict.
Two details in the gate block are worth quoting because they are the honesty mechanism in human-readable form:
measured = len(TASK_CLASSES) - len(self.unmeasured_test_tasks)
lines.append(f" classes with test support : {measured}/{len(TASK_CLASSES)}")
if self.unmeasured_test_tasks:
lines.append(
f" classes NOT measurable : {', '.join(self.unmeasured_test_tasks)}"
)
acc = self.test_metrics.task_accuracy
if self.unmeasured_test_tasks:
lines.append(
f" -> NOT MEASURABLE ({acc:.3f} over the {measured} classes that "
f"have test data)"
)
else:
verdict = "PASS" if acc >= 0.95 else "BELOW TARGET"
lines.append(f" -> {verdict} ({acc:.3f})")
(router/train.py:195-209.) An unmeasured class produces the phrase "NOT MEASURABLE", never
"BELOW TARGET" β the two are different facts and the report distinguishes them.
Part E β Inference
19. RouterPrediction β the decision plus its explanation
@dataclass
class RouterPrediction:
"""A router decision plus everything needed to explain it in a trace."""
intent: Intent
task_probs: dict[str, float] = field(default_factory=dict)
modality_probs: dict[str, float] = field(default_factory=dict)
binary_probs: dict[str, float] = field(default_factory=dict)
used_fallback: bool = False
fallback_rule: str | None = None
matched_terms: tuple[str, ...] = ()
above_threshold: bool = True
def to_trace(self) -> dict[str, Any]:
"""Observable facts only. No chain-of-thought."""
return {
"task": self.intent.task.value,
"modality": self.intent.modality.value,
"temporal": self.intent.temporal,
"spatial_output": self.intent.spatial_output,
"language_output": self.intent.language_output,
"confidence": round(self.intent.confidence, 4),
"source": self.intent.source,
"above_threshold": self.above_threshold,
"used_fallback": self.used_fallback,
"fallback_rule": self.fallback_rule,
}
(router/classifier.py:80-106.)
| Field | Populated by the learned path? | Populated by the fallback? |
|---|---|---|
intent |
yes β Intent(source="learned") |
yes β Intent(source="lexical_fallback") |
task_probs |
yes β all six softmax probabilities | no β empty dict |
modality_probs |
yes β all four | no β empty dict |
binary_probs |
yes β the three sigmoid values | no β empty dict |
used_fallback |
False |
True |
fallback_rule |
None |
the rule id, e.g. "spatial" |
matched_terms |
() |
the terms that fired |
above_threshold |
confidence >= 0.70 |
match.confidence >= 0.70 |
to_trace() deliberately omits the probability dicts. It emits the task, modality, three
booleans, the rounded confidence, the source, above_threshold, used_fallback and
fallback_rule β and nothing else. README.md:601-603 states the rule:
The router's own trace projection is the model of this: it emits the task, modality, the three booleans, the rounded confidence, the source,
above_threshold,used_fallbackandfallback_ruleβ and nothing else.
The docstring's "Observable facts only. No chain-of-thought." is the enforcement note. Note that
matched_terms is not in the trace projection even though it is on the dataclass β the
fallback's matched substrings are diagnostic metadata for a caller, not part of the trace contract.
20. IntentRouter β construction and the from_config warning
class IntentRouter:
"""Loads the adapter (and optionally the encoder) and routes queries."""
def __init__(
self,
adapter: IntentAdapter | None = None,
encoder: FrozenEncoder | None = None,
confidence_threshold: float = 0.70,
device: str = "cpu",
) -> None:
if not 0.0 <= confidence_threshold <= 1.0:
raise RoutingError(
f"confidence_threshold must be in [0,1], got {confidence_threshold}"
)
self.adapter = adapter
self.encoder = encoder
self.confidence_threshold = confidence_threshold
self.device = device
if self.adapter is not None:
self.adapter.eval()
self.adapter.to(device)
(router/classifier.py:109-130.)
confidence_threshold = 0.70 β the default matches configs/base.yaml
Β§router.confidence_threshold: 0.70, and from_config reads the config value rather than relying
on the default.
An adapter, once passed, is immediately put in eval mode and moved to the device. So the caller cannot accidentally route with a training-mode adapter.
20.1 from_config() and the warning that is the whole point
@classmethod
def from_config(
cls,
config,
adapter_path: str | Path | None = None,
load_encoder: bool = True,
device: str | None = None,
) -> "IntentRouter":
"""Build from the central config, optionally loading a trained adapter.
IMPORTANT: `adapter_path` defaults to None, so the default router runs
the LEXICAL FALLBACK, not the trained adapter. That default is
deliberate -- it keeps tests fast and offline -- but it means a caller
who never passes `adapter_path` gets fallback answers while believing
the trained model is serving. The two are distinguishable but not
obviously so: on the spec section 29 examples the fallback returns
confidence 0.850-0.920 against the trained model's 0.780-1.000.
Check `router.has_adapter` (and `router.adapter_source`) before
trusting a routing decision as model-backed.
"""
(router/classifier.py:134-154.)
This warning is the most consequential paragraph in the router. A default-constructed router is
a lexical router. The two paths produce answers in overlapping confidence bands, and the
fallback can be more confident than the trained model β so confidence alone cannot distinguish
them. core/planner.py:48-62 records this as the reason the planner applies a provenance discount:
A lexical fallback at 0.9 is not the same evidence as a learned model at 0.9: one is a regex that matched, the other is a learned distribution. Treating them identically would let a matched keyword outrank the model it fell back from.
Two dimension checks, both loud:
if adapter_path is not None:
adapter, metadata = load_adapter(adapter_path, device=device)
# An adapter trained against a different input dimension cannot be
# used with this encoder; fail loudly rather than producing garbage.
expected_dim = int(config.get("router.embedding_dim", 384))
if adapter.input_dim != expected_dim:
raise ModelLoadError(
f"adapter input_dim={adapter.input_dim} does not match the "
f"configured embedding dim {expected_dim}",
specialist="router",
)
if load_encoder:
from router.encoder import build_encoder
encoder = build_encoder(config, device=device)
if adapter is not None and adapter.input_dim != encoder.embedding_dim:
raise ModelLoadError(
f"adapter input_dim={adapter.input_dim} but the loaded encoder "
f"produces {encoder.embedding_dim}-d embeddings",
specialist="router",
)
(router/classifier.py:164-185.)
Check one compares the adapter to the config; check two compares it to the loaded encoder.
Both are needed: check one catches a config that disagrees with the adapter before the encoder is
loaded at all; check two catches a config that is right while the actual encoder is different.
The from router.encoder import build_encoder is a deferred import inside the method so that
load_encoder=False does not pay the import cost.
20.2 The three properties
@property
def has_adapter(self) -> bool:
return self.adapter is not None
@property
def adapter_source(self) -> str:
"""Which path `route()` will take: 'trained' or 'lexical_fallback'.
...
"""
return "trained" if self.has_adapter else "lexical_fallback"
@property
def has_encoder(self) -> bool:
return self.encoder is not None
(router/classifier.py:196-213.)
adapter_source returns "trained" when an adapter is present β regardless of whether an
encoder is loaded. So a router with an adapter but no encoder reports "trained" while route()
actually takes the fallback path (because has_encoder is False, so _predict_learned is never
called). This is a genuine edge case worth naming: the property answers "is a trained adapter
attached?", not "will the learned path run?". The two differ exactly when
has_adapter and not has_encoder. core/planner.py:50 relies on adapter_source for provenance, so
a caller in that state would mislabel the source. It is IMPLEMENTED as written; whether it is a
defect depends on the intended contract, and the property's docstring β "Which path route() will
take" β reads as the stronger claim. Recorded as an OPEN observation, not asserted as a bug.
21. _predict_learned() β the forward pass and the coherence repair
def _predict_learned(self, query: str) -> RouterPrediction | None:
"""Run the encoder + adapter. Returns None when either is unavailable."""
if self.adapter is None or self.encoder is None:
return None
embedding = self.encoder.encode_one(query, normalize=True)
tensor = torch.from_numpy(embedding).unsqueeze(0).to(self.device)
with torch.no_grad():
out = self.adapter(tensor)
task_probs = torch.softmax(out.task_logits, dim=-1)[0]
modality_probs = torch.softmax(out.modality_logits, dim=-1)[0]
temporal_p = torch.sigmoid(out.temporal_logit)[0]
spatial_p = torch.sigmoid(out.spatial_logit)[0]
language_p = torch.sigmoid(out.language_logit)[0]
task_idx = int(torch.argmax(task_probs).item())
modality_idx = int(torch.argmax(modality_probs).item())
confidence = float(task_probs[task_idx].item())
(router/classifier.py:217-235.)
The confidence is the max softmax probability of the task head, not a modality or binary probability. That is a deliberate choice: the task is what the controller keys off, so the gate gates on the task.
encode_one β unsqueeze(0) β torch.no_grad(). The unsqueeze(0) adds the batch dimension
the adapter's forward guard requires; the no_grad is the inference-mode guard.
The coherence repair, in full:
# Coherence repair. The heads are independent by construction, so a
# confident task label with an incoherent binary head is possible. We
# resolve in favour of the task label, because the task is what the
# controller keys off β and we record that we did so.
temporal = binary_probs["temporal"] >= 0.5
spatial = binary_probs["spatial_output"] >= 0.5
language = binary_probs["language_output"] >= 0.5
if task_label in ("change",):
temporal = True
if task_label == "grounding":
spatial = True
if task_label == "unsupported":
language = False
spatial = False
temporal = False
if task_label == "optical_sar" and modality_label != "optical_sar":
modality_label = "optical_sar"
(router/classifier.py:246-263.)
Four repairs, and each maps to a label-space invariant:
| Repair | Rule | Label-space source |
|---|---|---|
change β temporal = True |
a change task always needs two acquisitions | TEMPORAL_TASKS = {"change"} |
grounding β spatial = True |
grounding's output is spatial | SPATIAL_TASKS = {"grounding"} |
unsupported β all three False |
unsupported claims no capability | RouterCorpus.validate() check (d) |
optical_sar β modality = "optical_sar" |
a dual-modality task has a dual-modality label | DUAL_MODALITY_TASKS = {"optical_sar"} |
Note what is not repaired: a caption task with spatial_output=True is left alone; a vqa
task with temporal=True is left alone. The repairs are the necessary implications only, matching
the asymmetry in RouterCorpus.validate() (Β§11.2(e)). The comment's "we resolve in favour of the
task label β¦ and we record that we did so" is slightly stronger than the code: the repair happens,
but nothing records that a repair fired β RouterPrediction.to_trace() has no
coherence_repaired field. The repair is therefore silent. This is a small gap between the comment
and the implementation, named here rather than smoothed over.
The Intent construction is wrapped in a try/except that converts a Pydantic error into a
RoutingError:
try:
intent = Intent(
task=_TASK_TO_SCHEMA[task_label],
modality=_MODALITY_TO_SCHEMA[modality_label],
temporal=temporal,
spatial_output=spatial,
language_output=language,
confidence=confidence,
source="learned",
)
except Exception as exc: # noqa: BLE001 - pydantic validation
raise RoutingError(
f"learned router produced an invalid intent: {exc}",
context={"query_length": len(query), "task": task_label},
) from exc
(router/classifier.py:265-279.) The error context deliberately carries query_length, not the
query. That is a privacy-conscious choice: a routing failure does not put the user's text into an
error record.
Intent's own validator will also fire for two cases the repair already handled β the
_consistency model validator (core/schemas.py:107-114) forces temporal=True for CHANGE and
CHANGE_VQA, and forces modality=OPTICAL_SAR for an OPTICAL_SAR task with UNKNOWN modality.
So the repair and the schema validator agree, and the repair is what keeps the schema validator
from having to do the work.
22. _predict_fallback() β the lexical path
def _predict_fallback(self, query: str) -> RouterPrediction:
match: LexicalMatch = lexical_route(query)
intent = Intent(
task=_TASK_TO_SCHEMA[match.task],
modality=_MODALITY_TO_SCHEMA[match.modality],
temporal=match.temporal,
spatial_output=match.spatial_output,
language_output=match.language_output,
confidence=match.confidence,
source="lexical_fallback",
)
return RouterPrediction(
intent=intent,
used_fallback=True,
fallback_rule=match.rule,
matched_terms=match.matched_terms,
above_threshold=match.confidence >= self.confidence_threshold,
)
(router/classifier.py:292-309.)
No try/except here β because lexical_route is total: every path returns a valid
LexicalMatch, and self_check() asserts the label validity of the curated cases. The
matched_terms and fallback_rule are carried through, which is what makes the fallback
inspectable where the learned path is not.
23. route() β the three-step arbitration
def route(self, query: str) -> RouterPrediction:
"""Route a query to an Intent.
Order:
1. learned router, if confident enough -> use it
2. otherwise lexical fallback
3. if the fallback also lands below threshold -> return it anyway,
flagged, and let the controller decide. The router never refuses
to answer; `above_threshold` carries the uncertainty.
"""
if not isinstance(query, str) or not query.strip():
raise RoutingError("query must be a non-empty string")
learned: RouterPrediction | None = None
if self.has_adapter and self.has_encoder:
learned = self._predict_learned(query)
if learned is not None and learned.above_threshold:
return learned
fallback = self._predict_fallback(query)
if learned is None:
return fallback
# Both ran. Prefer whichever is more confident; the fallback wins ties
# on interpretability (its rule and matched terms are inspectable).
if fallback.intent.confidence >= learned.intent.confidence:
return fallback
return learned
(router/classifier.py:311-339.)
Trace every branch. This is the algorithm of the entire router in 28 lines.
| Situation | What runs | What is returned |
|---|---|---|
| no adapter and no encoder | fallback only | fallback |
| adapter + encoder, learned β₯ 0.70 | learned only (early return) | learned |
| adapter + encoder, learned < 0.70, fallback β₯ learned | both, then compare | fallback |
| adapter + encoder, learned < 0.70, fallback < learned | both, then compare | learned (flagged above_threshold=False) |
| adapter + encoder, learned < 0.70 and fallback also < 0.70 | both, then compare | whichever is larger, flagged below threshold |
The early return is the important control-flow fact. When the learned router clears the gate,
the fallback never runs β so on a confident learned route, used_fallback is False,
fallback_rule is None, and task_probs / modality_probs / binary_probs are all populated.
When both run, the tie-break favours the fallback, on the stated ground of interpretability:
the fallback's rule and matched terms are inspectable, the learned path's are not. Note that
>= means a tie goes to the fallback.
The router never raises on a low-confidence route. It returns the prediction with
above_threshold=False, and core/planner.py:43-46 explains why the planner consumes that bool
rather than re-reading the threshold:
So the planner consumes
above_threshold(a bool), never the numeric threshold. Re-readingrouter.confidence_thresholdhere would be a second, independent gate over the same quantity β two places bound to one knob, which is how a threshold ends up meaning two different things.
One input guard only: a non-string or blank query raises RoutingError. Nothing else about the
query is validated here; length, language and content are the specialist's concern.
route_batch(queries) is a trivial list comprehension over route (router/classifier.py:341-342)
β it is not batched inference, and the name should not be read as such. Each query gets its own
encode_one call.
24. Persistence β save_adapter and load_adapter
ADAPTER_WEIGHTS = "adapter.pt"
ADAPTER_METADATA = "metadata.json"
(router/classifier.py:349-350.)
def save_adapter(
directory: str | Path,
adapter: IntentAdapter,
metadata: dict[str, Any],
) -> Path:
"""Write the adapter plus its metadata. Metadata is mandatory."""
path = Path(directory)
path.mkdir(parents=True, exist_ok=True)
torch.save(
{
"state_dict": adapter.state_dict(),
"config": adapter.config_dict(),
},
path / ADAPTER_WEIGHTS,
)
payload = dict(metadata)
payload.setdefault("adapter_config", adapter.config_dict())
payload.setdefault("num_parameters", adapter.num_parameters())
(path / ADAPTER_METADATA).write_text(
json.dumps(payload, indent=2, sort_keys=True, default=str), encoding="utf-8"
)
return path
(router/classifier.py:353-376.)
Three details:
- Metadata is mandatory β the parameter has no default. A checkpoint without metadata cannot be written by this function.
setdefault, not assignment. Iftrain_routeralready putadapter_configornum_parametersin the metadata, they are preserved. The shipped artifact carriesadapter_configand a top-levelnum_parametersalongside theadapterblock β three representations of the same facts, which is why the file is verbose but self-describing.sort_keys=Trueβ the JSON is deterministic, so two identical runs produce byte-identical metadata (given identical inputs).default=strhandles any non-JSON-serialisable value by stringifying rather than raising.
24.1 load_adapter() and the doubled-path trap
def load_adapter(
directory: str | Path,
device: str = "cpu",
) -> tuple[IntentAdapter, dict[str, Any]]:
"""Load a saved adapter and its metadata.
Args:
directory: the artifact DIRECTORY written by `save_adapter`, OR the
weights file itself. Both are accepted: `save_adapter` writes
`<dir>/adapter.pt`, so a caller who was handed the weight path --
the obvious thing to try, since the name reads like a weight file --
would otherwise have `.pt` appended a second time.
...
"""
path = Path(directory)
# Accept the weights file directly. Without this, passing `.../adapter.pt`
# produces `<...>/adapter.pt/adapter.pt`, whose doubled filename is the
# tell but is easy to misread as a genuinely missing file.
if path.is_file():
if path.name != ADAPTER_WEIGHTS and path.suffix != ".pt":
raise ModelLoadError(
f"expected an adapter directory or a '{ADAPTER_WEIGHTS}' file, "
f"got the file {path}",
specialist="router",
)
weights = path
meta_path = path.parent / ADAPTER_METADATA
else:
weights = path / ADAPTER_WEIGHTS
meta_path = path / ADAPTER_METADATA
(router/classifier.py:379-413.)
This is a fix for a real usability failure, documented as such. save_adapter writes
<dir>/adapter.pt; a caller handed .../adapter.pt would naively get
<dir>/adapter.pt/adapter.pt. The loader accepts either form.
A second, more specific error for the same trap:
if not weights.exists():
# Name the specific confusion when a `.pt` path was given that exists
# as a directory or not at all, rather than reporting a doubled path.
if path.suffix == ".pt":
raise ModelLoadError(
f"no adapter weights at {weights}: {path} looks like a weights "
f"file, not an adapter directory. Pass the file itself (it must "
f"exist and end in .pt) or the directory containing "
f"'{ADAPTER_WEIGHTS}'.",
specialist="router",
context={"given": str(path), "resolved": str(weights)},
)
raise ModelLoadError(
f"adapter weights not found at {weights}", specialist="router"
)
(router/classifier.py:415-429.) The error's context dict carries both the given and the resolved
path, so a caller can see the doubling rather than infer it.
Metadata is optional at load time, and its absence is not fatal:
metadata: dict[str, Any] = {}
if meta_path.exists():
try:
metadata = json.loads(meta_path.read_text(encoding="utf-8"))
except json.JSONDecodeError:
metadata = {"_warning": "metadata.json was not valid JSON"}
(router/classifier.py:458-463.) A malformed metadata.json yields a _warning key rather than an
exception β because the metadata is descriptive, while the weights and the embedded config are
what the model needs to run.
The asymmetry between the two files is the design: adapter.pt is required and its absence
or mismatch is fatal; metadata.json is advisory and its absence is silent. router.pt's
embedded config is what makes the weights self-sufficient.
Part F β The lexical fallback
25. What the fallback is, and the four rules it obeys
"""SatQuery AI β deterministic lexical intent fallback.
The learned router handles natural phrasing. This module exists for the case
the router itself flags as low-confidence, and for environments where the
encoder cannot be loaded at all (a CPU-only Space with no cached weights).
Design rules:
* Purely lexical. No model, no embeddings, no randomness.
* Ordered rules, highest specificity first. The first match wins.
* Never invents capability. If nothing matches, it returns `unsupported`
with low confidence rather than guessing a task.
* Must agree with the learned router on the curated hard negatives. If the
two disagree on `"describe the water body"` vs `"show me the water body"`,
the fallback is wrong, not the router.
"""
(router/fallback.py:1-16.)
The fourth rule is unusual and important. It assigns the burden of disagreement: if the fallback
and the router disagree on a curated hard negative, the fallback is wrong. The learned router was
trained on those pairs; the fallback is hand-written. self_check() (Β§29) is the mechanism that
holds the fallback to it.
25.1 LexicalMatch β the returned evidence
@dataclass(frozen=True)
class LexicalMatch:
"""One rule hit, with the evidence that produced it."""
task: str
modality: str
temporal: bool
spatial_output: bool
language_output: bool
confidence: float
rule: str
matched_terms: tuple[str, ...]
(router/fallback.py:29-40.) frozen=True β a match is immutable, so a caller cannot mutate a
rule's output. The eight fields are exactly the five Intent slots plus confidence, plus the two
diagnostic fields (rule, matched_terms).
| Field | Example | Meaning |
|---|---|---|
task |
"grounding" |
one of TASK_CLASSES |
modality |
"optical" |
one of MODALITY_CLASSES |
temporal |
False |
β |
spatial_output |
True |
β |
language_output |
True |
β |
confidence |
0.88 |
the rule's fixed confidence |
rule |
"spatial" |
the rule id β one of 10 values (Β§27.10) |
matched_terms |
("where", "locate") |
the substrings that fired |
26. The seven term tables, in full
Every table is a tuple[str, ...], matched by substring containment after lower-casing and
stripping the query (router/fallback.py:121-126):
def _hits(text: str, terms: tuple[str, ...]) -> tuple[str, ...]:
return tuple(term for term in terms if term in text)
def _contains_any(text: str, terms: tuple[str, ...]) -> bool:
return any(term in text for term in terms)
in, not a regex, not a word-boundary match. So "where" matches inside "somewhere", and
"what" matches inside "whatever". This is a deliberate simplicity trade-off β the fallback is
the fallback, and the learned router handles nuance β but it is the mechanism behind the
residuals in Β§48.
26.1 _SPATIAL_TERMS β 14 terms, including bare "where"
#: "show me where" phrasings. Spatial intent.
_SPATIAL_TERMS: tuple[str, ...] = (
# Bare "where" is included deliberately. "Where did the change happen?"
# must resolve to change + spatial_output=True, and the temporal branch
# consults this table only AFTER the temporal terms have already matched.
# Without it, "where did" carries no spatial signal at all.
"where", "where is", "where are", "where's",
"locate", "location of",
"show me where", "show where",
"highlight", "mark the", "point out", "point to",
"draw a box", "draw box", "bounding box",
"find the", "find all",
"show me the", "show the region",
)
(router/fallback.py:47-60.)
Count: 19 entries, not 14. The comment explains the two load-bearing ones:
- Bare
"where"is the term that makes"Where did the change happen?"producechange + spatial_output=True. It is safe to include bare because the spatial table is consulted after the temporal table (Β§27.3), so a temporal query never reaches it as a task selector β only as a spatial flag. "show me the"is the term that separates"show me the water body"(grounding) from"describe the water body"(caption) β the first curated hard-negative family.
Note "draw box" as well as "draw a box" β the informal phrasing without the article.
26.2 _TEMPORAL_TERMS β 20 terms
#: Temporal phrasings.
_TEMPORAL_TERMS: tuple[str, ...] = (
"changed", "change", "changes", "changing",
"before and after", "before-and-after",
"between these two", "between the two",
# "Compare these two images." is a temporal request. The superficially
# similar "Compare the optical and radar images." is claimed by the
# dual-modality table, which is consulted first.
"these two images", "two images", "compare these", "compare the two",
"over time", "temporal", "time series", "time-series",
"differences between the two dates", "two dates",
"has the", "did the", "have the",
"compared to before", "since then",
)
(router/fallback.py:62-75.)
Count: 24 entries. The comment records the near-miss that the precedence rule resolves:
"Compare these two images." (temporal) versus "Compare the optical and radar images."
(dual-modality). The surface form is near-identical; the dual-modality table is consulted first, so
the second is claimed correctly.
Note that "change" is a substring of "changed", "changes" and "changing" β so the four
entries changed/change/changes/changing are redundant under substring matching: "change"
alone would match all four. They are listed explicitly for readability and for the matched-terms
output, which will report whichever entries the loop found. This is harmless redundancy, not a bug.
26.3 _DUAL_MODALITY_TERMS β 18 entries
#: Dual-modality phrasings.
_DUAL_MODALITY_TERMS: tuple[str, ...] = (
"optical and sar", "optical and radar", "sar and optical",
"radar and optical", "radar and optical", "both images",
"both sensors", "both modalities", "two modalities",
"sar image", "radar image", "radar data", "sar data",
"co-registered", "coregistered", "fuse", "fusion",
"jointly", "multi-modal", "multimodal",
)
(router/fallback.py:77-85.)
Count: 20 entries, with "radar and optical" listed twice β a duplicated entry, harmless under
_hits (which would report it twice in matched_terms) but a genuine source redundancy. Recorded
here as a cosmetic observation.
The explicit_pair subset used by rule 1 is:
explicit_pair = any(
term in dual_hits
for term in ("optical and sar", "optical and radar", "sar and optical",
"radar and optical", "both sensors", "both modalities",
"two modalities", "fuse", "fusion", "jointly",
"multi-modal", "multimodal", "co-registered", "coregistered")
)
(router/fallback.py:175-181.) Note that "both images", "sar image", "radar image",
"radar data" and "sar data" are in the table but not in explicit_pair β so a query
containing only "both images" does not trigger the dual-modality rule. The comment above the
rule explains why: "'both images' alone is ambiguous; require a modality word too."
(router/fallback.py:174.) This is the "two images" (temporal) versus "both images" (ambiguous)
distinction, and it is what keeps "Compare these two images." on the temporal branch.
26.4 _CAPTION_TERMS β 11 entries
#: Caption phrasings.
_CAPTION_TERMS: tuple[str, ...] = (
"describe", "description", "caption", "write a caption",
"summarize", "summarise", "summary of",
"tell me about this image", "what do you see",
"what can you see", "explain this image",
)
(router/fallback.py:87-93.) Count: 11. Note "summarize" and "summarise" are both present
for the en-US / en-GB split β a genuine, necessary pair, unlike the redundant change family.
26.5 _VQA_TERMS β 15 entries, including bare "what"
#: VQA phrasings.
_VQA_TERMS: tuple[str, ...] = (
# Bare "what" is needed because "What land cover is visible?" carries no
# other interrogative marker. Precedence protects it: the temporal,
# spatial and caption tables are all consulted before this one, so
# "what changed" and "what do you see" never reach here.
"what",
"how many", "how much", "is there", "are there",
"what is", "what are", "what kind", "what type",
"which", "does this", "do you see", "can you tell",
"is this", "are these", "identify the type",
)
(router/fallback.py:95-106.)
Count: 16. Bare "what" is the load-bearing entry, and the comment explains both why it is
needed and why it is safe: the temporal, spatial and caption tables are all consulted first, so
"what changed" and "what do you see" never reach the VQA rule.
This is the same bare-interrogative pattern as bare "where", and it is the pattern most
vulnerable to the substring-matching limitation. "what" inside "whatever", "what's" and
"somewhat" all fire.
26.6 _SAR_TERMS and _OPTICAL_TERMS
#: SAR-only phrasings.
_SAR_TERMS: tuple[str, ...] = (
"sar image", "sar data", "radar image", "radar data",
"synthetic aperture radar", "backscatter",
)
#: Optical-only phrasings.
_OPTICAL_TERMS: tuple[str, ...] = (
"optical image", "optical data", "multispectral", "true colour",
"true color", "rgb image", "visible image",
)
(router/fallback.py:108-118.) Six and seven entries. These two tables do not select a task
β they select a modality, and they are consulted in rules 4 and 5, after the temporal and
spatial rules. Note "true colour" and "true color" are both present, the second necessary
spelling pair.
Both tables overlap _DUAL_MODALITY_TERMS. "sar image" and "radar image" appear in both
_SAR_TERMS and _DUAL_MODALITY_TERMS, and "sar data"/"radar data" likewise. The precedence
rule resolves it: rule 1 (dual-modality) is checked first, so "sar image" alone triggers
explicit_pair only if a pair term is also present. If it is not, the query falls through to
rule 4 and becomes a SAR-only VQA request. That is the intended reading of
"What does this sar image show?" β a single-SAR-modality question, not a fusion request.
27. The eight ordered rules
lexical_route(query) is one function with eight sequential returns. The docstring states the
precedence and its two motivating examples:
def lexical_route(query: str) -> LexicalMatch:
"""Classify a query with ordered lexical rules.
Rule order encodes precedence, which matters because the phrasings overlap:
"show me where the change happened"
has both a spatial term AND a temporal term -> change + spatial
"compare optical and radar to locate built-up areas"
has dual-modality AND spatial -> optical_sar (spatial doesn't apply
to the joint workflow, whose output is a classification)
Precedence: dual_modality > temporal > spatial > caption > vqa > unsupported
"""
(router/fallback.py:134-147.)
Note the docstring's precedence line versus the code's actual rule order. The docstring lists six
levels; the code has eight rules, and the two extra ones β sar_single and optical_single β
sit between spatial and caption:
implemented order: dual_modality > temporal > spatial > sar_single > optical_single > caption > vqa > unsupported
docstring order: dual_modality > temporal > spatial > caption > vqa > unsupported
README order: dual_modality > temporal > spatial > caption > vqa > unsupported
The docstring and README.md:537 agree with each other and both omit the two modality rules.
The code is the authority, and the full eight-level precedence is the one stated above. This is a
documentation gap, not a behavioural one β the modality rules always precede caption, so a query
that would match both a modality term and a caption term is classified by modality. Recorded here so
a reader working from the README is not surprised by the code.
27.1 Rule 0 β the empty query
if not isinstance(query, str) or not query.strip():
return LexicalMatch(
task="unsupported",
modality="unknown",
temporal=False,
spatial_output=False,
language_output=False,
confidence=0.0,
rule="empty_query",
matched_terms=(),
)
text = query.lower().strip()
(router/fallback.py:148-160.)
confidence = 0.0 β the lowest confidence in the module, and task="unsupported". An empty
query is not a low-confidence guess; it is a definite non-request. Note this branch also catches a
non-string input, so lexical_route(None) returns a match rather than raising. The
text normalisation (lower-case + strip) is computed once, before any table is consulted.
All seven tables are evaluated before any rule fires:
dual_hits = _hits(text, _DUAL_MODALITY_TERMS)
temporal_hits = _hits(text, _TEMPORAL_TERMS)
spatial_hits = _hits(text, _SPATIAL_TERMS)
caption_hits = _hits(text, _CAPTION_TERMS)
vqa_hits = _hits(text, _VQA_TERMS)
sar_hits = _hits(text, _SAR_TERMS)
optical_hits = _hits(text, _OPTICAL_TERMS)
(router/fallback.py:162-168.) This is why spatial_hits is available to the temporal rule (rule 2
uses it to set spatial_output) even though the spatial rule (rule 3) has not been reached.
27.2 Rule 1 β dual_modality, confidence 0.92
if dual_hits:
explicit_pair = any(...)
if explicit_pair:
return LexicalMatch(
task="optical_sar",
modality="optical_sar",
temporal=False,
spatial_output=False,
language_output=True,
confidence=0.92,
rule="dual_modality",
matched_terms=dual_hits,
)
(router/fallback.py:170-192.) The highest confidence in the module (0.92) and the only rule
that sets modality="optical_sar". spatial_output=False is forced β the comment explains: the
joint workflow's output is a classification, not a box, so a spatial term in the query does not make
the output spatial. The if dual_hits: outer guard is redundant given the if explicit_pair:
inner guard (an empty dual_hits makes explicit_pair false), but it short-circuits the generator
expression. Harmless.
27.3 Rule 2 β temporal / temporal_spatial, confidence 0.85 / 0.90
if temporal_hits:
spatial = _contains_any(text, _SPATIAL_TERMS)
return LexicalMatch(
task="change",
modality="optical",
temporal=True,
spatial_output=spatial,
language_output=True,
confidence=0.90 if spatial else 0.85,
rule="temporal_spatial" if spatial else "temporal",
matched_terms=temporal_hits + spatial_hits,
)
(router/fallback.py:194-206.) This is the only rule with two confidences and two rule ids. The
spatial flag is computed from the spatial table regardless of rule order, which is the mechanism
that makes "Where did the change happen?" produce change + spatial_output=True at 0.90.
modality="optical" is hardcoded β a temporal request is assumed optical unless rule 1 claimed it.
matched_terms=temporal_hits + spatial_hits β both hit sets are concatenated, so the diagnostic
shows the temporal terms first.
27.4 Rule 3 β spatial, confidence 0.88
if spatial_hits:
return LexicalMatch(
task="grounding",
modality="optical",
temporal=False,
spatial_output=True,
language_output=True,
confidence=0.88,
rule="spatial",
matched_terms=spatial_hits,
)
(router/fallback.py:208-219.) The plain grounding rule. Because rule 2 has already claimed every
temporal query, a query reaching here has a spatial term and no temporal term.
27.5 Rule 4 β sar_single, confidence 0.75
if sar_hits and not optical_hits:
return LexicalMatch(
task="vqa",
modality="sar",
temporal=False,
spatial_output=False,
language_output=True,
confidence=0.75,
rule="sar_single",
matched_terms=sar_hits,
)
(router/fallback.py:221-232.) The task is vqa β a single-modality question about a SAR
image. The guard and not optical_hits is what distinguishes a SAR-only query from a dual-modality
one; a query mentioning both falls through to rule 5 (which also fails its and not sar_hits) and
then to caption/vqa.
27.6 Rule 5 β optical_single, confidence 0.72
if optical_hits and not sar_hits:
return LexicalMatch(
task="vqa",
modality="optical",
temporal=False,
spatial_output=False,
language_output=True,
confidence=0.72,
rule="optical_single",
matched_terms=optical_hits,
)
(router/fallback.py:234-245.) The lowest task-producing confidence (0.72), and the mirror of
rule 4. The two guards are mutually exclusive, so at most one of rules 4 and 5 can fire.
27.7 Rule 6 β caption, confidence 0.85
if caption_hits:
return LexicalMatch(
task="caption",
modality="unknown",
temporal=False,
spatial_output=False,
language_output=True,
confidence=0.85,
rule="caption",
matched_terms=caption_hits,
)
(router/fallback.py:247-258.) modality="unknown" β a caption request does not imply a sensor.
27.8 Rule 7 β vqa, confidence 0.78
if vqa_hits:
return LexicalMatch(
task="vqa",
modality="unknown",
temporal=False,
spatial_output=False,
language_output=True,
confidence=0.78,
rule="vqa",
matched_terms=vqa_hits,
)
(router/fallback.py:260-271.)
27.9 Rule 8 β no_match, confidence 0.30
# -- 8. nothing matched: refuse rather than guess -------------------
return LexicalMatch(
task="unsupported",
modality="unknown",
temporal=False,
spatial_output=False,
language_output=False,
confidence=0.30,
rule="no_match",
matched_terms=(),
)
(router/fallback.py:273-283.)
This is the rule that implements "never invents capability". A query matching nothing is
unsupported at 0.30 β below the 0.70 gate, so above_threshold is False and the planner
sees an uncertain refusal rather than a confident one. language_output=False is required: an
unsupported result must claim no capability, exactly as RouterCorpus.validate() check (d)
requires of the corpus.
The 0.30 confidence is the design's honesty mechanism. A "no match" is not a 0.0 (which would claim certainty that the query is nonsense) and not a 0.5 (which would claim uncertainty about a definite non-match). 0.30 is "probably not answerable, and I have no positive evidence either way".
27.10 The complete rule table
| # | Rule id | Fires when | Task | Modality | temp | spat | lang | Confidence |
|---|---|---|---|---|---|---|---|---|
| 0 | empty_query |
query is not a non-blank string | unsupported |
unknown |
F | F | F | 0.00 |
| 1 | dual_modality |
an explicit_pair dual-modality term is present |
optical_sar |
optical_sar |
F | F | T | 0.92 |
| 2 | temporal_spatial |
a temporal term and a spatial term | change |
optical |
T | T | T | 0.90 |
| 2 | temporal |
a temporal term, no spatial term | change |
optical |
T | F | T | 0.85 |
| 3 | spatial |
a spatial term | grounding |
optical |
F | T | T | 0.88 |
| 4 | sar_single |
a SAR term and no optical term | vqa |
sar |
F | F | T | 0.75 |
| 5 | optical_single |
an optical term and no SAR term | vqa |
optical |
F | F | T | 0.72 |
| 6 | caption |
a caption term | caption |
unknown |
F | F | T | 0.85 |
| 7 | vqa |
a VQA term | vqa |
unknown |
F | F | T | 0.78 |
| 8 | no_match |
nothing matched | unsupported |
unknown |
F | F | F | 0.30 |
Ten rule ids across eight rule positions (rule 2 has two ids). The confidence band is 0.72β0.92 for task-producing rules, plus the two boundary values 0.00 and 0.30.
The band overlaps and can exceed the trained model's band. router/classifier.py:148-150
records the measured comparison: "on the spec section 29 examples the fallback returns
confidence 0.850-0.920 against the trained model's 0.780-1.000." That overlap is why
core/planner.py applies LEXICAL_FALLBACK_DISCOUNT = 0.75 to its own reading (Β§43) rather than
trusting the raw number.
28. is_available() β the fallback's one dependency claim
def is_available() -> bool:
"""The fallback has no model dependency and is always available."""
return True
(router/fallback.py:286-288.) A constant True with a docstring that is the reason. The fallback
is the degradation path β it is what runs when the encoder cannot load, so it must never itself
have a load dependency. This is the function a registry would call to decide whether a lexical
capability exists; here it is unconditional.
29. self_check() β the ten curated cases
def self_check() -> list[str]:
"""Assert the fallback agrees with the curated hard negatives.
Returns a list of failures. An empty list means the fallback is internally
consistent with the label space.
"""
cases: tuple[tuple[str, str, bool | None], ...] = (
("Describe this image.", "caption", None),
("What land cover is visible?", "vqa", None),
("Show me the water body.", "grounding", None),
("Describe the water body.", "caption", None),
("What changed between these images?", "change", None),
("Where did the change happen?", "change", True),
("What changed?", "change", False),
("Compare the optical and radar images.", "optical_sar", None),
("Locate the buildings.", "grounding", None),
("Book me a flight to Delhi.", "unsupported", None),
)
(router/fallback.py:291-308.)
Each case is (query, expected_task, expected_spatial) where expected_spatial is None when only
the task is asserted. All ten cases, with the rule that satisfies each:
| # | Query | Expected task | Expected spatial_output |
Satisfying rule | Resulting confidence |
|---|---|---|---|---|---|
| 1 | Describe this image. |
caption |
β | caption |
0.85 |
| 2 | What land cover is visible? |
vqa |
β | vqa (bare "what") |
0.78 |
| 3 | Show me the water body. |
grounding |
β | spatial ("show me the") |
0.88 |
| 4 | Describe the water body. |
caption |
β | caption |
0.85 |
| 5 | What changed between these images? |
change |
β | temporal (spatial false) |
0.85 |
| 6 | Where did the change happen? |
change |
True | temporal_spatial |
0.90 |
| 7 | What changed? |
change |
False | temporal |
0.85 |
| 8 | Compare the optical and radar images. |
optical_sar |
β | dual_modality |
0.92 |
| 9 | Locate the buildings. |
grounding |
β | spatial ("locate") |
0.88 |
| 10 | Book me a flight to Delhi. |
unsupported |
β | no_match |
0.30 |
Cases 3 and 4 are the hard-negative pair β "Show me the water body." (grounding) versus
"Describe the water body." (caption) β and they are the reason the design rule says "the fallback
is wrong, not the router" if the two disagree. Cases 6 and 7 are the second hard-negative pair β
the same task with spatial_output flipped.
Case 5 needs a note. "What changed between these images?" contains "these two images"? No β
it contains "between these two"? No. It contains "change" and "these images". The temporal
table's "these two images" and "two images" entries do not match "these images". So the
rule fires on "change" (a substring of "changed") alone, giving temporal (not
temporal_spatial, since no spatial term is present) β confidence 0.85, spatial_output=False.
self_check() does not assert spatial_output for case 5 (None), so the case passes regardless.
Case 10 is the refusal case β an out-of-domain query that must reach no_match. It is in the
same table because "never invents capability" is as testable a property as any classification.
The five assertions per case:
failures: list[str] = []
for query, expected_task, expected_spatial in cases:
match = lexical_route(query)
if match.task != expected_task:
failures.append(
f"{query!r}: expected task {expected_task!r}, got {match.task!r} "
f"(rule={match.rule})"
)
if expected_spatial is not None and match.spatial_output != expected_spatial:
failures.append(
f"{query!r}: expected spatial_output={expected_spatial}, "
f"got {match.spatial_output}"
)
if not is_valid_task(match.task):
failures.append(f"{query!r}: produced invalid task {match.task!r}")
if not is_valid_modality(match.modality):
failures.append(f"{query!r}: produced invalid modality {match.modality!r}")
if match.task not in TASK_CLASSES:
failures.append(f"{query!r}: task outside the label space")
return failures
(router/fallback.py:310-330.)
The last three assertions are a label-space invariant check on every case, and the third is
redundant with the first (is_valid_task and task not in TASK_CLASSES test the same membership β
is_valid_task is literally name in TASK_TO_INDEX, and TASK_CLASSES is what TASK_TO_INDEX is
derived from). The redundancy is defensive, not harmful.
The return convention is worth naming: self_check() returns a list of failures, and an empty
list means pass. This is the same convention as RouterCorpus.validate(). A caller checks
truthiness of the returned list.
Status. IMPLEMENTED and exercised by tests/routing/test_router.py. The measured result of
self_check() on the shipped code is UNKNOWN β not established from the available evidence:
this document did not execute it, and no artifact records its output. What is established is that
the learned router and the fallback agree on the canonical queries
(docs/PHASE4_ROUTER_REPORT.md Β§"Standing caveats": "both paths currently agree on every canonical
query"), and that the learned router's test hard-negative accuracy is 0.800.
Part G β The measured result
30. The shipped adapter artifact
The router ships exactly one trained artifact, at artifacts/router/router_adapter_v001/:
| File | Role |
|---|---|
adapter.pt |
the weights + the embedded architecture config |
metadata.json |
626 lines of provenance, hyperparameters, split report, metrics and the full 60-epoch history |
docs/PHASE4_ROUTER_REPORT.md:36-37 records the size: "223.8 KB (51,725 adapter params on a
frozen 22,713,216-param encoder)."
30.1 The artifact's identity
| Key | Value |
|---|---|
artifact |
router_adapter |
created_at |
2026-09-16T08:24:38.236767+00:00 |
duration_seconds |
4.92 |
seed |
42 |
config_hash |
615478910dc266bf |
num_parameters |
51725 |
encoder_type |
frozen_sentence_transformer |
encoder.model |
sentence-transformers/all-MiniLM-L6-v2 |
encoder.revision |
1110a243fdf4 |
encoder.max_length |
128 |
encoder.parameters |
22713216 |
encoder.type |
frozen_sentence_transformer |
The created_at date is worth noting against the code. The adapter was trained on
2026-09-16, and the threshold sweep was created on 2026-09-21
(threshold_sweep_val.json.created_at: "2026-09-21T05:56:32.394830+00:00"). The corpus source in
the repository has clearly been edited after 2026-09-16 (Β§13.3), so the shipped adapter's training
data is not byte-identical to the current router/dataset.py. The adapter is a real, measured
artifact; it is simply not reproducible from the current source without reverting
router/dataset.py. This is stated because reproducibility claims depend on it (Β§50).
30.2 The corpus the adapter was trained on
"corpus": {
"by_source": { "curated": 66, "template": 510 },
"by_task": {
"caption": 91, "change": 115, "grounding": 128,
"optical_sar": 50, "unsupported": 105, "vqa": 87
},
"groups": 54,
"hash": "8054810736ef97c3db873b2d7073948773a8982f1d16411833d39e15e1871e83",
"positives": { "language_output": 471, "spatial_output": 164, "temporal": 115 },
"total": 576
}
The three positives counts are the binary-head label balance, and they explain the
combined_accuracy gap:
| Head | Positives | Positives / 576 | Interpretation |
|---|---|---|---|
language_output |
471 | 81.8 % | most requests want prose |
spatial_output |
164 | 28.5 % | a minority want coordinates |
temporal |
115 | 20.0 % | one fifth need two acquisitions |
temporal = 115 exactly equals the change class count (115), which is the corpus-level proof that
every change example is marked temporal and no non-change example is β the invariant
RouterCorpus.validate() check (e) enforces. spatial_output = 164 exceeds the grounding count
(128) by 36, which is the number of change examples marked spatial: 36 of the 115 change examples
carry spatial_output=True, i.e. 31.3 % β consistent with the 4-of-12 (33.3 %) ratio in the curated
change family (Β§12.1).
The corpus is synthetic and the phase says so. docs/PHASE4_ROUTER_REPORT.md Β§"Standing
caveats": "The corpus is synthetic. 576 examples, 54 groups, hand-written and templated.
hard-negative accuracy 0.800 is the most honest generalisation number here; the 0.975 headline is
partly earned on templates the split kept in training. Treat the router as working, not as
benchmarked."
31. Gate 2, and the three split metric blocks
The gate's verdict, from docs/PHASE4_ROUTER_REPORT.md Β§Verdict:
GATE 2 ROUTER ACCEPTANCE: task accuracy >= 0.95, all classes measured
classes with test support : 6/6
-> PASS (0.975)
31.1 The per-split table, from the artifact
| split | n |
task | modality | spatial | temporal | language | combined | macro F1 |
|---|---|---|---|---|---|---|---|---|
| train | 410 | 1.000 | 1.000 | 0.9707 | 1.000 | 1.000 | 0.9707 | 1.000 |
| val | 86 | 0.9651 | 0.8256 | 0.9302 | 1.000 | 0.9884 | 0.7907 | 0.9545 |
| test | 80 | 0.975 | 1.000 | 1.000 | 0.9875 | 1.000 | 0.9625 | 0.976 |
docs/PHASE4_ROUTER_REPORT.md:17-21 presents the same table rounded to three decimals (val task
0.965, val modality 0.826; test task 0.975, test modality 1.000). The artifact's four-decimal values
are the primary source; the report's three-decimal values are the same numbers rounded.
Two observations the table forces.
valis the weakest split on every head excepttemporal. Val task accuracy 0.9651 against test 0.975; val modality 0.8256 against test 1.000. Val is not a subset of test and the two are not comparable β val has 20changeand 19unsupportedexamples against test's 13 and 10 (metadata.json.split.task_counts), a different class mix.- The val
modality_accuracyof 0.8256 is what dragscombined_accuracyto 0.7907. The val task accuracy is 0.9651 and the three binary heads are 0.9302β1.000, so the combined figure's gap is dominated by modality errors. The valper_task_recallshowsoptical_sarat 0.7 (7 of 10) β the only class below 1.0 on val β which is consistent with modality being the weakest val head.
31.2 Per-class recall and support, test
| class | recall | n |
Misses |
|---|---|---|---|
vqa |
1.000 | 15 | 0 |
caption |
0.8462 | 13 | 2 |
grounding |
1.000 | 17 | 0 |
change |
1.000 | 13 | 0 |
optical_sar |
1.000 | 12 | 0 |
unsupported |
1.000 | 10 | 0 |
| total | β | 80 | 2 |
caption is the only class with a miss, and it accounts for exactly the 2 errors in the
0.975 headline (78/80 = 0.975). The phase's caveat names it:
captionrecall is 0.846 β 2 of 13 missed. Not investigated; within noise at n=13 and below this corpus's significance floor.
(docs/PHASE4_ROUTER_REPORT.md Β§"Standing caveats".) 2 of 13 at n=13 is not a finding. With 13
examples, one miss moves recall by 0.077; the class is below the corpus's significance floor.
31.3 macro_f1_task β 0.976 on test
Computed by the local _macro_f1 with n_classes = 6. Because all six classes have support, no
class contributes a structural 0.0, and the macro F1 (0.976) sits just above the accuracy (0.975)
β a sign that the two caption errors did not disproportionately hurt a rare class.
32. The 60-epoch history β the training trajectory, verbatim
metadata.json.history contains 60 rows, one per epoch. The shape of the curve is the whole
story of this model, so the key epochs are tabulated rather than summarised.
| epoch | loss | val task acc | val combined acc | lr |
|---|---|---|---|---|
| 1 | 2.276186 | 0.5814 | 0.1744 | 0.00099931 |
| 2 | 1.524010 | 0.7558 | 0.1977 | 0.00099726 |
| 3 | 0.852199 | 0.8721 | 0.1977 | 0.00099384 |
| 4 | 0.419326 | 0.9186 | 0.3372 | 0.00098907 |
| 5 | 0.219211 | 0.9302 | 0.6395 | 0.00098296 |
| 6 | 0.122340 | 0.9535 | 0.7558 | 0.00097553 |
| 7 | 0.074093 | 0.9535 | 0.7558 | 0.00096679 |
| 8 | 0.050349 | 0.9651 | 0.7907 | 0.00095677 |
| 9 | 0.037641 | 0.9651 | 0.7907 | 0.00094550 |
| 10 | 0.030423 | 0.9651 | 0.7558 | 0.00093301 |
| 20 | 0.008119 | 0.9651 | 0.8023 | 0.00075 |
| 30 | 0.004611 | 0.9651 | 0.8140 | 0.0005 |
| 40 | 0.003455 | 0.9651 | 0.8140 | 0.00025 |
| 50 | 0.002699 | 0.9651 | 0.8140 | 0.00006699 |
| 59 | 0.003170 | 0.9651 | 0.8140 | 0.00000069 |
| 60 | 0.003065 | 0.9651 | 0.8140 | 0.0 |
Five facts this table establishes.
- The task head converges by epoch 8. Val task accuracy reaches 0.9651 at epoch 8 and never
moves again β 53 further epochs at the same value. The strict
>best-state rule means the epoch-8 snapshot is the shipped model (Β§15.4). - The loss keeps falling while the metric is flat. Loss goes 0.0503 β 0.0031 across epochs 8β60, a 16Γ reduction, with zero val-task improvement. That is the signature of a model overfitting the training set's template variants while the group-held-out val set stays saturated β exactly what the group-split design is meant to expose.
val_combined_accuracydoes improve after epoch 8 β 0.7907 β 0.8140 by epoch 21 β because the binary heads keep improving even though the task head does not. Butbest_stateselects ontask_accuracyonly, so that later improvement is not in the shipped weights. The shipped adapter's val combined accuracy is therefore the epoch-8 value, 0.7907, which matchesmetadata.json.metrics.val.combined_accuracy.- The cosine schedule is exact.
lris exactly0.00075at epoch 20,0.0005at epoch 30,0.00025at epoch 40, and exactly0.0at epoch 60 β the analytic cosine values forT_max = 60, confirming no warm restarts and noeta_min. - The first epoch's loss (2.276) is consistent with the small-std init. With five heads
initialised at Ο = 0.02 (Β§6.3), the initial task cross-entropy is near
ln(6) = 1.79plus the modality and binary terms, and the weighted total starts above 2.0 β not saturated, not degenerate. The init is doing what its docstring claims.
33. The threshold sweep β and what "the test split was not run" means
artifacts/router/threshold_sweep_val.json is the release-level router artifact, and it is the
one README.md cites as the source of the router number. It is a sweep over 50 thresholds, and its
own metadata is unusually explicit about its limits. The note field, verbatim:
corpus-limited: val n=86 vs plan >=500. This is NOT a calibration -- the corpus is synthetic and too small (min per-class support 8, caption) and val carries 0 hard negatives (hn_* families are held out to TEST by design). Selecting a threshold here yields a justified default, not a calibrated value. The corpus was NOT padded with generated queries. Backlog P1-9's 'n=80' is the TEST split; the sweep target is val n=86. The test split was NOT touched.
33.1 The sweep's configuration
| Key | Value |
|---|---|
split |
val |
n_val |
86 |
n_val_examples_scored |
86 |
n_test_examples_scored |
0 |
test_split_touched |
false |
hard_negatives_in_val |
0 |
thresholds |
0.50 β¦ 0.99, step 0.01 β 50 rows |
select_by |
covered_accuracy |
shipped_threshold |
0.70 |
shipped_row |
{coverage: 0.848837, covered_task_accuracy: 0.972603, fallback_rate: 0.151163, n_covered: 73, threshold: 0.7} |
selected |
{coverage: 0.790698, covered_task_accuracy: 1.0, fallback_rate: 0.209302, n_covered: 68, threshold: 0.76} |
overall_ungated_accuracy |
0.965116 |
delta_vs_shipped |
{coverage: -0.0581, covered_task_accuracy: 0.0274} |
corpus_limited |
true |
corpus_total / corpus_groups |
576 / 54 |
plan_min_val_queries |
500 |
plan_min_hard_negatives |
100 |
val_min_support |
8 |
seconds |
0.206 |
adapter_config_hash |
615478910dc266bf |
config_hash |
78f1e3700da15aa1 |
environment |
{cuda_available: false, platform: "Windows-10-10.0.26200-SP0", python: "3.11.16", requested_device: "cpu", torch: "2.14.0+cpu"} |
33.2 The sweep's extremes, and what they show
| Threshold | n_covered |
coverage |
covered_task_accuracy |
fallback_rate |
|---|---|---|---|---|
| 0.50 (first row) | 84 | 0.976744 | 0.964286 | 0.023256 |
| 0.53 | 83 | 0.965116 | 0.963855 | 0.034884 |
| 0.55 | 81 | 0.94186 | 0.975309 | 0.05814 |
| 0.70 (shipped) | 73 | 0.848837 | 0.972603 | 0.151163 |
| 0.76 (selected) | 68 | 0.790698 | 1.000000 | 0.209302 |
| 0.99 (last row) | 8 | 0.093023 | 1.000000 | 0.906977 |
The monotone structure is the point. As the threshold rises, fewer queries clear the gate
(n_covered 84 β 8), coverage falls (0.977 β 0.093), the fallback rate rises (0.023 β 0.907), and
the covered task accuracy rises (0.964 β 1.000) β because the gate keeps only the queries the
model is sure about.
The shipped threshold (0.70) is not the sweep's optimum. The sweep's select_by: covered_accuracy would pick 0.76, at which all 68 covered queries are correct
(covered_task_accuracy: 1.000000). The delta_vs_shipped block quantifies the difference: 0.76
trades β0.0581 coverage for +0.0274 covered accuracy. The shipped 0.70 was kept anyway.
The reason is in the note: the val split has min per-class support 8 (caption) and 0 hard negatives, against a plan minimum of 500 val queries and 100 hard negatives. The sweep's own conclusion is that "selecting a threshold here yields a justified default, not a calibrated value." Choosing 0.76 on an 8-example-per-class validation set would be fitting to noise.
33.3 Reconciling "Gate 2 PASS (0.975)" with "the test split was NOT RUN"
These two statements appear to conflict and do not. The resolution is entirely in
threshold_sweep_val.json, which records both n_test_examples_scored: 0 and
test_split_touched: false while also recording split_sizes: {test: 80, train: 410, val: 86} β so
the sweep knew about the test split and deliberately did not score it.
| Statement | Source | What it is about |
|---|---|---|
test task accuracy 0.975, n=80, Gate 2 PASS |
metadata.json.metrics.test; docs/PHASE4_ROUTER_REPORT.md |
the training-time evaluation, produced by train_router's own evaluate_split(..., "test", ...) call |
test_split_touched: false, n_test_examples_scored: 0 |
threshold_sweep_val.json |
the release-level threshold sweep, which scored val only |
| router "overall ungated accuracy" 0.965116, val, n = 86, "TEST NOT RUN" | README.md metric table; DOCS_STYLE_GUIDE.md Β§3 |
the release-level claim, which quotes the val number and declines to promote the training-time test score |
The honest description, stated precisely:
- The router was evaluated on a test split, once, at training time, and the result is 0.975 with n = 80, all six classes measured. That number is in the shipped artifact.
- The release-level router evaluation β the threshold sweep that
README.mdcites as the router's source artifact β was run on val only. The test split was not touched by it. - The release therefore quotes 0.965116 (the val, ungated number) as the router's headline and
labels the test as
NOT RUN, because the release-level evaluation of the test split is the thing that does not exist. DOCS_STYLE_GUIDE.mdΒ§3 states the release rule as "0.965116 is validation, ungated, n = 86; the test split was NOT RUN", and this chapter obeys it.
What is not established: whether the release team considers the training-time test score
(0.975) a valid test-set result, or whether it is discounted because it is an artifact-internal
number. README.md limitation 2 says the test set "was never run", which is stricter than the
artifact supports. This chapter reports both facts and does not resolve the discrepancy β it is
recorded as an OPEN documentation question, because resolving it would require an owner
ruling, not an inference from the files.
One more detail from the sweep that is easy to miss. n_val_examples_scored is 86, and the
val task_counts sum to 86 (8 + 20 + 14 + 10 + 19 + 15 = 86) β so the sweep scored every val
example, and coverage is the fraction that cleared the gate rather than the fraction evaluated.
34. The router's standing caveats
docs/PHASE4_ROUTER_REPORT.md Β§"Standing caveats" is the phase's own list. It is reproduced here
because a reader of this chapter should not have to open a second file to find the limits.
| Caveat | Statement |
|---|---|
| The corpus is synthetic | 576 examples, 54 groups, hand-written and templated. Hard-negative accuracy 0.800 is the most honest generalisation number; the 0.975 headline is partly earned on templates the split kept in training. "Treat the router as working, not as benchmarked." |
caption recall is 0.846 |
2 of 13 missed. Not investigated; within noise at n=13 and below this corpus's significance floor. |
| The stub-encoder path exists only for tests | Any artifact it produces is marked encoder_type: "stub" and must never be deployed. |
| Threshold 0.70 is uncalibrated | It gates the lexical fallback, and both paths currently agree on every canonical query. "Calibration belongs in Phase 13 with real validation data, not here." |
One caveat the phase states and this chapter must repeat: the phase's F4-2 section says the
adapter has "51,725 parameters" while the router/train.py docstring says 50,822. Β§8 resolves the
arithmetic in favour of 51,725; the phase report itself already uses the correct figure.
Part H β The frontend router: interpret() vs chooseTask()
35. Two functions, two information sets
The frontend console runs its own router, and it is split in two. The file header states the whole design:
TWO DRIVERS, ONE EVENT SEAM
---------------------------
* LIVE (default when files are chosen): the browser uploads the user's own
imagery to `POST /api/assets`, receives asset IDs, posts them to
`POST /api/infer`, and feeds the REAL result into the same eight events.
Every value on screen then traces to the server's own response.
* PREVIEW (no files chosen): the deterministic router still runs so the
instrument is legible, but the specialist/result stages stay honestly empty
("awaiting backend") instead of pretending an analysis happened.
What is NEVER done in either mode: fabricating an answer, a confidence value,
an evidence record, a run id or a coordinate. If the backend is unreachable
the page says which step failed and shows the server's own message.
(frontend/assets/js/mission.js:1-18.)
The router is what makes the PREVIEW path legible. With no assets, no specialist can run, but the
page can still show what the question was understood to be β because interpret() is pure lexical
text processing with no model dependency.
The comment above interpret() states the contract:
Maps the question to an Intent-shaped reading, using only lexical rules.
This is the *interpretation* the page exposes β clearly a mock router,
never a measurement. Slots follow the real schema's Intent type.
(frontend/assets/js/mission.js:66-69.)
| Function | Line | Inputs | Blind to asset count? | Returns |
|---|---|---|---|---|
interpret(query) |
mission.js:70 |
the query text | yes | a reading: task, modality, temporal, spatial_output, evidence, source, language_output |
chooseTask(intent, query, assetCount) |
mission.js:184 |
the reading, the query, the asset count | no | {task, substituted, wanted, reason} |
The asymmetry is intentional and documented. docs/RESEARCH_NOTES.md Β§6 records the ruling:
For "What changed between the earlier and later image?" with one asset attached, the console reads
changewhile dispatch correctly falls back tochange_vqa. This is not a bug: the reading describes the question's intent, the dispatch respects what can actually be computed with the assets present. It is documented so it is not mistaken for a defect.
RESOLVED β and the reason it must be documented is that a reader who sees "read change, dispatched
change_vqa" in a trace will otherwise file it as a bug.
The server-side pair has the same shape (Β§39): IntentRouter.route() reads, PolicyPlanner.plan()
decides. The frontend pair and the server pair were written independently and converged on the same
two-stage split β which is itself evidence that the split is the right one.
36. interpret() in full
function interpret(query) {
var q = (query || '').toLowerCase();
var where = /where|locate|position|which part|bound|outline|coordinate/.test(q);
/* The change stem is matched WITHOUT a trailing \b: `\bchang\b` cannot
match "changed", "changes" or "changing", because there is no word
boundary between the stem and its inflection. With the boundary, the
page's own default question ("What changed here?") fell through to the
vqa branch β so the change path was unreachable from the UI that exists
to reach it. The other terms keep their boundaries; they are whole words. */
var changeStem = /chang/.test(q);
/* The temporal slot is `required` ONLY for a genuine change/pair marker.
Pair wording names two dates outright, so it can never be a location
question and is tested on its own. */
var pairWording = /\b(between|versus|vs\.?|pair|temporal|before|after|differ\w*|expand\w*|grow\w*|encroach\w*|lost|removed)\b/.test(q);
/* `new` is a place descriptor as often as a change marker: the repo ships
eo/new-airport.jpg, so "Where is the new airport?" is a real question.
It counts as a change ONLY when the query is not a `where` question.
`built` was dropped outright -- "built-up areas" is land-cover
vocabulary, not a change marker. While it sat in the temporal set, the
location question "Where are the built-up areas in this image?" was read
as a change request and the server answered it with the degenerate
one-word "River". Measured live, 2026-09-25. */
var newAsChange = /\bnew\b/.test(q) && !where;
var temporal = (changeStem || pairWording || newAsChange) ? 'required' : 'none';
var sar = /\bsar\b|radar|backscatter|insar/.test(q);
/* A CAPTION task is a request to DESCRIBE the whole scene, not to answer a
specific question about it. The lexical markers below are deliberately
narrow so a question like "what is the building" stays on the VQA branch:
caption fires only on explicit describe/caption/summarise wording or on the
"what is in / do you see" open forms. This is the path that was missing β
descriptive queries previously fell through to `vqa`, so the page never
asked the server for a caption even though `caption` is a valid task. */
var caption = /describe|caption|summari[sz]e|give me a (description|summary)|what (do|can) you see|what (is|are) (in|shown in|depicted in) (this|the) (image|scene|picture)|tell me about (this|the) (image|scene|picture)|what does this (image|scene|picture) (show|contain)/.test(q);
var task, modality, evidence, spatial;
if (sar) { task = 'optical_sar'; modality = 'optical_sar'; evidence = 'joint_feature_region'; spatial = where ? 'box' : 'scene'; }
else if (temporal === 'required') { task = 'change'; modality = 'optical'; evidence = 'change_region'; spatial = where ? 'box' : 'scene'; }
else if (where) { task = 'grounding'; modality = 'optical'; evidence = 'bounding_box'; spatial = 'box'; }
else if (caption) { task = 'caption'; modality = 'optical'; evidence = 'caption'; spatial = 'scene'; }
else { task = 'vqa'; modality = 'optical'; evidence = 'statistic'; spatial = 'scene'; }
return {
task: task, modality: modality, temporal: temporal,
spatial_output: spatial, evidence: evidence,
source: 'lexical_fallback', language_output: 'en'
};
}
(frontend/assets/js/mission.js:70-114.)
36.1 The four lexical predicates
| Variable | Pattern | Kind |
|---|---|---|
where |
`/where | locate |
changeStem |
/chang/ |
stem, no trailing \b |
pairWording |
`/\b(between | versus |
newAsChange |
/\bnew\b/ AND NOT where |
word-bounded, gated |
sar |
`/\bsar\b | radar |
caption |
the long alternation above | unanchored, multi-form |
where is unanchored β "where" matches inside "somewhere", and "bound" matches inside
"boundary". This is the same substring-matching limitation as the Python fallback (Β§26), and it is
the source of the residuals in Β§48.
36.2 The dispatch chain β a strict if/else ladder
The five branches are evaluated in order, and the first match wins:
| # | Condition | task | modality | evidence | spatial |
|---|---|---|---|---|---|
| 1 | sar |
optical_sar |
optical_sar |
joint_feature_region |
box if where else scene |
| 2 | temporal === 'required' |
change |
optical |
change_region |
box if where else scene |
| 3 | where |
grounding |
optical |
bounding_box |
box |
| 4 | caption |
caption |
optical |
caption |
scene |
| 5 | otherwise | vqa |
optical |
statistic |
scene |
The precedence mirrors the Python fallback's β dual-modality (sar) first, then temporal, then
spatial, then caption, then VQA β with one difference: the frontend has no SAR-only or
optical-only rule, so sar alone (a single modality word) sends the request to optical_sar
rather than to a SAR-modality VQA. That is a genuine behavioural difference between the frontend
router and the server fallback, and it is the reason a bare "radar" in a query reads as a fusion
request in the console. It is IMPLEMENTED as written; whether it is intended is not recorded, and
this chapter does not assert either way.
The evidence and spatial slots are the frontend's own vocabulary, not the server's
EvidenceType enum. 'bounding_box' and 'change_region' and 'statistic' are display strings for
the console's reading panel; the server's EvidenceType uses bounding_box, change_map and
statistic (core/schemas.py:65-76). So 'change_region' is not a server evidence type β the
console's reading is a display projection, and the server's evidence is authoritative.
language_output: 'en' β a string, not a boolean, where the server's Intent.language_output is
a bool. The reading is an Intent-shaped object, not an Intent; the comment says "Slots
follow the real schema's Intent type", meaning the slot names, not the types. source: 'lexical_fallback' does match the server's literal (core/schemas.py:105).
37. chooseTask() in full
The comment block above chooseTask is the clearest statement anywhere of why dispatch is
asset-aware:
THE FRONTEND MUST SPEAK THE SERVER'S VOCABULARY.
`core.schemas.Task` accepts exactly: vqa, caption, grounding, change,
optical_sar, change_vqa, unsupported. Any other string is a 422 from
`AnalysisRequest`, because the model is `extra="forbid"` and `force_task`
is a `Task | None`. The legacy prototype names (SAR_ANALYSIS,
CHANGE_ANALYSIS, VLM_QA, VLM_CAPTION) are NOT valid and would fail.
So the router's reading is mapped to the server enum here, in one place.
`change` -> `change_vqa` when the question is quantificational, because the
server's `change` returns a spatial change map with NO language output while
`change_vqa` returns a short answer -- and the page's Answer block promises
an answer. Sending `change` for "how much changed" would fill the map and
leave the answer empty, which looks like a bug and is really a wrong task.
PAIR-AWARE, BECAUSE THE DEPLOYMENT SAYS SO
------------------------------------------
`/api/capabilities` declares `requires_pair` and `max_assets` per task, and
the server enforces them: with one asset, `change` answers
`invalid_request` ("change requires exactly 2 assets (T1 and T2); got 1")
and the whole envelope comes back `degraded: true`. Measured live, 2026-09-25.
So the task cannot be chosen from the question alone -- it depends on how
many files were actually selected. The page's OWN default question ("What
changed here?") reads as a change request, and a user who selects one image
and presses Run would get a degraded non-answer for asking a reasonable
thing. That is a wiring defect, not a user error.
The rule below therefore: request a pair-requiring task ONLY when a second
asset exists; otherwise fall back to a task that accepts one asset, and
report the substitution rather than hiding it.
(frontend/assets/js/mission.js:116-147.)
37.1 The three constants
var ROUTER_TASK_TO_SERVER = {
vqa: 'vqa',
caption: 'caption',
grounding: 'grounding',
change: 'change',
optical_sar: 'optical_sar',
change_vqa: 'change_vqa'
};
/*: Tasks the deployment reports as `requires_pair: true`. Mirrors
`capabilities_payload()`; the live capabilities are fetched at boot and
override this when available, so a server-side change is picked up rather
than hardcoded wrong. */
var PAIRED_TASKS = { change: true, change_vqa: true, optical_sar: true };
/*: What to ask for when a pair-requiring task was chosen but only one asset
is available. `vqa` accepts a single asset and answers a question about it,
which is the honest closest behaviour available for one image. */
var SINGLE_ASSET_FALLBACK = 'vqa';
(frontend/assets/js/mission.js:148-166.)
PAIRED_TASKS is a fallback for the live capability contract, not the source of truth. The
comment says the live capabilities are fetched at boot and override it:
var pairRequired = {}; // task -> bool, from /api/capabilities when reachable
var singleAssetTasks = {}; // task -> bool
function requiresPair(task) {
if (Object.prototype.hasOwnProperty.call(pairRequired, task)) return pairRequired[task];
return !!PAIRED_TASKS[task];
}
(frontend/assets/js/mission.js:168-174.)
So a server-side change to requires_pair is picked up at boot, and the hardcoded table is only
the offline default. That is a genuinely careful piece of design: the frontend does not hardcode
the deployment's contract as fact, it hardcodes it as a fallback.
SINGLE_ASSET_FALLBACK = 'vqa' β and the comment states the reason it is vqa and not
caption: vqa "accepts a single asset and answers a question about it, which is the honest
closest behaviour available for one image." A caption would also accept one asset, but a caption is
not an answer to a question.
37.2 The function
function chooseTask(intent, query, assetCount) {
var wanted = ROUTER_TASK_TO_SERVER[intent.task] || 'caption';
if (wanted === 'change') {
var q = (query || '').toLowerCase();
/* `\barea\b`, not bare `area`: without the boundary the substring matched
inside "areas", so "Where are the built-up areas ..." (already a
mis-read change question) was upgraded again to change_vqa. The
boundary keeps the quantifier reading for a real "how much area
changed" while refusing the plural land-cover noun. */
if (/\bhow (much|many|large)\b|\bhas\b|\bdid\b|percent|\barea\b|quantif/.test(q)) {
wanted = 'change_vqa';
}
}
if (assetCount < 2 && requiresPair(wanted)) {
/* No pair available: ask for a single-asset task instead, and name the
substitution so the UI can show it. Silently sending a pair-requiring
task would produce a degraded envelope whose reason ("requires exactly
2 assets") the user never sees. */
return {
task: SINGLE_ASSET_FALLBACK,
substituted: true,
wanted: wanted,
reason: wanted + ' needs two images (T0 and T1); only one was provided'
};
}
return { task: wanted, substituted: false, wanted: wanted, reason: '' };
}
(frontend/assets/js/mission.js:184-213.)
Three steps, in order:
- Map the reading to the server enum.
ROUTER_TASK_TO_SERVER[intent.task] || 'caption'β the|| 'caption'is the default for an unmapped reading. - The quantifier upgrade. Only for
change: a quantificational question becomeschange_vqa. The upgrade predicate is/\bhow (much|many|large)\b|\bhas\b|\bdid\b|percent|\barea\b|quantif/. - The pair guard. If the wanted task requires a pair and fewer than two assets exist, substitute
vqaand name the substitution.
The pair guard is checked after the quantifier upgrade, so a quantificational change question
with one asset goes change β change_vqa β vqa. That is the path recorded in the live run
run_62ca98d510be β wait, no: that run dispatched change_vqa with a pair present. With one asset
it would be vqa. Both are correct under the rule.
The upgrade predicate, term by term:
| Term | Matches | Why |
|---|---|---|
\bhow (much|many|large)\b |
"how much", "how many", "how large" | quantity questions |
\bhas\b |
"has the coastline advanced" | present-perfect change questions |
\bdid\b |
"did the area grow" | past-tense change questions |
percent |
"what percent changed" | ratio questions |
\barea\b |
"how much area changed" | word-bounded, so "areas" does not match |
quantif |
"quantify", "quantification" | stem |
\barea\b versus bare area is one of the four defect fixes (Β§46), and the comment states the
failure precisely: without the boundary, the substring matched inside "areas", so
"Where are the built-up areas ..." β already a mis-read change question β was upgraded again
to change_vqa. Two wrongs compounding: the reading was wrong, and the dispatch escalated the wrong
reading.
The returned object's four fields:
| Field | Meaning |
|---|---|
task |
the task actually dispatched |
substituted |
true when the pair guard fired |
wanted |
what the reading asked for, before the guard |
reason |
a human-readable explanation, empty when no substitution |
substituted and wanted are what make the substitution visible rather than silent. The
comment is explicit: "Silently sending a pair-requiring task would produce a degraded envelope whose
reason β¦ the user never sees." This is the same "absence must not be indistinguishable from a
non-event" principle the planner applies in Python (core/planner.py:454, Β§42).
37.3 serverTaskFor() β the test shim
/* Back-compat shim for the unit tests that assert the pure question->task
mapping. Pair-awareness lives in `chooseTask`; this is the no-pair-limit
view of the same rule, exercised with a 2-asset assumption. */
function serverTaskFor(intent, query) {
return chooseTask(intent, query, 2).task;
}
(frontend/assets/js/mission.js:215-220.)
A pure questionβtask mapping, obtained by assuming a pair. This is how the unit tests assert the upgrade rule (change β change_vqa) independently of the pair rule. Two behaviours, two entry points, one implementation.
38. assetsForTask() and validateOpticalSar()
38.1 assetsForTask() β only the files the task consumes
function assetsForTask(task, t1, t0) {
if (task === 'vqa' || task === 'grounding' || task === 'caption') {
/* Single-image tasks: the T0 frame is NOT part of the request. */
return [t1];
}
/* change / change_vqa / optical_sar all consume the pair. */
var pair = [t1];
if (t0) pair.push(t0);
return pair;
}
(frontend/assets/js/mission.js:245-254.)
This is the third place the asset contract is enforced β after PAIRED_TASKS and
/api/capabilities β and its comment records the live failure that motivated it:
Sending the optional T0 frame to a single-image task makes the backend reject the whole request with
invalid_request("... requires exactly 2 assets ...; got 1") β measured live, 2026-09-25, when a pair was uploaded and a VQA question asked. Isolating the file set here is the fix: the pair is only ever sent to the tasks that declared it.
(frontend/assets/js/mission.js:229-233.)
The failure mode is worth naming: the request was rejected because it sent too many assets.
vqa declares max_assets: 1, so uploading a pair and asking a VQA question produced
invalid_request. The fix is not to drop the extra file silently but to send only the files the
dispatched task consumes β the single-image tasks get [t1] and the pair tasks get [t1, t0].
The optical-SAR slot semantics are stated in the comment:
For
optical_sarthe pair is read as optical (T1) + SAR/radar (T0) β the same two slots, a different meaning, and the input validator (validateOpticalSar) warns when they look wrong.
(frontend/assets/js/mission.js:236-238.)
So t1 and t0 carry two different meanings depending on the task: for change they are two
dates of the same sensor; for optical_sar they are two modalities. The slots are reused, and the
ambiguity is resolved by the task.
38.2 validateOpticalSar() β an advisory client-side check
function validateOpticalSar(t1, t0) {
if (!t1 || !t0) {
return {
level: 'error',
message: 'Optical-SAR fusion needs two images: an optical scene (T1) and a SAR/radar image (T0). Add the radar image before running.'
};
}
var raster = { jpg: 1, jpeg: 1, png: 1 };
if (raster[_extOf(t0)] && raster[_extOf(t1)]) {
/* Both files are single-band photos β almost certainly NOT an optical +
SAR product. The model expects a real radar backscatter (usually a
GeoTIFF). We cannot prove the failure, but the pattern is wrong enough
to warn rather than let the round-trip fail opaquely. */
return {
level: 'warn',
message: 'Both files look like plain photos (JPEG/PNG). Optical-SAR fusion pairs an optical image with a radar/SAR backscatter β usually a GeoTIFF. The model may reject this pair.'
};
}
return { level: 'ok', message: '' };
}
(frontend/assets/js/mission.js:273-292.)
The docstring above it states exactly what it is and is not:
Best-effort client-side check that an optical-SAR request has the right SHAPE of inputs. The browser cannot read sensor modality or band count from a File, so this is advisory, not a measurement: it flags the obviously-wrong case (two plain photos masquerading as an optical+SAR pair) so the user is warned BEFORE the backend returns
invalid_request, and it names the missing-input case outright.
(frontend/assets/js/mission.js:263-269.)
| Level | Condition | Meaning |
|---|---|---|
error |
either file missing | the request cannot be formed |
warn |
both files are .jpg / .jpeg / .png |
the pair is probably not an optical+SAR product |
ok |
otherwise | no client-side objection |
The three levels are honest about their epistemics. error is a fact the client can establish
(a file is absent). warn is a heuristic β the comment says "We cannot prove the failure, but
the pattern is wrong enough to warn rather than let the round-trip fail opaquely." ok means "no
objection", not "valid". The function never claims to validate the pair; it claims to catch the
obvious case before an opaque round-trip failure.
Note the asymmetry: .jpg/.jpeg/.png produce a warning, but so would any non-GeoTIFF. A
.tif file gets ok regardless of whether it is actually SAR. The check is a shape check on the
extension, and its docstring says so.
Part I β The server-side decision: PolicyPlanner
39. Why the planner exists at all
core/planner.py opens with the reason, and it is the same reason interpret() and chooseTask()
are separate functions:
A design where
intent.taskselects a specialist in one step has collapsed understand into decide. Three things break when that happens:
- The router can never be overruled, downgraded, or widened. A confident-but-wrong prediction becomes unappealable.
- A query needing two specialists can never get both. The router returns ONE task; "what changed between these two images, and describe the scene" routes to
changealone, and the caption is lost. Section Β§3.5 is what recovers it.- There is no record of the decision. A plan step that carries a
reasonnaming the rule that produced it is auditable; anargmaxover a softmax is not a policy and cannot be inspected or tested.
(core/planner.py:23-34.)
Three failure modes, each with a mechanism that prevents it. Point 1 is what the closed refusal
list (Β§41) and the widening rules (Β§42) provide. Point 2 is the widening rules specifically. Point 3
is PlanStep.reason, which is "a machine-generated RULE IDENTIFIER from a closed set, not a
narrative" (core/planner.py:174-176) β the reason a free-text field would be a defect:
A free-text field here would become a place to put chain-of-thought, and eventually would contain some (design Β§6).
39.1 The planner is pure, and the purity is load-bearing
if TYPE_CHECKING: # pragma: no cover - typing only, never executed
# Imported for annotation only. `router.classifier` imports torch at module
# scope, and this module must stay importable and testable without torch β
# the planner is pure, and design Β§9.3 leans on that ("the planner is pure
# ... no models, no torch, no filesystem"). The only surface the planner
# actually uses is `prediction.intent` and `prediction.above_threshold`,
# so the dependency is structural and need not be a runtime import.
from router.classifier import RouterPrediction
(core/planner.py:93-100.)
This is a real engineering constraint, not a style preference. router.classifier imports torch
at module scope; importing it for real would make the planner un-importable in a torch-free
environment. Because the planner touches only prediction.intent and
prediction.above_threshold, the dependency is structural and the import can be annotation-only.
The class docstring restates the contract:
Pure. Takes dataclasses, returns a dataclass. No models, no torch, no filesystem, no clock β every rule below is unit-testable with a hand-built
RouterPrediction.
(core/planner.py:255-258.)
39.2 The two frozen constants
#: Multiplier the planner applies to its *reading* of a lexical-fallback
#: confidence. Frozen by design Β§2.4. It exists so a matched keyword cannot
#: outrank a learned model when the planner decides whether to widen a plan.
#: The router's own reported confidence is never altered.
LEXICAL_FALLBACK_DISCOUNT: float = 0.75
#: Above this effective confidence the planner treats the route as settled and
#: does not add the discretionary explanation step. Below it, and when the
#: request also asks for language output, a `vlm` step is added so the run
#: explains itself. Also frozen by Β§2.3.
SETTLED_CONFIDENCE: float = 0.60
(core/planner.py:106-116.)
SETTLED_CONFIDENCE is defined but not referenced in plan(). The explanation step is added
when not prediction.above_threshold and prediction.intent.language_output (Β§40.4) β the gate is
above_threshold, the router's 0.70 threshold, not SETTLED_CONFIDENCE. So the constant documents a
design intent (Β§2.3) that the implementation expresses through above_threshold instead. This is
IMPLEMENTED with a constant that has no reader; it is named here because a reader will search for
its use and not find one.
39.3 The capability maps
TASK_CAPABILITY: Mapping[Task, str] = {
Task.VQA: "vqa",
Task.CAPTION: "caption",
Task.GROUNDING: "grounding",
Task.CHANGE: "change",
Task.OPTICAL_SAR: "optical_sar",
Task.CHANGE_VQA: "change_vqa",
}
CAPABILITY_ASSETS: Mapping[str, int] = {
"vqa": 1,
"caption": 1,
"grounding": 1,
"change": 2,
"optical_sar": 2,
# Two assets, like `change` β the difference is the output, not the input:
# a short language answer rather than a change map.
"change_vqa": 2,
}
(core/planner.py:118-142.)
TASK_CAPABILITY maps six Task members β note it maps Task.CHANGE_VQA, so a forced
change_vqa is a first-class capability. Task.UNSUPPORTED is deliberately absent, because it is a
refusal, not a capability.
CAPABILITY_ASSETS mirrors the specialists' own validate_request, and the docstring names the
relationship:
Mirrors the specialists' own
validate_request, which stays authoritative; this is the planner's cheap precondition so it can refuse before construction is attempted.
(core/planner.py:130-132.)
The authority is the specialist, not the planner. CAPABILITY_ASSETS exists so the planner can
refuse cheaply; it does not replace the specialist's check, and the _indices_for helper (Β§40.5)
deliberately does not duplicate the pairing logic:
A two-asset capability takes the first two in request order; a one-asset capability takes the first. Deliberately simple: the specialists validate their own asset count and pairing rules, and duplicating that logic here would give two places to disagree.
(core/planner.py:530-534.)
40. plan() β the single decision point
def plan(
self,
prediction: "RouterPrediction",
request: AnalysisRequest,
*,
assets: list[AssetMetadata] | None = None,
) -> ExecutionPlan:
(core/planner.py:276-282.)
assets is accepted and never read for policy. The docstring says so: "Not read for policy β
only the COUNT matters here, and that comes from request.assets. Accepted so the controller can
pass its already-resolved list without a second conversion."
40.1 The force_task bypass
asset_count = len(request.assets)
source = prediction.intent.source
# --- force_task bypasses the router entirely (Β§8) -----------------
# `Intent(source="forced")` already exists in the schema. When the
# caller forces a task, the router's opinion is not consulted at all,
# and that fact is recorded rather than hidden.
forced = request.force_task
task = forced if forced is not None else prediction.intent.task
if forced is not None:
source = "forced"
(core/planner.py:296-306.)
The router is not consulted when a task is forced, and the source is recorded as "forced" β
which is the third literal of Intent.source (core/schemas.py:105:
Literal["learned", "lexical_fallback", "forced"]). A forced task still goes through the refusal
check and the widening rules, so force_task=Task.CHANGE with one asset still refuses (Β§41).
40.2 The refusal check, first
refusal = self._refusal_for(task, asset_count)
if refusal is not None:
return ExecutionPlan(
steps=(),
refused=True,
refusal=refusal,
uncertain=not prediction.above_threshold,
router_source=source,
effective_confidence=self._effective_confidence(prediction),
notes=(f"refused:{refusal.reason}",),
)
(core/planner.py:309-319.)
A refusal is returned, never raised. The module docstring states the contract:
Refusing is not an error. It returns a valid plan with
refused=Trueand a typedPlanRefusal, so the controller can assemble a normal envelope carrying the reason.
(core/planner.py:75-79.)
The refusal's uncertain and effective_confidence are still populated, so a refusal carries the
same router-provenance information a plan does.
40.3 The primary step
primary = TASK_CAPABILITY[task]
self._append(
steps,
capability=primary,
indices=self._indices_for(primary, asset_count),
reason=f"task:{task.value}",
required=True,
params=self._params_for(primary, task),
)
(core/planner.py:325-333.)
The primary step is always required=True β a failure in it degrades the whole run, unlike the
widening steps which are required=False. The reason is task:<value>, e.g. "task:change" β
a rule identifier from the closed set, as the dataclass requires.
_params_for exists because the VLM specialist branches on it:
@staticmethod
def _params_for(capability: str, task: Task) -> dict[str, Any]:
"""Specialist kwargs a step must carry.
The VLM specialist branches on `params["task"]` to decide between VQA
and captioning, so a `vlm`-backed step without it is unrunnable. The
mapping is by CAPABILITY, not by specialist, so `caption` passes
`"task": "caption"` even though the same object serves both.
"""
if capability in ("vqa", "caption"):
return {"task": capability}
return {}
(core/planner.py:514-525.) Only vqa and caption carry params, and both carry {"task": <cap>}
β because one specialist object serves both capabilities and needs to be told which.
40.4 The discretionary explanation step
if (
not prediction.above_threshold
and prediction.intent.language_output
and "vqa" not in {s.capability for s in steps}
and self._capability_known("vqa")
):
self._append(
steps,
capability="vqa",
indices=(0,),
reason="uncertain_route:explain",
required=False,
)
notes.append("uncertain route: added an explanation step")
(core/planner.py:339-352.)
Four conditions, all required. The route must be uncertain; the request must want language; no
vqa step may already be planned (so a VQA request does not get a second VQA step); and the vqa
capability must be declared. The step is required=False and reads asset index (0,).
Note the ordering difference from the widening steps: this check uses _capability_known
at plan time, whereas _widening_steps deliberately does not (Β§42.0). The comment above
_widening_steps explains the principle, and this block is a case where the availability gate is
applied inline β so the explanation step is never added and then dropped; it is simply not added.
Both approaches avoid the "silent omission" failure, because this one records
"uncertain route: added an explanation step" only when it fires and the drop path records its own
note when it drops.
40.5 _indices_for and _append
@staticmethod
def _indices_for(capability: str, asset_count: int) -> tuple[int, ...]:
needed = CAPABILITY_ASSETS.get(capability, 1)
return tuple(range(min(needed, asset_count)))
(core/planner.py:527-537.) range(min(needed, asset_count)) β so a two-asset capability on a
one-asset request yields (0,), and the specialist's own validation is what ultimately refuses.
@staticmethod
def _append(
steps: list[PlanStep],
*,
capability: str,
indices: tuple[int, ...],
reason: str,
required: bool,
params: Mapping[str, Any] | None = None,
) -> None:
steps.append(
PlanStep(
# Provisional id; `_renumber` assigns the canonical one after
# widening, availability filtering and any truncation, so the
# ids always match the final order.
step_id=f"step_{len(steps) + 1:03d}",
capability=capability,
specialist=capability,
asset_indices=indices,
params=params or {},
reason=reason,
required=required,
)
)
(core/planner.py:539-562.)
step_id is provisional β _renumber reassigns after widening, filtering and truncation. This
is why a truncated plan's step ids are still step_001..step_N with no gaps. specialist is set
equal to capability β the docstring on PlanStep notes they are kept distinct "because the VQA
specialist answers to "vqa" AND "caption" while its own name is "vlm""
(core/planner.py:167-171); today the planner sets them equal and the registry resolves the rest.
40.6 The availability gate and the second refusal
steps, avail_notes = self._drop_unavailable(steps)
notes.extend(avail_notes)
if not steps:
return ExecutionPlan(
steps=(),
refused=True,
refusal=self._refusal(
"model_unavailable",
"no_available_specialist",
"No specialist is available for this request in this "
"environment.",
),
uncertain=not prediction.above_threshold,
router_source=source,
effective_confidence=self._effective_confidence(prediction),
notes=tuple(notes),
)
(core/planner.py:354-372.)
A plan that loses every step to availability becomes a refusal, with code
"model_unavailable" and reason "no_available_specialist". This is the second way a refusal
arises, and it is distinct from the closed list of Β§41 β which is why the reason string differs.
def _drop_unavailable(
self, steps: list[PlanStep]
) -> tuple[list[PlanStep], list[str]]:
"""Remove steps whose capability the registry does not declare.
...
Note this filters only on *declaration*. Whether a declared capability
can actually be CONSTRUCTED is discovered by the controller, which
records an `UNAVAILABLE` registry entry β the planner must not attempt
construction, which would defeat the lazy-load design.
"""
known = {c for c in self.registry.available()} if self._registry_ok() else None
if known is None:
return steps, []
kept: list[PlanStep] = []
notes: list[str] = []
for step in steps:
if step.capability in known:
kept.append(step)
else:
notes.append(
f"dropped step {step.step_id} ({step.capability}): "
f"capability not registered"
)
return kept, notes
(core/planner.py:572-602.)
Three design points, each stated in the source:
- The planner filters on declaration only. Whether a declared capability can be constructed
is the controller's discovery, which records an
UNAVAILABLEregistry entry. The planner must not attempt construction, "which would defeat the lazy-load design." - A broken registry is not a plan error.
_registry_ok()catches any exception and returnsFalse, in which case no filtering happens (known is Noneβ return steps unchanged). A registry that cannot be queried degrades to "plan everything", not "plan nothing". - Every drop is recorded in
notes, so "not planned because it cannot run" stays distinguishable from "not planned because the planner chose not to" (design Β§4.3).
40.7 Truncation, renumbering, and the mode
if self.max_steps is not None and len(steps) > self.max_steps:
dropped = [s.capability for s in steps[self.max_steps:]]
notes.append(
f"plan truncated at {self.max_steps} steps; dropped {dropped}"
)
steps = steps[: self.max_steps]
steps = self._renumber(steps)
(core/planner.py:375-382.)
Truncation is recorded with the names of the dropped capabilities β never silent. The cap
mirrors agent.max_specialists, per the constructor docstring.
@staticmethod
def _mode_for(steps: list[PlanStep]) -> PlanMode:
"""`PARALLEL_SAFE` only when the asset partitions are demonstrably safe.
Requires at least two steps, pairwise-disjoint asset indices, and no
step declaring a resource constraint. With today's four specialists
asset indices commonly overlap β a change plan and a caption plan can
both read asset 1 β so this legitimately returns SEQUENTIAL in normal
operation. That is correct, not a limitation (Β§3.4).
"""
if len(steps) < 2:
return PlanMode.SEQUENTIAL
seen: set[int] = set()
for step in steps:
indices = set(step.asset_indices)
if indices & seen:
return PlanMode.SEQUENTIAL
seen |= indices
return PlanMode.PARALLEL_SAFE
(core/planner.py:613-632.)
PARALLEL_SAFE is a declaration, not a concurrency directive. PlanMode's docstring says so:
PARALLEL_SAFEis a declaration that the steps share no mutable state, not a concurrency directive. v1 executes both modes sequentially β freeze section 5 forbids worker pools, queues and async frameworks. The marker records that a futureThreadPoolExecutorwould be sound.
(core/planner.py:149-155.)
In normal operation this returns SEQUENTIAL, because a change + caption plan has both steps
reading asset 1 (_indices_for("change", 2) = (0, 1) and
_indices_for("caption", 2) = (0,) β index 0 in both). That overlap is correct, not a defect: the
two steps genuinely both read the same image.
41. The closed refusal list
def _refusal_for(self, task: Task, asset_count: int) -> PlanRefusal | None:
"""The closed refusal list from Β§8. Order matters: query first."""
if task is Task.UNSUPPORTED:
return self._refusal(
UnsupportedQueryError.code,
"task_unsupported",
UnsupportedQueryError.user_message,
)
if asset_count < 1:
return self._refusal(
InvalidRequestError.code,
"zero_assets",
InvalidRequestError.user_message,
)
return None
(core/planner.py:412-426.)
Two conditions, checked in a fixed order. The docstring says "Order matters: query first" β
an unsupported task with zero assets reports task_unsupported, because the query is the more
fundamental problem.
| Order | Condition | Code | Reason | User message source |
|---|---|---|---|---|
| 1 | task is Task.UNSUPPORTED |
UnsupportedQueryError.code |
"task_unsupported" |
UnsupportedQueryError.user_message |
| 2 | asset_count < 1 |
InvalidRequestError.code |
"zero_assets" |
InvalidRequestError.user_message |
Both codes and user messages are imported from core.errors, not written inline β so the
refusal's client-facing text and its error code come from the same definitions the exception classes
use. The module docstring's claim that the list is closed is what makes it testable:
The refusal conditions are a closed list (Β§8) β no free-form logic, because a refusal rule invented at the call site is a policy that cannot be tested.
(core/planner.py:80-81.)
Note what the list does not contain. There is no refusal for "asset count too high for this
task" and no refusal for "asset count wrong for this specific capability". Those are the
specialists' validate_request responsibilities (Β§39.3) β so a change request with one asset
reaches the specialist and is refused there, not by the planner. That is the division the comment on
CAPABILITY_ASSETS states: the specialist stays authoritative.
_refusal() is a thin constructor:
@staticmethod
def _refusal(code: str, reason: str, user_message: str) -> PlanRefusal:
return PlanRefusal(code=code, reason=reason, user_message=user_message)
(core/planner.py:428-430.)
42. The Β§3.5 widening rules β one task, several steps
def _widening_steps(
self,
prediction: "RouterPrediction",
task: Task,
asset_count: int,
notes: list[str],
) -> list[PlanStep]:
"""The Β§3.5 rules that turn ONE router task into SEVERAL steps.
These are the reason the planner exists rather than a switch on
`intent.task`. The router's binary heads describe *aspects* of the
request that a single task label cannot carry.
"""
extra: list[PlanStep] = []
intent = prediction.intent
(core/planner.py:434-448.)
This is the function that justifies the whole two-stage design. The router returns one task; the binary heads describe aspects; the widening rules are what recover the aspects the task label cannot carry.
42.0 Why availability is not gated here
# Availability is NOT gated here. The step is added whenever its rule
# fires, and `_drop_unavailable` removes it with a recorded note if the
# capability is unregistered. Gating here as well would create a second,
# silent availability check whose omission leaves no trace β exactly the
# "absence indistinguishable from a non-event" failure that Β§4.3 forbids.
(core/planner.py:450-454.)
This is a deliberate choice, and it contradicts the pattern in Β§40.4. The widening rules add
unconditionally; the explanation step checks availability inline. The comment's reasoning is sound
for the widening rules: a gate here plus _drop_unavailable would be two checks, and the inline one
would leave no trace when it silently declined. Recorded here as an internal inconsistency in
approach β not in outcome, because both paths produce a notes entry.
42.1 Rule one β change + language_output
if task is Task.CHANGE and asset_count >= 2 and intent.language_output:
if self._capability_known("change_vqa"):
if not self._capability_planned(extra, "change_vqa"):
self._append(
extra,
capability="change_vqa",
indices=self._indices_for("change_vqa", asset_count),
reason="task:change+language_output:answer",
required=False,
)
notes.append(
"change + language request: added a change-VQA step"
)
elif not self._capability_planned(extra, "caption"):
self._append(
extra,
capability="caption",
indices=(asset_count - 1,),
reason="task:change+language_output",
required=False,
)
notes.append("change + language request: added a caption step")
(core/planner.py:470-491.)
Three conditions: the task is change; there are at least two assets (because change needs a
pair, and so does change_vqa); and the language head fired.
The two branches are alternatives, not a sequence β and the comment explains why:
WHICH language output, though, depends on what the deployment has. With a change-VQA capability registered (R-02), the request is satisfied by an ANSWER to the change question β which is the specific thing that was asked. Without it, the fallback is to caption the later acquisition, which is the one a "what changed" answer describes.
These are alternatives, not two things to do: captioning the post image is not a second requirement, it is the best available substitute when the answer capability is absent. Adding both would spend a step on a strictly weaker output.
(core/planner.py:460-469.)
Two reasons, two rule ids, and a different asset index for the fallback:
| Branch | Reason id | Asset indices | Meaning |
|---|---|---|---|
change_vqa registered |
task:change+language_output:answer |
(0, 1) |
answer the change question |
change_vqa absent |
task:change+language_output |
(asset_count - 1,) |
caption the later acquisition |
indices=(asset_count - 1,) is the subtle one. The caption step reads the last asset β the
post-change image β because that is the acquisition a "what changed" description describes. With two
assets, that is index 1.
42.2 Rule two β spatial_output + language_output
# A request that wants BOTH coordinates and prose on one asset gets
# grounding and vqa. The router can only return one task, so without
# this rule the second aspect is silently lost.
if intent.spatial_output and intent.language_output and asset_count >= 1:
if task is not Task.GROUNDING:
if not self._capability_planned(extra, "grounding"):
self._append(
extra,
capability="grounding",
indices=(0,),
reason="aspect:spatial_output",
required=False,
)
notes.append("spatial+language request: added a grounding step")
(core/planner.py:493-507.)
The task is not Task.GROUNDING guard prevents adding a grounding step when grounding is
already the primary step. The step reads asset (0,) and is required=False.
_capability_planned is the de-duplication helper:
@staticmethod
def _capability_planned(steps: list[PlanStep], capability: str) -> bool:
return any(step.capability == capability for step in steps)
(core/planner.py:510-512.) Note it checks the extra list, not the full step list β so a
primary step with the same capability does not suppress a widening step. The task is not Task.GROUNDING guard is what prevents the duplicate in the one case that matters.
42.3 The widening rules, tabulated
| Rule | Fires when | Adds | Reason id | Indices | Required |
|---|---|---|---|---|---|
| 1a | task=change β§ asset_count β₯ 2 β§ language_output β§ change_vqa known |
change_vqa |
task:change+language_output:answer |
(0, 1) |
F |
| 1b | same, but change_vqa not known β§ caption not already planned |
caption |
task:change+language_output |
(asset_count-1,) |
F |
| 2 | spatial_output β§ language_output β§ asset_count β₯ 1 β§ task β grounding |
grounding |
aspect:spatial_output |
(0,) |
F |
Worked example β the query the module docstring names. "what changed between these two images,
and describe the scene" with two assets, a change reading and language_output=True:
| Step | Capability | Reason | Required | Indices |
|---|---|---|---|---|
step_001 |
change |
task:change |
T | (0, 1) |
step_002 |
change_vqa |
task:change+language_output:answer |
F | (0, 1) |
Mode: SEQUENTIAL β both steps read asset 0, so the partitions overlap. Notes:
("change + language request: added a change-VQA step",).
Without the widening rule, step 2 would not exist and the caption/answer aspect would be lost β which is precisely failure mode 2 in the module docstring.
43. The provenance discount
@staticmethod
def _effective_confidence(prediction: "RouterPrediction") -> float:
"""The planner's reading of the router's confidence.
Applies the provenance discount for a lexical fallback (Β§2.4). The
router's own `Intent.confidence` is never modified β this is a separate
number used only for planning.
"""
raw = float(prediction.intent.confidence)
if prediction.intent.source == "lexical_fallback":
return raw * LEXICAL_FALLBACK_DISCOUNT
return raw
(core/planner.py:397-408.)
The discount is 0.75, applied to a lexical-fallback confidence only. The module docstring states the reasoning in full:
A lexical fallback at 0.9 is not the same evidence as a learned model at 0.9: one is a regex that matched, the other is a learned distribution. Treating them identically would let a matched keyword outrank the model it fell back from. So the planner applies a provenance discount β not a second numeric gate β to its own reading of the confidence, and never edits
Intent.confidenceitself. A measurement is the router's to report; the discount is the planner's reading.
(core/planner.py:57-62.)
The discounted values, for the fallback's ten rule ids (Β§27.10):
| Rule | Raw confidence | Effective (Γ 0.75) | Above SETTLED_CONFIDENCE 0.60? |
|---|---|---|---|
dual_modality |
0.92 | 0.690 | yes |
temporal_spatial |
0.90 | 0.675 | yes |
spatial |
0.88 | 0.660 | yes |
temporal |
0.85 | 0.6375 | yes |
caption |
0.85 | 0.6375 | yes |
vqa |
0.78 | 0.585 | no |
sar_single |
0.75 | 0.5625 | no |
optical_single |
0.72 | 0.540 | no |
no_match |
0.30 | 0.225 | no |
empty_query |
0.00 | 0.000 | no |
Five of the ten rules discount below 0.60. The three lowest-confidence task-producing rules β
vqa, sar_single, optical_single β all fall below SETTLED_CONFIDENCE after the discount, which
is exactly the intended effect: a bare "what" or "optical image" match is weak evidence and
should be treated as an unsettled route. The discount does not by itself change the plan (because
SETTLED_CONFIDENCE is not read by plan()), but it is the number ExecutionPlan.effective_confidence
carries into the trace β so a reader of the trace sees 0.585 for a fallback VQA, not 0.78.
ExecutionPlan.effective_confidence is rounded to 4 dp in the trace
(core/planner.py:245), and the router_source field carries "trained", "lexical_fallback" or
"forced" β so the discount and the provenance travel together.
Part J β The router-defect case study
44. Symptom
docs/RESEARCH_NOTES.md Β§3.1 states it in three lines:
The query "Where are the built-up areas in this image?" collapsed to
vqaand answered "River" β instead of routing togrounding. A second query, "Where is the new airport?", behaved the same way.
Two queries, both legitimate location questions, both answered as if they were questions about a
single object. The answer "River" is the tell: the VQA specialist was asked where something was
and answered with a land-cover class. The request was for coordinates and the response was a noun.
README.md:556-562 gives the release-level summary:
An earlier revision evaluated the temporal rule before the location rule, so "Where are the built-up areas in this image?" matched
\bbuilt\bas a change marker andareainside "areas" as a quantifier. With one asset it collapsed tovqaand answered "River". Fixed on 2026-09-25 infrontend/assets/js/mission.js; the fix is covered by regression tests and verified live. The same defect existed on a second surface (SQ.policyincore.js) and was fixed the same day.
Three things in that paragraph matter:
- Two compounding mis-reads.
\bbuilt\bfired the temporal predicate, andareamatched inside"areas"to fire the quantifier upgrade. - The collapse was to
vqa, not tochange. With one asset, achangereading would be substituted tovqaby the pair guard (Β§37.2) β so the temporal mis-read and the pair guard together produced the VQA answer. Two layers of the pipeline each did the locally-correct thing on a wrong input. - The defect existed on two surfaces β
frontend/assets/js/mission.jsandSQ.policyinfrontend/assets/js/core.jsβ and both were fixed on 2026-09-25. This is the sibling-site pattern: a fix applied to one implementation of a rule is not a fix until every implementation of that rule is checked.
45. Root cause
docs/RESEARCH_NOTES.md Β§3.2:
Two functions with different information:
interpret()β produces the console's reading; asset-count-blind (text only).chooseTask()β performs dispatch; asset-count-aware.The defect was in the dispatch path's handling of spatial/lexical cues, so region queries fell through to the generic VQA specialist.
The root cause is a lexical predicate, not a control-flow bug. The interpret() ladder (Β§36.2)
is correct in structure β sar β temporal β where β caption β vqa β but its inputs were
wrong: temporal was 'required' for a query that was not temporal, because built was in the
temporal term set.
Trace the defective path, step by step:
| Stage | Defective behaviour |
|---|---|
interpret("Where are the built-up areas in this image?") |
where = true; changeStem = false; pairWording = false; newAsChange = false; temporal = 'required' because built was a temporal term |
| branch selection | sar false β temporal === 'required' is TRUE β task = 'change', spatial = 'box' |
chooseTask(..., assetCount=1) |
wanted = 'change'; the upgrade predicate matched area inside "areas" β wanted = 'change_vqa'; assetCount < 2 and change_vqa requires a pair β substitute vqa |
| server | vqa specialist answers the question about one image β "River" |
Both mis-reads were in the reading, and the dispatch compounded them. The interpret() output
already had spatial_output = 'box' β the reading knew this was a spatial request β but the
task was change, and the ladder's change branch is what carried the wrong task forward. The
spatial flag was never consulted for task selection because the temporal branch had already claimed
the query.
This is why the fix had to be in the predicates, not the ladder. Reordering the branches would
not help: with temporal === 'required' true, the temporal branch fires before where under any
ordering that puts temporal above spatial β and the correct reading requires where to win. The only
fix that makes "Where are the built-up areas in this image?" a grounding request is to stop
built from making it temporal.
46. The four lexical changes
README.md:564-571 presents the fix as a table, and the source comments record each change
verbatim. All four are in frontend/assets/js/mission.js.
| # | Change | Source location | Why |
|---|---|---|---|
| 1 | built removed from the temporal term set entirely |
mission.js:87-91 |
"built-up areas" is land-cover vocabulary, not a change marker. While it sat in the temporal set, the location question "Where are the built-up areas in this image?" was read as a change request and answered with the degenerate one-word "River". Measured live, 2026-09-25. |
| 2 | \barea\b instead of bare area |
mission.js:189-194 |
Without the boundary the substring matched inside "areas", so the already-mis-read change question was upgraded again to change_vqa. The boundary keeps the quantifier reading for a real "how much area changed" while refusing the plural land-cover noun. |
| 3 | new counts as a change marker only when the query is not a where question |
mission.js:84-92 |
The repo ships eo/new-airport.jpg, so "Where is the new airport?" is a real question, and new is a place descriptor as often as a change marker. |
| 4 | the change stem is matched without a trailing \b |
mission.js:73-79 |
\bchang\b cannot match "changed", "changes" or "changing" β there is no word boundary between the stem and its inflection. With the boundary, the page's own default question ("What changed here?") fell through to the vqa branch, so the change path was unreachable from the UI that exists to reach it. |
46.1 Change 1 β built removed
The source comment, verbatim (frontend/assets/js/mission.js:87-91):
`built` was dropped outright -- "built-up areas" is land-cover
vocabulary, not a change marker. While it sat in the temporal set, the
location question "Where are the built-up areas in this image?" was read
as a change request and the server answered it with the degenerate
one-word "River". Measured live, 2026-09-25.
Note the corroboration in the corpus. The curated examples include
Compare optical and SAR to identify built-up regions. and
Show built-up areas using radar and optical together. β both labelled optical_sar, and both
containing "built-up" (Β§12.1). The corpus had already learned that "built-up" is a land-cover
noun; the frontend predicate had not. The fix aligns the predicate with the corpus.
46.2 Change 2 β \barea\b
The source comment, verbatim (frontend/assets/js/mission.js:189-193):
`\barea\b`, not bare `area`: without the boundary the substring matched
inside "areas", so "Where are the built-up areas ..." (already a
mis-read change question) was upgraded again to change_vqa. The
boundary keeps the quantifier reading for a real "how much area
changed" while refusing the plural land-cover noun.
This is the fix that prevents the second error. Even after change 1, the query would be
grounding β but without change 2, a different query would still be mis-dispatched. The comment's
parenthetical β "(already a mis-read change question)" β records that this boundary fix was applied
knowing the upstream reading was also being fixed, i.e. it is defence in depth rather than the
primary repair.
46.3 Change 3 β new gated on where
var newAsChange = /\bnew\b/.test(q) && !where;
(frontend/assets/js/mission.js:92.)
A single boolean AND, and it is the whole change. new fires the temporal predicate only when
the query is not a where question. So:
| Query | where |
newAsChange |
temporal |
|---|---|---|---|
Where is the new airport? |
true |
false |
'none' β grounding |
What is the new runway? |
false |
true |
'required' β change |
Show me the new construction. |
true (show me the) |
false |
'none' β grounding |
Row 2 is the documented residual (Β§48): "What is the new runway?" reads change rather than
vqa. The gate is where-specific, so a non-where question containing new still reads temporal.
That is the known and accepted cost of the fix β the alternative would be to drop new entirely,
which would lose the temporal reading of "What is new here?".
46.4 Change 4 β the chang stem
var changeStem = /chang/.test(q);
(frontend/assets/js/mission.js:79.)
The single most severe of the four, because it made a whole path unreachable. The comment explains the mechanism exactly:
The change stem is matched WITHOUT a trailing \b: `\bchang\b` cannot
match "changed", "changes" or "changing", because there is no word
boundary between the stem and its inflection. With the boundary, the
page's own default question ("What changed here?") fell through to the
vqa branch β so the change path was unreachable from the UI that exists
to reach it. The other terms keep their boundaries; they are whole words.
(frontend/assets/js/mission.js:73-78.)
\bchang\b requires a word boundary after the g. In "changed", the character after chang
is e, which is a word character β so there is no boundary, and the pattern fails. The same
applies to "changes" and "changing". The predicate matched only the bare word "chang", which
appears in no English sentence.
The consequence is stated in the comment and is worse than a single mis-routed query: the page's
own default question is 'What changed here?' (mission.js:26:
var QUERY = params.get('q') || 'What changed here?';). So the default question β the one the page
asks when no query is supplied β fell through to vqa. The change path was unreachable from the
default UI state.
The final sentence is the generalisable lesson: "The other terms keep their boundaries; they are
whole words." The chang stem is the one predicate that is deliberately not word-bounded,
because it is a stem rather than a word. Every other predicate in interpret() is either a whole
word or an explicit alternation.
46.5 The four changes, and which defect query each fixes
| Query | Fixed by | Result |
|---|---|---|
Where are the built-up areas in this image? |
changes 1 + 2 | grounding |
Where is the new airport? |
change 3 | grounding |
What changed here? (the page default) |
change 4 | change |
How much area changed? (regression guard) |
change 2 (preserved) | still change_vqa |
Change 2 is a two-sided fix: it fixes the "areas" false positive and preserves the
"how much area changed" true positive. A fix that broke the quantifier reading would have been a
regression, which is why the boundary form is \barea\b rather than removing area from the
predicate.
47. Live verification
47.1 Three independent passes
docs/RESEARCH_NOTES.md Β§3.3 and README.md:935-942 record the same table:
| Pass | Deployed HEAD | Result |
|---|---|---|
| 1 | ff46eba42b18 + d413d3672311 |
8/8 |
| 2 | 2d7ae53b482d |
8/8 |
| 3 | 2d7ae53b482d |
8/8 |
8 cases per pass β 6 regression plus 2 defect (README.md:927). 3 of 3 passes at 8/8, for
24 live runs and 24 correct dispatches, with 0 mock-node contamination on every run
(README.md:924-933).
Two of the three passes ran against the same HEAD (2d7ae53b482d). That is not redundancy for
its own sake β it is what makes pass 3 an independent confirmation of pass 2 rather than a repeat
of a shared failure mode. README.md:935 states the property that makes the passes genuinely
independent: "Each pass produced fresh run identifiers β no run id is shared between passes."
47.2 The two defect queries, with run ids
| Query | Run id | Dispatched |
|---|---|---|
Where are the built-up areas in this image? |
run_467ffa406f22 |
grounding |
Where is the new airport? |
run_46980ba55c62 |
grounding |
(docs/RESEARCH_NOTES.md:139-142; README.md:970-971.) README.md:973 adds:
"The last two are the router-defect queries. Both previously collapsed to vqa and answered
'River'."
Both now dispatch to grounding β the correct task for a location question.
47.3 The full pass-3 run-id table
README.md:962-971 records one complete pass, and the same pass the screenshots are drawn from. It is
reproduced here because it is the strongest available evidence that the router works end-to-end:
| Case | Query | Dispatched | Run ID |
|---|---|---|---|
| vqa | What type of terrain dominates this scene? | vqa |
run_fef26e91e7e6 |
| caption | Describe the main visual characteristics of this scene. | caption |
run_96281bdfcc08 |
| grounding | Where are the visible buildings in this image? | grounding |
run_e49adc8d319f |
| change | What changed between the earlier and later image? | change |
run_aedc59cbcdc9 |
| change_vqa | Did the coastline advance between the two observations? | change_vqa |
run_62ca98d510be |
| optical_sar | β¦combining the optical and SAR observations? | optical_sar |
run_beacf6aa4e21 |
| grounding | Where are the built-up areas in this image? | grounding |
run_467ffa406f22 |
| grounding | Where is the new airport? | grounding |
run_46980ba55c62 |
All six tasks are exercised in one pass. The change_vqa row is the documented quantifier
upgrade working as designed:
Note the fifth row: "Did the coastline advance between the two observations?" is read as
changeand dispatcheschange_vqaβ the documented quantifier upgrade, because the page's Answer block promises an answer and the server'schangereturns a spatial map with no language output.
(README.md:975-979.)
47.4 The harness false-positive β why the passes are trustworthy
docs/RESEARCH_NOTES.md Β§4 records a failure of the harness, not the code, and it is included
because it is the reason the passes can be believed:
An earlier live-validation harness typed queries with synthetic CDP key events, which Chrome silently drops when the window lacks OS focus. The harness therefore dispatched the page's default query and still recorded a "result" β a false pass.
Fix: the current harness asserts form state before dispatch (
q_ok,obs_ok,t0_ok), and uses deterministic query entry (js()value-set +type_text()via CDPInput.insertText).Independent check: the earlier 8/8 run was re-examined and confirmed not infected β its answers were query-specific and the query text was embedded in the answers.
The false-positive mechanism is worth naming precisely. Chrome dropped the synthetic keystrokes,
so the query box still held the page's default ('What changed here?'), the harness pressed Run, and
a real result came back for the wrong query. Because a result came back, the harness recorded a
pass. The default query is 'What changed here?' β which is exactly the query change 4 was needed
to fix (Β§46.4). So the harness false positive and the chang-stem defect interacted: a harness that
dispatched the default question would have been asking the one question the broken predicate
mis-routed.
The independent re-examination is what makes the earlier pass usable, and it used the right test: "its answers were query-specific and the query text was embedded in the answers." A genuine pass cannot be produced by a query the harness never entered.
47.5 The frontend regression suite
README.md:933 records 106 passed in tests/unit/test_frontend_live_wiring.py. This is the
suite that pins the routing behaviour, and it is the reason the four lexical changes cannot silently
regress.
Status of the fix: RESOLVED. Deployed to SatQuery-Frontend, verified across three independent
live passes, and covered by regression tests. The style guide's deployed-HEAD fact applies:
Frontend 2d7ae53b482d.
Part K β Residuals and honest boundaries
48. The known residuals
README.md limitation 8 states them, and they are the honest frontier of a lexical router:
Router lexical residuals. "What is the new runway?" reads
changerather thanvqa(thenew-as-change heuristic fires outsidewherequestions), and "How much built-up area was added?" readsvqa(under-trigger). A lexical router cannot cleanly separate "the new X" from "what's new"; a trained intent router exists inartifacts/router/but is not attached.
Two residuals, two different mechanisms:
| Query | Reads as | Correct would be | Mechanism |
|---|---|---|---|
What is the new runway? |
change |
vqa |
over-trigger β newAsChange fires because the query is not a where question |
How much built-up area was added? |
vqa |
change_vqa (or change) |
under-trigger β no predicate claims it |
48.1 Residual 1 β the new over-trigger
What is the new runway? has where = false (no where/locate/position/bound/outline/
coordinate term), so newAsChange = /\bnew\b/.test(q) && !where is true, and the reading is
change.
This is the accepted cost of change 3 (Β§46.3). The gate is where-specific: it fixes
"Where is the new airport?" without dropping new from the temporal vocabulary, which would lose
"What is new here?". The residual is the queries in between β a non-where question about a
newly-built feature.
The README's own diagnosis is the generalisable statement: "A lexical router cannot cleanly
separate 'the new X' from 'what's new'." Both readings are defensible from the surface form; the
distinction requires knowing whether new modifies a noun (a new runway) or a state (what is
new).
48.2 Residual 2 β the built-up area under-trigger
How much built-up area was added? reads vqa. It has:
where = falseβ no location term;changeStem = falseβ nochangstem;pairWording = falseβ none of the bounded pair terms match (addedis not in the list);newAsChange = falseβ nonew;sar = false;caption = false;- so the ladder falls through to the final
elseβvqa.
Note that change 2 is what prevents this query from being upgraded: \barea\b does not match
"area" inside "built-up area"... actually it does β "built-up area" ends with the word
area, so \barea\b matches. But the upgrade predicate is only consulted inside
if (wanted === 'change'), and wanted is 'vqa' here because the reading was vqa. So the
upgrade never runs. The under-trigger is in interpret(), not in the upgrade predicate.
The residual is a missing predicate, not a wrong one. added is a change verb, but it is not in
pairWording's alternation (between|versus|vs\.?|pair|temporal|before|after|differ\w*|expand\w*| grow\w*|encroach\w*|lost|removed). Adding add\w* would fix this query β and would risk new false
positives, which is why a lexical router has a frontier rather than a fixpoint.
48.3 The trained router exists and is not attached
The README's final sentence is the most important part of the residual entry: "a trained intent
router exists in artifacts/router/ but is not attached."
This is the state of the whole router subsystem at this release. The trained adapter is real,
measured, and shipped (Β§30βΒ§33). The frontend console does not use it β the console runs
interpret() and chooseTask(), both lexical. And IntentRouter.from_config defaults
adapter_path to None, so the server's default router is also lexical unless a caller passes a
path (router/classifier.py:143-153).
So at this release, in normal operation, the router is lexical on both surfaces, and the trained
adapter is a shipped artifact that is not wired in by default. That is the single most important
fact a reader should take from this chapter, and it is OPEN β not a defect, but an explicit
deployment decision that the docs record rather than hide.
49. What is NOT RUN, OPEN, BLOCKED or REJECTED for this topic
Per DOCS_STYLE_GUIDE.md Β§4, every chapter ends with an explicit list. For the router:
| Item | Status | Detail |
|---|---|---|
| Router test split, at the release level | NOT RUN |
threshold_sweep_val.json β n_test_examples_scored: 0, test_split_touched: false. The training-time test score (0.975, n=80) exists in metadata.json; the release-level test evaluation does not. |
| Router calibration | NOT RUN |
The 0.70 threshold is uncalibrated by the phase's own admission: "Calibration belongs in Phase 13 with real validation data, not here." |
| Router accuracy as a benchmark claim | NOT RUN |
The corpus is synthetic β 576 examples, 54 groups, corpus_limited: true. "Treat the router as working, not as benchmarked." |
| Plan minimum val queries β₯ 500 | NOT RUN β missed |
86 scored against a plan minimum of 500 (plan_min_val_queries). |
| Plan minimum hard negatives β₯ 100 | NOT RUN β missed |
The sweep's val split carries 0 hard negatives (hard_negatives_in_val: 0) against a plan minimum of 100. The hn_* families are held out to test by design, so val can never satisfy this. |
The two hn_* test misses (0.800 = 11/14) |
OPEN |
Which two examples were mis-classified is not recorded in any artifact. |
| The 50,822 vs 51,725 parameter discrepancy in three comments | OPEN |
Documentation-only; the code and the artifact both give 51,725. Β§8. |
| Whether the training-time test score (0.975) counts as a test-set result | OPEN |
README.md limitation 2 says the test set "was never run"; the artifact contains a test evaluation. Β§33.3. |
Router lexical residuals ("What is the new runway?", "How much built-up area was added?") |
OPEN |
README.md limitation 8. Β§48. |
| The trained adapter is not attached by default on either surface | OPEN |
IntentRouter.from_config(adapter_path=None) defaults to the fallback; the console is lexical. Β§48.3. |
adapter_source returns "trained" when an adapter exists but no encoder is loaded |
OPEN |
The property answers "is an adapter attached?", while its docstring claims "which path route() will take". Β§20.2. |
SETTLED_CONFIDENCE = 0.60 has no reader in plan() |
OPEN |
The constant documents design Β§2.3; the implementation gates on above_threshold. Β§39.2. |
| The coherence repair fires silently β nothing records that it did | OPEN |
The comment says "we record that we did so"; to_trace() has no such field. Β§21. |
| Per-family provenance of the shipped 576-example corpus | UNKNOWN |
The current dataset.py declares 70 curated + 45 templates = 585 before dedupe; the recorded totals exceed the declared totals for four families. Β§13.3. |
The exact list of duplicates dedupe() removed |
UNKNOWN |
Four within-CURATED duplicates are visible by inspection; the complete list requires running build_corpus() + dedupe(). Β§12.3. |
The derived class_weights used in training |
UNKNOWN |
Not recorded in metadata.json; computable from split.task_counts.train. Β§15.3. |
The output of self_check() on the shipped code |
UNKNOWN |
Not executed by this document and not recorded in any artifact. Β§29. |
| The shipped adapter's training corpus, byte-for-byte | UNKNOWN |
The adapter is dated 2026-09-16; router/dataset.py has been edited since. Β§30.1. |
Whether interpret()'s bare-sar rule (single modality word β optical_sar) is intended |
OPEN |
Differs from the Python fallback, which has separate SAR-only and optical-only rules. Β§36.2. |
| The frontend/backend routing parity | NOT RUN |
No test compares interpret()/chooseTask() against IntentRouter.route()/PolicyPlanner.plan() on the same queries. |
50. Where the evidence lives
50.1 Source of truth (the implementation)
| Path | What it establishes |
|---|---|
router/encoder.py |
the frozen MiniLM contract, max_length β€ 256, build_encoder |
router/adapter.py |
the 5-head architecture, init, forward, serialisation |
router/label_space.py |
the six task classes, four modalities, three binary heads |
router/fallback.py |
the seven term tables, eight rules, self_check() |
router/classifier.py |
IntentRouter, the 0.70 gate, the coherence repair, persistence |
router/dataset.py |
CURATED, TEMPLATES, SUBJECTS, split_by_group, validate |
router/train.py |
train_router, the loss, the cache, gate2_passed |
core/planner.py |
PolicyPlanner, the refusal list, the widening rules, the discount |
core/schemas.py |
Task (7), Modality (4), Intent + _consistency |
configs/base.yaml Β§router |
every configured router value |
frontend/assets/js/mission.js |
interpret(), chooseTask(), assetsForTask, validateOpticalSar |
frontend/assets/js/core.js |
SQ.policy β the second surface of the router defect |
50.2 Artifacts (the measurements)
| Path | What it records |
|---|---|
artifacts/router/router_adapter_v001/metadata.json |
the shipped adapter: 51,725 params, corpus 576/54, all three split metric blocks, the full 60-epoch history, duration_seconds: 4.92 |
artifacts/router/router_adapter_v001/adapter.pt |
the weights + the embedded architecture config |
artifacts/router/threshold_sweep_val.json |
the release-level sweep: val only, 50 thresholds, test_split_touched: false, overall_ungated_accuracy: 0.965116 |
artifacts/router/cache/ |
the cached corpus embeddings (.npy + .json per fingerprint) |
50.3 Phase records and notes
| Path | What it establishes |
|---|---|
docs/PHASE4_ROUTER_REPORT.md |
Gate 2 PASS, F4-1/F4-2/F4-3, four defects found by running, the standing caveats |
release/repo/docs/RESEARCH_NOTES.md Β§3 |
the router-defect case study, the live-pass table, the two run ids |
release/repo/docs/RESEARCH_NOTES.md Β§4 |
the harness false-positive and its fix |
release/repo/docs/RESEARCH_NOTES.md Β§6 |
the interpret()/chooseTask() asymmetry ruling (RESOLVED) |
release/repo/README.md Β§"Routing and the execution trace" |
the two-stage description, the five-head table, the confidence-gate diagram, the four lexical changes |
release/repo/README.md Β§"Live validation" |
the three passes, the 24 runs, the full pass-3 run-id table |
release/repo/README.md Β§"Known limitations" rows 2, 8 |
the validation-only status and the two residuals |
release/repo/DOCS_STYLE_GUIDE.md Β§3 |
the router fact that must never be stated wrongly: "0.965116 is validation, ungated, n = 86; the test split was NOT RUN" |
50.4 Tests
| Path | What it pins |
|---|---|
tests/routing/test_router.py |
82 tests (docs/PHASE4_ROUTER_REPORT.md Β§Files) β the label space, the encoder guards, the adapter shapes, the splitter, the fallback's self_check(), the gate |
tests/unit/test_frontend_live_wiring.py |
106 passed (README.md:933) β the frontend routing and the live-wiring assertions |
Named guards recorded in the phase report:
| Test | Guards against |
|---|---|
test_stratification_does_not_weaken_leakage_safety |
stratification breaking group-level leakage safety, across five seeds |
test_no_template_uses_another_tasks_characteristic_vocabulary |
a template whose surface form belongs to another task |
test_no_class_dominates_the_corpus |
any class exceeding 35 % of the corpus |
test_leakage_report_catches_a_deliberately_leaky_split |
the leakage audit itself β this guard was once deleted by a patch that meant to append, and the suite still reported 258 passed |
test_every_guard_in_the_suite_still_exists |
the meta-test that catches a deleted guard; proven to fire by renaming a guard and confirming the build breaks |
(docs/PHASE4_ROUTER_REPORT.md Β§"Four real defects", defect 3.)
The deleted-guard defect is the most instructive item in the phase record, and it is reproduced here in the report's own words:
A patch intended to append a test instead replaced
test_leakage_report_catches_a_deliberately_leaky_split. The suite still reported green β 258 passed β because nothing was asserting leakage detection any more.Fix: restored the guard, and added
test_every_guard_in_the_suite_still_existsβ a meta-test that imports each module and asserts the named controls are present. It was then proven to fire by renaming the guard, confirming the build breaks, and restoring.Its first run also failed, correctly: it listed
test_fusion_dim_mismatch_is_rejectedin the routing module, where it does not live. A meta-test making a false claim should fail, and it did.
A green suite is not evidence that a guard exists. The meta-test exists because the phase learned that lesson by running the suite and finding 258 passing tests with the leakage guard gone.
Chapter summary
The router is the first verb β it understands, and nothing more. It is two stages on both surfaces:
a reading that is blind to asset count (interpret(), IntentRouter.route()) and a dispatch
that is not (chooseTask(), PolicyPlanner.plan()).
The learned path is a frozen MiniLM encoder (1110a243fdf4, 384-d, max_length 128 inside a verified
256 ceiling) feeding a 51,725-parameter five-head adapter trained on cached embeddings in
4.92 seconds on CPU. It scored 0.975 on its training-time test split with all six classes
measured, and 0.800 hard-negative accuracy β the most honest generalisation number in the
artifact. The release-level threshold sweep was val only, and its own note says "the test split
was NOT touched."
The lexical path is eight ordered rules over seven term tables, never inventing capability, refusing
at 0.30 rather than guessing. It is what runs by default, because adapter_path defaults to
None.
The router's most instructive episode is a defect that was found live, diagnosed to two
compounding lexical mis-reads, fixed with four documented changes, and verified across three
independent live passes at 8/8 β with run ids run_467ffa406f22 and run_46980ba55c62 now
dispatching grounding where they once collapsed to vqa and answered "River". Two residuals
remain, and a trained router sits in artifacts/router/ not yet attached.