docs/04_experiment_protocol.md

Experiment Protocol

Runs

• Run ID format: YYYYMMDD-HHMMSS__seed

Training

• Train structured S and dense D_a side-by-side on the same batch stream.
• Update order per step:
  1. forward/backward/step S
  2. forward/backward/step D_a
• Eval cadence: every 500–1000 steps (fixed).

Splits

• Fixed IID eval set
• Fixed OOD eval set
• Fixed structure-break eval set (e.g., bind-shuffle)

Metrics

Primary:

• IID accuracy/loss
• OOD accuracy/loss
• structure-break accuracy/loss

Secondary (structured-only):

• expert usage entropy
• hemisphere disagreement vs error (simple AUROC)

Early-kill criteria

PoC success if ANY:

1. S > D_a on OOD by clear margin at same budget.
2. S degrades less under structure-break perturbation.
3. Disagreement is a useful error signal.

PoC failure:

• No meaningful separation on OOD/robustness and no disagreement signal within a small budget.

Artifacts

All outputs must go under: /store/speedy/dualbrain/out/bihmoe-poc//