engine/CONTROLLER_NOTES.md

WhenToSpeak Controller Notes

The controller is training-free. It consumes signals from a Brain interface and never loads or calls a model itself. The live brain must provide, for each incremental transcript update and each agent context, mean token surprise, a last-layer hidden vector, readiness, and turn-end probability.

Signals

• surprise: mean per-token negative log-likelihood of the newly added user tokens, teacher-forced.
• hidden: mean last-layer hidden-state vector for the newly added tokens.
• readiness: speculative reply confidence for this agent. Draft about eight tokens and compute readiness = 1 / (1 + mean_token_entropy).
• p_end: heuristic probability that the human turn is complete. The live loop should combine trailing silence, sentence-final punctuation, and high EOS probability.

Urge

Each agent keeps an online running mean/std of surprise and uses the current z-score. Hidden-state cosine deltas feed Adams-MacKay BOCPD; a collapse in MAP run-length becomes the change-point score.

U_t = w_surprise*z(surprise)
    + w_change*changepoint_score
    + w_readiness*readiness
    + w_end*p_end
    + w_barge*max(z(surprise), 0)*readiness*(1 - p_end)

tau is the single global conversational-aggressiveness knob. Lower tau makes the panel take the floor sooner; higher tau makes it wait.

Arbitration

Each tick is deterministic. Agents first classify local intent as SILENT, BACKCHANNEL, TAKE_FLOOR, or INTERRUPT. Only the highest-urge agent above tau may take the floor or interrupt. Non-winning agents may still backchannel if their urge clears the derived backchannel threshold. A short refractory period prevents repeated firing on adjacent ASR updates.