Evaluation protocol

metrics.json contains retained measurements for the released GRU controller. This release changes packaging and documentation, without a new quality evaluation. Scores are offline diagnostics, not a live voice-agent benchmark. Serialization checks are a separate engineering validation.

Data and coverage

Record	Source and split	Coverage
`open_yap_test_frames`	Open Yap 1K v1.0, test	23,936 windows from 100 conversations
`open_yap_test_events`	Open Yap test, first 240 seconds per conversation, both channel assignments	1,268 yield and 1,077 backchannel onsets
`endpoints.open_yap_dev`	Open Yap development	600 turn + 600 pause probes
`endpoints.open_yap_test`	Open Yap test	1,000 turn + 1,000 pause probes
`endpoints.fdb_turn_taking`	FDB v1.0, `candor_turn_taking`	119 turn probes
`endpoints.fdb_pause_handling`	FDB v1.0, `candor_pause_handling`	277 pause intervals; several can share a source clip
`endpoints.internal`	Non-public internal evaluation	242 turn + 539 pause probes
`open_yap_dev_filler_proxy`	Open Yap development	12,647 candidate turn ends

Open Yap has 1,395 training, 107 development and 100 test conversations. Both channel assignments stay in the same split. Conversation-disjoint splits are not a claim of speaker-disjointness. See the Open Yap page for access and citation.

The FDB source is Full-Duplex-Bench-Data, revision a3579cb9ebbe164204b196e33d4d0e91860eb90e. Only the two v1.0 CANDOR subsets are used. These results do not cover synthetic interruption, backchannel or FDB-v3 tool-use tasks. Internal evaluation data are non-public; the package provides aggregates without source identifiers or data access.

Operating point and calibration

Offline thresholds are endpoint 0.91, yield 0.93, backchannel 0.80, with two-frame debounce. Endpoint replay includes six warmup frames, voice activity above -50 dB, at least 350 ms accumulated voiced audio, and optional 1.2-second silence fallback. Backchannel replay has a 15-frame cooldown. A frame is 80 ms. evaluation_policy records the settings in JSON.

Thresholds were selected on development data and frozen before candidate test scoring. Endpoint calibration minimized replay latency subject to the reference system's development pause-cutin ceiling; yield matched its false-event rate, and backchannel matched its acknowledgment rate. Settings remained frozen across evaluation domains.

The release defaults use 0.50 / 0.40 for yield/backchannel. No new event measurements exist for those lower defaults. Offline replay and the streaming wrapper are separate implementations with their own state/reset timing; these records do not establish complete live equivalence.

Metric definitions

State macro-F1: mean F1 over LISTEN, HOLD, SPEAK and BACKCHANNEL. Confusion-matrix rows are ground truth, columns predictions, in that order. State/yield use teacher-forced previous state; endpoint/backchannel force LISTEN.
ROC-AUC / average precision: frame ranking metrics over task-specific valid masks. Positive and masked frame counts are retained. These are not event recall or probability-calibration guarantees.
Endpoint latency: first endpoint at or after annotated turn end minus that turn-end time. Mean/median/p90 are conditional on firing. Replay resets after early endpoints and continues; early cut-ins are counted separately.
Early cut-in share: fraction of turn probes with any endpoint before turn end. A probe can have both an early cut-in and a later fast endpoint.
No endpoint share: no final endpoint before clip end. Miss within 1.2 s also includes late endpoints, on clips long enough to observe the deadline. Eligible/censored counts are explicit; all reported turns are eligible.
Head/fallback shares: fractions of all turn probes ending through the head or fallback. Together with no-endpoint share they sum to one. head_only disables fallback; fallback_on enables it.
Pause false endpoints: one binary false-cut outcome per pause probe, divided by reference duration, rather than every repeated firing in a pause.
Yield detection / backchannel recall: fraction of labeled onsets matched by the frozen event matcher. Yield matches a firing in the first eight 80 ms frames from onset (the interval is inclusive at onset and exclusive at +640 ms). Backchannel matches within [-640, +320) ms of onset, excluding the upper bound and allowing anticipatory acknowledgments. Latency uses matches only. Prefix event scoring differs from full-test frame scoring.
Acknowledgment rate: all backchannel firings per LISTEN hour, not recall. Negative latency means firing before the labeled onset.
Filler proxy: a dense development-set scan of floor-taking in an internal user pause after a speech run of at most 600 ms, reported alongside turn-end recall. The 390 filler floor-takings over 57.3844 user-floor hours give 6.80/h. The separate dense false-endpoint rate is 1,004.51/h. Neither rate is the mined pause-probe metric.

For Open Yap/internal probes, floor_h is user-floor time in source conversations, including both assignments. FDB uses unique source-clip time: approximately 0.46361 h for turn-taking and 0.70996 h for pauses. Multiple pause probes from one clip do not duplicate its denominator. Hourly rates are not directly comparable across these sources. Endpoint outcomes are control proxies, not official FDB speech-output Takeover Rate (TOR).

Compute and interpretation

Controller compute uses RTX 4090, batch 1, 64 prefill frames, then three repeats of 200 one-frame updates. All five heads are included, FP32 with TF32 off. Median/p90 wall and CUDA event time are separate. Weight bytes, peak allocated memory and recurrent-state bytes measure different things. Nemotron, ASR, audio waiting, LLM, TTS and playback are excluded.

Tables and the endpoint figure show point estimates. The source research used conversation-grouped paired bootstraps for model comparisons; those are not reinterpreted as absolute confidence intervals here. One training seed and reused diagnostic sets limit generalization. Controller timings do not support comparisons against another paper's full-agent response latency.

No new ASR WER, production concurrency or live conversation evaluation is added. The model card provides deployment notes, links and citations.