Failure-analysis protocol

Turn detection fails in sequences, not just rows. This protocol turns model predictions into reviewable product errors without leaking the official test back into development.

Scope and non-negotiable rules

Analyze train/development and the locked local Hinglish development split while iterating.
Inspect the official test only once after freeze. Its failure report is for disclosure and future data collection, not same-benchmark tuning.
Keep raw confusion counts and denominators beside every rate.
Group confidence intervals and sampling by speaker/conversation/leakage group, never by correlated rows alone.
Do not listen to or expose audio without appropriate access and consent.
Do not put names, phone numbers, addresses, order IDs, transcripts containing PII, or direct speaker identifiers in review exports.

Error priorities

Priority	Error	Product effect	Primary measure
P0	False `END` on `HOLD`	User is interrupted	FP count, FPR, interrupted-turn rate, interruptions/hour
P1	Repeated/double `END`	Duplicate agent response	events per true turn in controller replay
P1	False `HOLD` on `END`	Response delay or timeout	FN count, FNR, endpoint-delay p50/p90/p95
P1	Forced timeout on true `HOLD`	Policy interruption despite low model score	timeout reason count/rate
P2	Poor calibration	Threshold fails after shift	Brier, log loss, ECE, reliability bins
P2	Latency/resource regression	Slow or expensive response	end-to-end p50/p95/p99, load time, RSS

The first promotion gate is a predeclared false-interruption budget. A model with higher aggregate F1 but materially more P0 errors is not an improvement.

Failure taxonomy

Assign one primary cause and any secondary tags. Use unknown when evidence is insufficient rather than inventing an explanation.

Linguistic and conversational

filled_pause: uh/um/haan/matlab/achha, including elongated fillers;
discourse_marker: “so”, “but”, “toh”, “actually” before continuation;
syntactic_continuation: conjunction, subordinate clause, unfinished phrase;
slot_continuation: partial address, PIN, phone, order/tracking ID, date/time;
enumeration: pause between list items;
self_repair: correction, restart, or abandoned phrase;
backchannel: short acknowledgement while retaining/yielding the floor;
code_switch_boundary: Hindi↔English switch near the pause;
prosody_rising, prosody_level, prosody_falling: perceived boundary tone;
complete_but_short: short legitimate endpoint;
deliberate_long_hold: long thinking pause followed by continuation;
annotation_ambiguous: reasonable listeners disagree.

Acoustic and collection

background_speech, traffic, music, impulse_noise;
far_field, reverberation, low_snr;
clipping, low_gain, codec, sample_rate_conversion;
vad_truncation: checkpoint cuts speech/filler incorrectly;
trailing_silence, leading_silence, non_speech_only;
synthetic_prosody, source_artifact, duplicate_or_template;
duration_crop: relevant context fell outside the audio suffix.

System and policy

frontend_mismatch: train/runtime log-mel disagreement;
mask_or_padding: padded frames influence score;
onnx_drift, quantization_drift;
threshold_shift: calibration does not transfer to slice/domain;
minimum_silence, relaxation, maximum_timeout, debounce;
latency_budget: correct decision arrived too late;
state_reset: speech after END did not start a clean turn.

Required review record

Store review metadata separately from raw audio. One JSONL row per checkpoint:

{
  "review_id": "random-id",
  "run_id": "E-010-seed17",
  "split": "validation",
  "group_id": "hashed-group",
  "turn_id": "hashed-turn",
  "checkpoint_index": 2,
  "target_endpoint": false,
  "endpoint_probability": 0.73,
  "threshold": 0.61,
  "model_prediction": true,
  "controller_state": "END",
  "controller_reason": "model_endpoint",
  "silence_ms": 420,
  "inference_ms": 5.8,
  "language": "hinglish",
  "dataset": "private_hinglish_v1",
  "synthetic": false,
  "midfiller": true,
  "endfiller": false,
  "duration_seconds": 3.4,
  "primary_tag": "filled_pause",
  "secondary_tags": ["code_switch_boundary"],
  "reviewer_count": 3,
  "reviewer_agreement": 0.67,
  "notes": "no transcript or PII"
}

Use random or hashed IDs. Free-text notes must not quote a participant or reconstruct customer data.

Sampling plan

For each candidate and baseline, review:

every false interruption when the set is operationally manageable;
otherwise, a group-stratified random sample of at least 100 false interruptions, with inclusion probabilities retained;
at least 100 false holds, oversampling long endpoint delays;
50 highest-confidence correct predictions of each class to catch label or shortcut errors;
50 near-threshold examples on each side;
a balanced sample across language, source, human/synthetic, filler, duration, device/noise, and consented demographic slices;
full controller sequences for any repeated END, timeout, or state-reset issue.

When a slice has fewer examples, review all and report the small denominator. Do not silently drop undefined FPR/recall for one-class slices.

Analysis workflow

1. Validate the evaluation plumbing

confirm 1 = END in labels, predictions, and confusion counts;
recompute FP/(FP+TN) and FN/(FN+TP) from stored counts;
verify prediction IDs join one-to-one with the frozen manifest;
verify the threshold came from development, not test;
compare frontend metadata and ONNX input/output names with the checkpoint;
test masked-padding invariance and PyTorch/ONNX parity;
verify no leakage group/audio hash/metadata key crosses splits.

2. Quantify before listening

Generate overall and sliced counts, score histograms, reliability bins, and duration/silence deciles. Rank potential shortcut signals: source, synthetic status, audio length, codec, and sample rate. This prevents a few memorable clips from replacing distributional evidence.

3. Conduct blinded review

Reviewers see the checkpoint audio/context and target question, but not model score, prediction, source model, or other reviewers' votes. Randomize baseline and candidate failures. Preserve uncertain and collect an optional taxonomy tag. Adjudicate only after independent review.

4. Connect clip errors to controller behavior

Replay chronological pause checkpoints. A clip false positive may be suppressed by minimum silence or debounce; a modest false negative may be harmless if the next checkpoint ends promptly. Conversely, a low aggregate FPR can still produce too many interrupted turns if errors cluster across different conversations.

5. Choose the smallest justified intervention

Examples:

filler-specific failures → rebalance real filler minimal pairs or adjust auxiliary weighting;
crop failures → compare 2/4/8-second suffixes;
calibration-only slice shift → recalibrate or use a validated controller policy, not a larger network by default;
synthetic-source shortcut → reduce synthetic dominance and validate on real speakers;
VAD truncation → fix checkpoint generation before retraining endpointing;
INT8 score drift → recalibrate INT8 threshold or retain FP32;
maximum-timeout interruptions → product-policy review, not label relabeling.

Every new intervention gets a new registered experiment and an untouched development comparison. Do not manually patch individual test failures.

Current findings

The following aggregate development findings are supportable; causal taxonomy labels still require authorized blinded listening:

Source-holdout confounding is severe. In the inspected shard, source-held train is 2,614/2,617 synthetic, while validation is 550/648 human-tagged and has only English/Spanish. A delta cannot be assigned to source alone.
The packaged TinyTCN misses most endpoints at the strict threshold. On 326 IID development clips it produces 10 TP, 3 FP, 163 TN, and 150 FN at threshold 0.7410008. All three false interruptions are synthetic-tagged; two occur in the 134-row midfiller slice. These counts do not establish cause.
Simple acoustics remain stronger. The full partial-shard logistic baseline ranks validation examples at AUROC 0.752, yet recovers only 21/160 END examples when limited to 3/166 HOLD false positives. The TinyTCN recovers 10/160 at the same FP count; its paired recall delta is −0.06875 with 95% CI [−0.13298, −0.01220].
Trailing silence is a model/controller interface risk. Appending 800 ms of valid silence flips 26/326 decisions, raises FPR from 0.0181 to 0.0361, and shifts probability by 0.1058 on average in absolute value. The external controller also consumes silence, creating potential double-counting.
No Hinglish failure claim exists. There are no local participant recordings. The Hindi-tagged slice has only 16 examples and is not verified code-switched speech.
Current grouping cannot establish identity separation. All 3,265 base IID linkage groups are singletons because the shard lacks usable speaker, conversation, and voice IDs. Zero observed-key crossings are real, but speaker/TTS leakage is unknown and the IID grouped bootstrap is effectively row bootstrap. The source stress split instead clusters by dataset source.

Privacy-safe hashed case IDs identify the 3 false interruptions and 150 missed ends for authorized local review. Public reports contain no audio, transcript, raw record ID, or source path. No claim is made about specific acoustic or linguistic causes because no blinded listening analysis has been performed.

Promotion gates

A release candidate must satisfy all of the following:

no unresolved data, split-leakage, frontend-parity, or ONNX-parity defect;
FPR within the predeclared budget on locked development and official test;
false-interruption and delay targets met in full controller replay;
no material regression on real-human, Hinglish, filler, long-hold, identifier, device/noise, and major source/language slices;
grouped confidence intervals rule out a practically harmful regression against the chosen baseline;
at least two reviewers approve every sampled P0 taxonomy cluster and document residual risks;
final threshold is calibrated for the exact FP32 or INT8 artifact;
model/data cards list all known failures, denominators, and excluded records;
privacy, consent, licensing, and redistribution reviews are complete.

If a gate cannot be measured, mark it unknown. Unknown is not a pass.