docs/architecture.md

Architecture

Chosen pipeline

Audio
  |
  v
16 kHz Mono Preprocessing
  |
  v
Whisper Tiny
Frozen Encoder
  |
  v
Mean Pooling (over encoder time axis -> 384-dim vector)
  |
  v
Logistic Regression
  |
  v
P(END)
  |
  v
Threshold / Debounce / Hysteresis
  |
  v
END / CONTINUE

Why each stage exists

• 16kHz mono preprocessing: matches both the dataset's native format (confirmed: pipecat-ai/smart-turn-data-v3.2-train is 16kHz mono for every row inspected — docs/PHASE3_REAL_AUDIO_VALIDATION.md) and Whisper's expected input format. No resampling needed for in-distribution data; src/turn_detector/audio_io.py resamples defensively if a caller provides audio at a different rate.
• Whisper Tiny frozen encoder: chosen over the acoustic-only baseline based on measured evidence, not the assessment brief's suggestion alone — see docs/RESULTS.md for the actual numbers (F1 0.693 vs. 0.575, directional improvement on independent samples from the same dataset) and docs/ERROR_ANALYSIS.md for why: it specifically reduced the filler-associated false-END errors the acoustic model struggled with. Frozen (not fine-tuned) because fine-tuning was explicitly out of scope for this submission's timeline — see "Future Work" in the README.
• Mean pooling: the simplest, cheapest way to collapse Whisper's variable-length encoder output into a fixed-size vector for a linear classifier. Untested alternatives (attention pooling, last-token pooling, temporal-aware pooling) are noted as future work, not assumed inferior — mean pooling is what was actually measured.
• Logistic Regression: smallest classifier that worked well in testing (92 parameters, ~2.9KB) — chosen over an MLP per the "prefer the smallest head that works" instruction, and because it was never shown to underfit in the experiments actually run.
• Threshold / debounce / hysteresis: exists because a turn detector that commits to END the instant P(END) crosses 0.5 on a single noisy call will end turns prematurely on one uncertain pause. This layer (src/turn_detector/inference.py: TurnDecisionConfig, TurnDecisionState) lets a caller require sustained evidence — a minimum confidence margin, and/or several consecutive END calls in a streaming context — before committing to END, without training a second model. Defaults are simple (threshold=0.5, no debounce) so the basic case stays basic; both knobs are opt-in.

What this diagram does NOT claim

• Not a continuous per-frame streaming architecture. Like the dataset it was trained on, and like the reference Smart Turn project it takes inspiration from, this is an event-triggered classifier over a buffered clip, not a frame-by-frame streaming model. See docs/INITIAL_ANALYSIS.md §5 for the full discussion of this design choice and its tradeoffs.
• Not fine-tuned. The Whisper encoder's weights are frozen exactly as measured in EXP-004 — this diagram reflects what was actually built and evaluated, not a target architecture that was never tested.