docs/RESULTS.md

Results

All results below are measured, not estimated. Sample sizes and sources are stated for every number — see experiments/EXPERIMENTS.md for the full experiment log with hypotheses and decisions.

Summary table

Experiment	F1	Sample	Key finding
EXP-001 Energy/silence baseline	0.400	90-clip val (Phase 3, 300-clip sample, one training shard)	Weak — fixed silence-duration threshold can't distinguish filler-preceded pauses from real endpoints
EXP-002 Acoustic features + Logistic Regression	0.575	Same 90-clip val as EXP-001	Strong improvement over EXP-001; still limited by lack of any lexical/semantic signal
EXP-003b Temporal (recent-window) acoustic features	+0.112 F1 delta vs. global-only	Same 90-clip val, feature-set ablation on EXP-002's classifier	Recent-window (100–1000ms tail) features meaningfully help; kept in the default feature set
EXP-004 Whisper Tiny frozen encoder + Logistic Regression	0.693	75-clip val (fresh, independent 250-clip sample, same dataset)	Strongest measured approach; specifically reduces filler-associated false-END errors

On the EXP-002 vs. EXP-004 comparison: EXP-002 and EXP-004 used different development samples — both drawn from pipecat-ai/smart-turn-data-v3.2-train using the same general methodology (stratified sampling across endpoint_bool, language, dataset source, synthetic), but not the same clips, and not a paired before/after comparison on identical data. The correct way to describe the +0.118 F1 difference is: a directional improvement on independent samples drawn from the same Smart Turn training distribution — not a controlled, paired experiment. This is stated explicitly wherever this comparison appears in this project's documentation.

Full metrics

EXP-001 — Energy/silence baseline

• n=90 (validation split)
• Accuracy: 0.567
• F1: 0.400
• False-END rate: 0.309
• False-CONTINUE rate: 0.629
• Params: 2 (thresholds)

EXP-002 — Acoustic features + Logistic Regression

• n=90 (same validation split as EXP-001)
• Accuracy: 0.622
• F1: 0.575
• False-END rate: 0.400
• False-CONTINUE rate: 0.343
• Params: 92 (91 features + intercept), ~2,920 bytes
• Inference latency (CPU, Claude sandbox): mean 12.2ms / p95 22.4ms per clip

EXP-003b — Temporal (recent-window) features ablation

• Same 90-clip val set, same classifier family (Logistic Regression)
• Global-only: accuracy 0.511, F1 0.463, false-END rate 0.509
• Global + recent-window: accuracy 0.622, F1 0.575, false-END rate 0.400
• Delta F1: +0.112; delta false-END rate: −0.109; delta false-CONTINUE rate: −0.114
• Both error rates improved simultaneously — not a precision/recall tradeoff in one direction

EXP-004 — Whisper Tiny frozen encoder + Logistic Regression

• n=75 (validation split from a fresh, independent 250-clip sample: 175 dev / 75 val, seed=42)
• Accuracy: 0.693
• F1: 0.693
• False-END rate: 0.257
• False-CONTINUE rate: 0.350
• Encoder params: 8,208,384 (~32.8MB fp32) — matches the confirmed reference Smart Turn architecture's documented ~8M params almost exactly
• Classifier params: 92 (~2,920 bytes)
• Inference latency (Colab T4 GPU): preprocessing 13.5ms + encoder ~1.7ms + classifier ~0.45ms ≈ **15.6ms/clip total**

Filler analysis

Phase 3's error analysis on EXP-002 found that 73% of false-END errors had filler metadata present (midfiller or endfiller = True), versus 50% of false-CONTINUE errors — meaning the acoustic-only model's dominant failure mode was exactly the case the assessment brief names as hard: a pause that acoustically resembles an ending but linguistically isn't (filler-preceded).

Phase 4's EXP-004 evaluation targeted this directly:

	Acoustic (EXP-002, Phase 3, n=39, midfiller only)	Whisper (EXP-004, Phase 4, n=35, midfiller OR endfiller)
F1 on filler-flagged clips	0.529	0.722

And at the individual-error level: Whisper's false-END errors had a filler flag present 56% of the time (5/9), down from the acoustic model's 73% — consistent with, though not definitive proof of, Whisper's representation helping specifically on filler-associated pauses.

This filler-subset result should be treated as directional, not conclusive — sample sizes are modest (35–39 clips), the two measurements come from different samples, and the slice definitions differ slightly (Phase 3 used midfiller alone, Phase 4 used midfiller OR endfiller). The no_filler_known slice in Phase 4 (n=7) was too small to draw any conclusion from and is explicitly excluded from this comparison — see docs/ERROR_ANALYSIS.md.

Latency

~15.6ms per clip, measured on a Colab T4 GPU, for the full pipeline (preprocessing + Whisper encoder forward pass + classifier). This is hardware-dependent and should not be read as a CPU-deployment number — the acoustic baseline's 12.2ms figure was measured on CPU, so the two latency numbers are not directly comparable to each other. No CPU-measured Whisper latency exists yet for this project.

Limitations

• Development sample sizes are small. EXP-001/002/003b: 90-clip validation set. EXP-004: 75-clip validation set. Neither approaches the scale needed for tight confidence intervals on F1 or slice metrics.
• EXP-002 and EXP-004 used different samples — not a paired, controlled comparison. See note above.
• No transcripts exist anywhere in the dataset (spoken_text is null for every row, confirmed at the schema level) — no result in this project relies on or claims access to what was actually said.
• No true conversation-level endpoint timestamps exist — this project measures classification accuracy and false-END/false-CONTINUE rates, not true wall-clock endpoint latency (time from actual turn boundary to system decision). This is stated explicitly everywhere latency is discussed.
• Hindi/Hinglish code-switching cannot be directly verified from dataset metadata — Hindi is present as a language tag (docs/LANGUAGE_ANALYSIS.md documents real, measured Hindi proportions: 4.2% of the full inspected shard), but whether it's naturally code-switched Hindi-English speech or monolingual Hindi (plausibly TTS-generated) was never independently verified via ASR in this project (EXP-000, proposed in Phase 2, was never run). This project does not claim Hinglish robustness.
• GPU-measured Whisper latency differs from CPU deployment latency — a production CPU deployment would likely see meaningfully higher encoder latency than the ~1.7ms GPU figure above; this hasn't been measured and shouldn't be assumed.
• No official test-set evaluation has been run. pipecat-ai/smart-turn-data-v3.2-test was reserved throughout this project and was never used for tuning or evaluation — this is a deliberate methodological choice, not an oversight, but it does mean no number in this document represents true held-out generalization performance in the strictest sense (the "held-out" splits used were carved from the training distribution, not the official test set).

Future work

• Streaming incremental inference (avoid full-buffer recomputation on every VAD-triggered call — see docs/INITIAL_ANALYSIS.md §5 for the design tension this raises with the reference Smart Turn architecture)
• Training/evaluating on a larger sample (the full 270,946-row training set, or at least a much larger subset than 250–300 clips)
• A controlled Hinglish challenge set with verified code-switched content (via ASR-based verification, EXP-000, never run in this project)
• Final evaluation on the official held-out test set (pipecat-ai/smart-turn-data-v3.2-test), once the architecture is fully locked