experiments/EXPERIMENTS.md

Experiment Log

Rule for this file: a row only gets metrics once the corresponding script has actually been run against real data in this environment. Until then the Status column stays NOT RUN and the metric columns stay blank — never estimated, never "expected."

Context as of end of Phase 3: the user uploaded a REAL Parquet shard (train-00000-of-00010.parquet, 464MB, 3,153 rows) from pipecat-ai/smart-turn-data-v3.2-train. No pyarrow/fastparquet are installed and none are installable (no network — same host_not_allowed block covers pypi.org), so a minimal pure-Python Parquet reader was built from scratch (src/turn_detector/parquet_reader.py: Thrift compact-protocol footer parser, hand-written Snappy decompressor, RLE/ bit-packed definition-level decoder) to actually read this file — see docs/PHASE3_REAL_AUDIO_VALIDATION.md for the full account, including real bugs caught and fixed while building it against this exact file (wrong definition-level bit-width for the nested audio struct; missing INT32 dictionary-page support). Audio is FLAC-encoded (fLaC magic, confirmed by direct byte inspection); ffmpeg (already present in this sandbox) is used for decoding instead of soundfile (not installed, not installable).

All 3,153 rows of this shard were successfully decoded — 0 corrupted rows. A reproducible stratified sample of 300 real clips (seed=42) was drawn and materialized as real 16kHz mono WAV files (data/raw/phase3_sample/). EXP-001/002/003 below were run against a 90-clip validation split held out from those 300 (210 dev / 90 val, random split, seed=42) — tuning done on dev only. Every result below is labeled SMALL REAL-AUDIO VALIDATION and must not be read as full-dataset or full-shard performance — it's 90 held-out clips from one of ten shards.

ID	Name	Hypothesis	Status	Accuracy	F1	Premature-END rate	Endpoint latency	Params	CPU latency	Decision
EXP-000	Audio content check (Hinglish verification)	The dataset has no transcripts and no code-switch label; running Whisper ASR over a sample of hin-tagged clips (vs. a comparison sample of eng-tagged clips) will reveal whether genuine Hindi-English code-switching is present, or whether hin rows are monolingual Hindi. This determines whether "Hinglish robustness" can be meaningfully trained/evaluated from this dataset at all.	NOT RUN — needs sampled audio in-sandbox first
EXP-001	Energy/silence endpointing baseline	A fixed-threshold silence-duration rule (classic VAD-style endpointing) will catch many obvious ends-of-turn but will fail badly on filler-preceded pauses, given the label methodology explicitly defines "incomplete" as often ending in trailing silence after a filler word — acoustically similar to a genuine end. This gives a real lower-bound baseline and a first look at real premature-END behavior.	SMALL REAL-AUDIO VALIDATION — RUN. Threshold tuned on 210-clip dev set (best: silence_duration_threshold_sec=0.25, dev accuracy=0.638), evaluated on 90-clip held-out val set.	0.567	0.400	0.309	N/A (dataset lacks conversation-level timestamps)	2	not separately benchmarked (rule-based, negligible)	Confirmed weak, as hypothesized; kept as floor baseline
EXP-002	Classical audio features + lightweight classifier	Adding RMS/ZCR/MFCC/spectral stats/pause-duration/energy-slope features to a small classifier (e.g. logistic regression / small GBM) should beat EXP-001 by capturing trend information a fixed threshold can't, while remaining extremely cheap.	SMALL REAL-AUDIO VALIDATION — RUN. Logistic regression, median-impute→standardize→LR pipeline, fit on 210-clip dev set, evaluated on same 90-clip val set as EXP-001.	0.622	0.575	0.400	N/A (dataset lacks conversation-level timestamps)	92 (91 features + intercept), 2920 bytes	mean 12.2ms / p95 22.4ms per clip, this dev machine, single-clip calls, 90 repetitions	Beats EXP-001 on this val set; kept as current best acoustic baseline
EXP-004	Whisper Tiny frozen encoder + lightweight classifier	Whisper Tiny's encoder representations, trained on large-scale multilingual speech, should encode prosodic/phonetic cues (trailing intonation, breath, filler acoustics) that classical features miss — specifically reducing false-END errors on filler-associated pauses, the exact failure mode identified in Phase 3's error analysis.	SMALL REAL-AUDIO VALIDATION — RUN (Colab, GPU). Frozen openai/whisper-tiny encoder, mean-pooled over time, + logistic regression head. Fresh 250-clip HF-streaming sample (175 dev / 75 val, seed=42), independent of Phase 3's 300-clip sample — same dataset/method, not the same clips.	0.693	0.693	0.257	N/A (dataset lacks conversation-level timestamps)	8,208,384 (encoder) + 152 (LR head)	preprocessing 13.5ms + encoder ~1.7ms + classifier 0.45ms ≈ 15.6ms/clip, Colab T4 GPU (not directly comparable to EXP-002's CPU latency)	Clear improvement over EXP-002 (ΔF1 +0.118, Δfalse-END −0.143); filler-slice F1 0.722 vs. acoustic's 0.529 supports the hypothesis. Adopted as the leading candidate architecture — see decision below.
EXP-003b	Global-only vs. global+recent-window (temporal) features	The acoustic state near the end of the observed audio (last 100–1000ms) contains more useful turn-completion signal than global whole-clip statistics alone.	SMALL REAL-AUDIO VALIDATION — RUN. Same classifier (logistic regression), same 210/90 dev/val split; only the feature set changes (global-only vs. global+recent-window).	global-only: 0.511 / both: 0.622	global-only: 0.463 / both: 0.575 (Δ +0.112)	global-only: 0.509 / both: 0.400 (Δ −0.109)	N/A	n/a (feature-set ablation, same classifier family)	not separately benchmarked	Hypothesis supported on this val set: adding recent-window features improved F1 by +0.112 and reduced false-END rate by 0.109 and false-CONTINUE rate by 0.114 — all three moved in the favorable direction simultaneously. Kept in the default feature set.
EXP-004b	Source-aware vs. random split comparison	Because there are no speaker/session IDs, dataset (source/voice-engine) is the strongest available grouping variable; a random split may let a model learn source-specific acoustic shortcuts, making random-split evaluation look artificially better than source-aware evaluation.	Code implemented & unit-tested (synthetic grouped records) — NOT yet run on the real shard. src/turn_detector/splits.py verified correct; Phase 3/4's actual runs used a plain random dev/val split (not source-aware) for simplicity given the small samples involved — the random-vs-source-aware accuracy-gap comparison itself is still open.							Deferred
EXP-005	Hybrid acoustic + Whisper representation	Combining cheap classical features (fast, precise about energy/silence timing) with Whisper embeddings (semantic/prosodic richness) may outperform either alone, since they likely capture complementary error patterns — to be checked via error-mode overlap analysis, not assumed.	NOT RUN — depends on EXP-004 (Whisper, now run — see above), candidate next experiment.							Deferred to Phase 5
EXP-006	Temporal decision logic (hysteresis/debounce)	Raw per-invocation P(end) will be noisy call-to-call (especially if re-invoked on a growing buffer as the reference design does); adding a smoothing/hysteresis rule should reduce false END flips without meaningfully increasing true-END latency.	NOT RUN — depends on having a working classifier evaluated on real streaming-style sequences first.							Deferred
EXP-007	Hinglish / filler robustness pass	Contingent on EXP-000: if genuine Hinglish content is confirmed present, targeted analysis + augmentation/reweighting on those cases should close part of the accuracy gap on code-switched input. Filler-label supervision (midfiller/endfiller) is confirmed only populated for a subset of data sources (chirp3_1/chirp3_2 in the sample inspected), which constrains how this can be trained.	NOT RUN — blocked on EXP-000 and real dataset access.							Deferred
EXP-008	Latency/model-size optimization	The winning model from EXP-003–007 is likely oversized for the accuracy it needs; quantization/distillation/pruning should recover most of the accuracy at a fraction of the CPU latency and model size — directly comparable against the confirmed reference numbers (8MB int8 CPU version, 32MB fp32 GPU version, ~65ms Pipecat Cloud inference).	NOT RUN — depends on having a trained model first.							Deferred

New in Phase 2 — a "temporal/recent-window features" sub-experiment (part of EXP-002's scope per the Phase 2 brief §8): features.py's extract_recent_window_features() computes RMS mean/slope over the last 100/250/500/750/1000ms of each clip, with explicit *_available flags for clips shorter than a given window (never silently zero-filled/padded). Unit-tested (test_recent_window_features_*, 2/2 pass) for correct availability-flagging behavior. Whether recent-window features actually outperform whole-clip statistics is an empirical question that needs real data — the code to test it exists and is verified correct, the finding does not exist yet.

Notes

• Numbering note (updated Phase 4): the temporal/recent-window feature comparison is labeled EXP-003b (not EXP-003, which is Whisper Tiny — now actually run and reported as EXP-004). The source-aware vs. random split comparison is labeled EXP-004b to avoid colliding with the Whisper row. Numbering diverges slightly from any single phase brief's own local numbering by design, to keep one consistent ID per experiment across the whole project.
• EXP-000 is new since the first Phase-1 draft: it exists specifically because we confirmed the dataset has no transcripts and no code-switch label, so Hinglish coverage is currently unverified and must be checked empirically before EXP-007 can be meaningfully planned. Still not run — would need ASR (e.g. Whisper) over the Hindi-tagged clips in the real sample now available (data/raw/phase3_sample/), which is feasible next since real audio now exists in-sandbox, but wasn't part of this phase's scope.
• Experiments will be run in order, but a later experiment may be skipped if an earlier one already provides conclusive evidence against it — the brief explicitly says not to assume the final architecture before seeing data.
• "Premature-END rate" and "Endpoint latency" columns: per docs/INITIAL_ANALYSIS.md §7, this dataset does not support true wall-clock endpoint-latency measurement (no conversation-level timestamps). Populated rows use false_end_rate (premature-endpoint proxy, computable) and mark endpoint latency N/A as documented.
• No row in this table will be edited to look better after the fact. If a result is worse than expected, that's the result, and it drives the next hypothesis. EXP-001's 0.567 accuracy / 0.400 F1 on real validation data is a weak result and is reported as such, not smoothed over.

Status as of end of Phase 4

EXP-004 (Whisper Tiny frozen encoder) run successfully via the generated Colab notebook (notebooks/EXP004_whisper_baseline.ipynb), on a fresh 250-clip HF-streaming sample (75-clip val), independent of Phase 3's 300-clip sample. Real artifacts saved: artifacts/exp004/.

Decision (Phase 4, per the A/B/C framework): Option A — strong improvement. ΔF1 = +0.118, false-END rate improved by −0.143, and the improvement concentrates specifically on the filler-associated failure mode Phase 3's error analysis identified (filler-slice F1 0.722 vs. acoustic's 0.529) — a coherent, multi-signal result, not a single-metric artifact. Caveated honestly: val n=75 is modest, the sample differs from Phase 3's exact clips (same dataset/method), and the no_filler_known slice (n=7) is too small to trust and is excluded from the "supports the hypothesis" claim.