docs/PHASE2_REPORT.md

Phase 2 Report

Dataset

• Accessible rows in this sandbox: 0 of 270,946 (train) / 31,473 (test). Confirmed, tested, and documented blocker: bash network egress returns HTTP 403 (x-deny-reason: host_not_allowed) for huggingface.co, the datasets/huggingface_hub Python packages are not installed, and pip install cannot reach PyPI from this sandbox either. load_dataset(..., streaming=True) was attempted per the Phase 2 instructions (src/turn_detector/data.py::load_hf_dataset) and fails with a clear, caught DatasetAccessError rather than crashing or hanging — verified by actually running scripts/create_dev_subset.py, which exits cleanly with exit code 1 and a diagnostic message.
• Development subset size: 0 (not yet created). scripts/create_dev_subset.py is complete and implements reproducible stratified sampling (seed-based, streaming-compatible, stratifies on endpoint_bool, language, dataset, synthetic, midfiller, endfiller, and a derived duration_bucket), but cannot run without dataset access.
• Distributions: everything knowable without full access is already in docs/INITIAL_ANALYSIS.md and docs/LANGUAGE_ANALYSIS.md from the n=79 hand-parsed HF-viewer sample (not re-derived here to avoid duplication). No new distributional data was obtained this phase — further datasets-server API endpoints were attempted and blocked by this session's fetch-tool URL allowlist (see docs/LANGUAGE_ANALYSIS.md, top).

Baseline 1 (Energy/Silence)

• Architecture: trailing-silence-duration rule against a tunable RMS threshold and duration threshold. Implemented in src/turn_detector/baseline_energy.py, including a tune_threshold() grid-search function that must be called with dev/val data only (the brief's "never tune on test" rule is stated in the docstring; not mechanically enforceable, so it's a process discipline, documented as such).
• Metrics: not available. No real data to evaluate against.
• Latency: not measured. evaluation.py::measure_inference_latency is implemented and generic (works on any predict_fn); running it against this baseline on real clips is a Phase 3 action item.
• What IS verified: 5 unit tests against synthetic audio pass — correctly predicts END given long trailing silence, CONTINUE given none, and tune_threshold selects a sensible threshold and achieves >80% accuracy on cleanly-separable synthetic data (a sanity check on the mechanism, not a claim about real-world accuracy).

Baseline 2 (Classical features + Logistic Regression)

• Features: RMS/ZCR/spectral centroid/bandwidth/rolloff/13 MFCCs (summary stats: mean/std/min/max each) + energy slope + trailing silence duration + duration, all implemented from scratch in numpy/scipy (src/turn_detector/features.py) since librosa isn't installed here and can't be installed without network access. This is a real engineering tradeoff, stated in the module docstring — not a compromise on feature quality, since MFCC math is standard (mel filterbank + DCT) and unit-tested directly (test_mfcc_shape and related tests verify finite, correctly-shaped output).
• Classifier: sklearn Pipeline (median-impute → standardize → LogisticRegression, class-balanced). src/turn_detector/baseline_classifier.py.
• Metrics: not available. No real data.
• Latency/model size: not measured against real data, but the plumbing exists: param_count() and model_size_bytes() are implemented and unit-tested (return positive, sane values on a synthetic-data fit).
• What IS verified: fits and predicts correctly on synthetic cleanly-separable data (>70% accuracy — expected for a sanity check, not a real-world number), exposes inspectable top-weighted features via top_features() for future error analysis.

Temporal feature experiment

• Whether recent-window features help: not yet determined — needs real data.
• What's implemented: extract_recent_window_features() computes RMS mean and slope over the last 100/250/500/750/1000ms of each clip, with an explicit <window>_available flag for clips shorter than a given window (never silently zero-padded, which would fabricate signal that isn't there). Unit-tested for correct availability-flagging on both short and long synthetic clips.
• This is real, working infrastructure to answer the question the Phase 2 brief asks — the answer itself doesn't exist yet.

Source-aware evaluation

• Random vs. source-aware split comparison: not yet run — needs real labeled data to compare accuracy on.
• What's implemented and verified: src/turn_detector/splits.py provides split_random (may leave dataset-source overlap between train and val) and split_source_aware (group-disjoint by dataset source, verified by a passing unit test to produce zero source overlap). Both operate on lightweight metadata records, not audio, so they're cheap to run against a real dev subset once one exists.
• The docstrings are explicit that this is a source-disjoint split, not a speaker-disjoint split (the dataset has no speaker IDs — see docs/INITIAL_ANALYSIS.md §4) — this distinction is preserved end-to-end rather than glossed over.

Language analysis

New document: docs/LANGUAGE_ANALYSIS.md. Structured exactly as requested — Known (schema facts + hand-counted n=79 sample facts, each sourced), Inferred (reasonable but unconfirmed readings, e.g. "Hindi content in this dataset may skew synthetic" based on 6/6 sampled Hindi rows being TTS-sourced — explicitly flagged as a weak, small-n signal), and Not measurable from available metadata (whether code-switching is actually present; the true full-scale language distribution). No claim that the dataset "is Hinglish" merely because Hindi exists — the document is explicit that this remains unverified pending EXP-000 (an ASR-based content check), which requires real audio access this sandbox doesn't yet have.

Error analysis

Not produced this phase. There is nothing to analyze — no model has been evaluated against real data, so there are no real misclassifications to inspect. docs/ERROR_ANALYSIS.md was deliberately not created with placeholder content; creating it now would either be empty or would invite fabricated-looking failure-mode claims, which the project rules explicitly prohibit. It will be created once EXP-001/EXP-002 actually run against real data.

Key conclusion

"How much can we achieve without Whisper?" — cannot be answered yet. This is stated plainly rather than filled in with a plausible-sounding guess. Both non-Whisper baselines are fully implemented, correctly functioning (verified via 31 passing unit tests against synthetic audio), and ready to run — but "ready to run" and "evaluated" are different things, and only the latter would justify a real conclusion about whether Whisper Tiny is empirically necessary. Presenting a confident answer here without real-data results would violate this project's most basic rule ("never fabricate metrics").

What Phase 2 does establish, honestly:

• The dataset is well-understood at the schema/provenance level (Phase 1 + this phase's language analysis).
• The full non-Whisper pipeline is built, modular, and correctness-tested.
• The single blocking issue for getting real numbers is data access in this specific sandbox — not a design or methodology gap.

Recommended next experiment (exactly one)

Get a real, even if small, slice of actual audio into this sandbox, and run EXP-001 (energy/silence baseline) against it.　This is deliberately chosen over jumping to EXP-002 or EXP-000: EXP-001 is the cheapest possible real signal about whether this whole pipeline behaves sensibly on real data, requires the least amount of real audio to be informative, and directly unblocks tuning a defensible baseline before any heavier comparison (classical classifier, Whisper) is worth attempting. Concretely, this needs one of:

1. A person-provided upload of a small representative batch of real audio files (even a few hundred clips would be enough to run and sanity-check the full EXP-001 pipeline end-to-end), or
2. Continued small-scale sampling of individual signed audio URLs via the HF dataset viewer (web_fetch), accepting the same small-sample caveats already documented in docs/LANGUAGE_ANALYSIS.md, or
3. Running scripts/create_dev_subset.py and scripts/inspect_dataset.py in an environment with real network access to Hugging Face.