docs/PHASE3_REAL_AUDIO_VALIDATION.md

# Phase 3: Real-Audio Validation Report

## How real audio got into this sandbox

The user uploaded a real Parquet shard:
`/mnt/user-data/uploads/train-00000-of-00010.parquet` (464 MB, confirmed
via `du`), one of ten training shards of
`pipecat-ai/smart-turn-data-v3.2-train`. Verified as a genuine Parquet
file by its magic bytes (`PAR1` at both start and end) before doing
anything else.

**No `pyarrow`, `fastparquet`, or `datasets` library is installed in this
sandbox, and none are installable** — `pip`/`uv pip install pyarrow`
fails with the same `403 host_not_allowed` block that covers
`huggingface.co` (confirmed again this phase: `pypi.org` returns the
identical proxy denial). So reading this real file required writing a
**pure-Python Parquet reader from scratch**:
`src/turn_detector/parquet_reader.py`. This is the single biggest piece of
new engineering this phase, and it's worth documenting honestly, including
the real bugs it had and how they were caught — because "trust but verify"
matters more than usual when the verification code and the code under test
were written by the same process in the same session.

## What was built, and the real bugs caught while building it

1. **Thrift compact-protocol decoder** (generic struct/list/map/binary
   reader) — needed because Parquet's footer metadata is Thrift-encoded.
2. **Footer parser** — extracts schema, row groups, column chunk metadata
   (codec, sizes, offsets) from the file's trailing metadata block.
   **First real result**: this shard has **3,153 rows across 32 row
   groups**, not the ~27,000 I'd have guessed by linearly scaling
   270,946/10 — a good reminder not to estimate when you can just read the
   actual footer.
3. **Hand-written Snappy decompressor** (raw block format) — the shard's
   columns are all SNAPPY-compressed (confirmed from the footer, not
   assumed). Verified correct via round-trip: decompressed sizes matched
   the footer's declared `uncompressed_size` exactly for every page
   decoded.
4. **RLE/bit-packed hybrid decoder** for definition levels (needed because
   every column here is `OPTIONAL`/nullable, including scalars).
   - **Real bug #1, caught and fixed**: my first version hardcoded a
     1-bit definition-level width for every column. This happened to work
     for top-level scalar columns (`language`, `endpoint_bool`, etc.) but
     silently corrupted `audio.bytes`, which is nested two levels deep
     inside the `OPTIONAL` `audio` struct and therefore needs a 2-bit
     definition level (0 = audio missing, 1 = audio present but bytes
     null, 2 = value present). The symptom was concrete and easy to
     catch: every decoded `audio.bytes` value came back `None`, even
     though `language` for the same rows decoded correctly. Fixed by
     computing the true max-definition-level per column from a real
     schema-tree walk (`_build_max_def_levels`), not an assumption.
   - **Real bug #2, caught and fixed**: dictionary-page decoding only
     handled `BYTE_ARRAY` values; `spoken_text` (typed `INT32`, entirely
     null) hit a dictionary page and raised `ValueError: Unsupported
     dictionary value type: INT32`. Fixed by adding an INT32 dictionary
     decode path. Re-verified afterward: `spoken_text` decodes cleanly to
     100% `None` across the row group tested — consistent with the
     schema-level `dtype: null` fact already confirmed in Phase 1, now
     independently re-confirmed by actually decoding real values, not
     just trusting the declared schema.
5. **FLAC STREAMINFO parser** (`parse_flac_streaminfo`) — once
   `audio.bytes` decoded correctly, the first 4 bytes were `fLaC`: **the
   audio is FLAC, not raw WAV.** Rather than write a full FLAC decoder
   (LPC prediction + Rice coding — a much bigger undertaking), I parse
   just the 34-byte `STREAMINFO` metadata block to get sample rate,
   channel count, and total samples (hence duration) without decoding any
   audio — cheap enough to run across the entire 3,153-row shard in a few
   seconds. Cross-checked against `ffmpeg`'s own reported duration for a
   sample file: **exact match** (7.62775s both ways).
6. **`ffmpeg`-based audio decoding** for actual PCM samples (real decoding,
   not header-only): `ffmpeg` is already installed in this sandbox (unlike
   `soundfile`), so `src/turn_detector/audio_io.py` gained an `ffmpeg`
   subprocess fallback path, used transparently when `soundfile` is
   unavailable. Verified end-to-end: FLAC bytes → `ffmpeg` → WAV → loaded
   via the same fallback path → duration matches the STREAMINFO-derived
   duration exactly.

Every one of these was tested against the real file at each step, not
just written and assumed correct — see the tool-call history in this
conversation for the actual intermediate outputs (row counts, sample
values, byte-for-byte size checks) that caught bugs #1 and #2 above.

## Step 1: Full-shard inventory (real, all 3,153 rows)

Computed by fully decoding every scalar metadata column plus
STREAMINFO-derived duration for **every row in the shard** (not a sample —
this was cheap enough, ~5s, to just do exhaustively):

| Property | Value |
|---|---|
| Total rows | 3,153 |
| Corrupted/unreadable rows | **0** |
| `endpoint_bool` | 1,589 True (END) / 1,564 False (CONTINUE) — 50.4%/49.6%, matching the dataset authors' stated 50:50 design target almost exactly |
| Sample rate | 16,000 Hz for all 3,153 rows (uniform) |
| Channels | 1 (mono) for all 3,153 rows (uniform) |
| Duration | min 0.36s, max 30.0s, mean 7.73s, median 7.32s |
| `synthetic` | 2,575 True (82%) / 578 False (18%) |
| `midfiller` | 1,293 True / 1,187 False / **673 None** (null ≠ false — unavailable for those rows) |
| `endfiller` | 810 True / 1,670 False / **673 None** |

**Language distribution (real, full shard, 23 languages present):**
English dominates (789, 25%); Hindi (`hin`) = **132 rows (4.2%)** — a real
measurement, not the ~7.6% rough estimate from Phase 1's 79-row hand
sample (which was too small and non-random to trust for this — this is
exactly why that estimate was labeled provisional at the time).

**Source (`dataset` column) distribution — 12 distinct values, all now
seen** (Phase 1's 79-row sample only ever saw 6 of these):
`chirp3_1` (1,617), `chirp3_2` (852), `liva_1` (386), `midcentury_1` (108),
`mundo_1` (40), `rime_2` (39), `human_5` (35), `orpheus_grammar_1` (24),
`orpheus_endfiller_1` (19), `orpheus_midfiller_1` (13),
`chirp3_3_short` (11), `human_convcollector_1` (9). The `orpheus_*` and
`human_*` sources are new information this phase — not visible in Phase 1's
smaller sample.

## Step 2: Verify label mapping

Confirmed the same way documented in Phase 1 (upstream contribution guide:
`endpoint_bool=true`→END, `false`→CONTINUE), plus a real smoke test on 60
real clips this phase: mean trailing silence for `endpoint_bool=True`
clips was 0.300s vs. 0.114s for `endpoint_bool=False` clips — directionally
consistent with the documented semantics (not proof, but nothing suggests
an inverted or meaningless label).

## Step 3: Development sample materialized

Stratified reservoir sample (seed=42, `src/turn_detector/data.py`),
target 300, actual 300, stratified across `endpoint_bool` × `language` ×
`dataset` × `synthetic` × `midfiller` × `endfiller` × duration bucket.
Materialized as real 16kHz mono WAV files at
`data/raw/phase3_sample/audio/<id>.wav` + `data/raw/phase3_sample/metadata.csv`.
**300/300 clips converted successfully, 0 ffmpeg failures.**

## Steps 4-6: EXP-001 / EXP-002 / EXP-003(b) results

All **SMALL REAL-AUDIO VALIDATION** — 210 dev / 90 val clips (random
70/30 split, seed=42), threshold/model tuned on dev only, all metrics
below on held-out val:

| | EXP-001 (energy/silence) | EXP-002 (classical features + LR) | EXP-003b Δ (temporal features) |
|---|---|---|---|
| Accuracy | 0.567 | 0.622 | global-only 0.511 → both 0.622 |
| F1 | 0.400 | 0.575 | **+0.112** |
| False END rate | 0.309 | 0.400 | **−0.109** (improved) |
| False CONTINUE rate | 0.629 | 0.343 | **−0.114** (improved) |
| Params / size | 2 thresholds | 92 params / 2,920 bytes | same classifier, feature-set ablation |
| Latency (this machine) | negligible (rule-based) | mean 12.2ms / p95 22.4ms per clip | n/a |

**EXP-003b's hypothesis is supported on this validation set**: adding
recent-window (100–1000ms tail) features to the global whole-clip
statistics improved F1 by +0.112 and reduced *both* error rates
simultaneously — not just a precision/recall tradeoff in one direction.
This is a real, if small-sample, result in favor of temporal features
mattering for this task, consistent with what streaming-relevant intuition
would predict.

## Step 7: Slice / filler analysis (real, on EXP-002's val predictions)

- **By language**: only `eng` (n=22) met the `min_samples=10` bar in the
  90-clip val set; every other language slice (including Hindi, n=3 in
  val) was too small and excluded rather than reported as if meaningful.
  `eng`-only: accuracy 0.636, F1 0.600.
- **By synthetic flag**: non-synthetic (n=12): accuracy 0.583, F1 0.667.
  Synthetic (n=78): accuracy 0.628, F1 0.554. Interesting but n=12 for the
  non-synthetic slice is thin — flagged as exploratory, not conclusive.
- **Filler metadata**: 72/90 val rows had `midfiller` populated (18 `None`
  — correctly excluded, not treated as "no filler"). `has_midfiller`
  (n=39): F1 0.529. `no_midfiller` (n=33): F1 0.593 — filler-present clips
  were somewhat harder for EXP-002, consistent with the qualitative error
  pattern below.

## Step 8: Error analysis (real false END / false CONTINUE cases)

Full write-up: `docs/ERROR_ANALYSIS.md`. Headline, real, measured finding:
**16 of 22 (73%) false-END errors have a filler flag present**, versus
50% for false-CONTINUE errors — and false-END clips have longer mean
trailing silence (0.332s) than false-CONTINUE clips (0.217s). This is
exactly the failure mode the assessment brief names as the hard case:
a pause that acoustically resembles an ending but linguistically isn't
(filler-preceded), which an audio-only, transcript-blind model has no way
to catch beyond acoustic residue.

## Step 10/11: Decision

**Question:** is the acoustic-only approach promising enough to continue,
per the three-option framework?

**Answer: Option B** — acoustic-only provides useful signal but is
insufficient. Evidence:

- EXP-002 (0.622 accuracy / 0.575 F1) clearly beats EXP-001 (0.567 / 0.400)
  and both clearly beat chance-level performance on a roughly balanced
  task, so acoustic features are picking up **real** signal — this isn't
  noise.
- But 0.622 accuracy on a near-50/50 binary task is not close to
  production-usable on its own, and the error analysis identifies a
  specific, well-understood, real ceiling: **filler-word-preceded pauses
  are systematically harder**, and no acoustic-only feature set can fully
  resolve that without something closer to lexical/semantic content — the
  exact thing Whisper Tiny's speech representations (not just its
  transcription output) are trained to encode implicitly, which is the
  actual empirical justification the brief asked for, now grounded in a
  real measured failure mode rather than "the brief suggested it."
- EXP-003b gives a concrete, real reason to believe **temporal/recency
  features help** (+0.112 F1), which is a design element worth carrying
  forward into whichever architecture comes next, Whisper-based or not.

This is 90 validation clips from one of ten shards — a first real signal,
not a final verdict. It's a directionally clear enough result to justify
moving toward EXP-003 (Whisper Tiny) as the next architecture to actually
try, while keeping EXP-002 as the benchmark it needs to beat.

## One recommended next experiment

**EXP-003: Whisper Tiny encoder representations + the same lightweight
classifier head, evaluated on the identical 210/90 dev/val split used
here**, so the comparison against EXP-002's 0.575 F1 / 0.400 false-END
rate / 0.343 false-CONTINUE rate is apples-to-apples. This directly tests
whether Whisper's representations close the filler-word gap identified in
error analysis, which is the specific, evidenced reason to expect it might
help — not just "the brief suggested Whisper Tiny."