# Tiny Hinglish Turn Detector

An audio-native, low-latency **HOLD vs END** classifier for voice agents, built
for the Shiprocket Data Scientist challenge. The model is evaluated only at VAD
pause checkpoints: `HOLD` means “keep listening”; `END` means “the agent may
respond.”

The packaged artifact is a development preview: a 151,812-parameter causal
TinyTCN using the most recent four seconds of audio. It exports to a 625,431-byte
FP32 ONNX file and runs waveform-to-probability inference at 1.63 ms p95 on the
measured Apple-arm CPU setup. It does **not** beat the acoustic logistic baseline
on the current IID development split, and it has not been evaluated on the
official test or on verified Hinglish speech. The repository is submission-grade
engineering evidence, not a production-readiness claim.

## Current evidence, at a glance

| Item | Status | What is actually established |
|---|---|---|
| Upstream train snapshot | Partial locally | 1 of 83 Parquet shards audited: 3,265 rows, 6.857 hours |
| Data integrity on that shard | Measured | 3,265/3,265 valid; no exact-audio, ID, or label-conflict duplicates |
| Split integrity | Measured, best effort | IID and source-held-out manifests pass observed-key crossing checks; all 3,265 groups are singletons because speaker/conversation/voice IDs are absent |
| Interpretable development baseline | Measured, partial-shard only | 10 acoustic statistics: AP 0.7310, AUROC 0.7522, END recall 0.1313 at FPR 0.0181 |
| TinyTCN preview | Measured, partial-shard only | 151,812 parameters: AP 0.6881, AUROC 0.7390, END recall 0.0625 at FPR 0.0181 |
| Paired comparison | Baseline remains winner | TinyTCN minus baseline: AP −0.0429 (95% bootstrap CI −0.1195 to 0.0352); constrained recall −0.0688 (−0.1330 to −0.0122) |
| Domain-shift stress | Measured, confounded | Source-held-out AP 0.5500/0.5368 for TinyTCN/baseline; both near chance in ranking, with wide nine-source bootstrap intervals |
| Runtime/export | Measured | 611 KiB FP32 ONNX; exact PyTorch/ONNX parity; ONNX p95 0.956 ms neural-only and 1.629 ms waveform-to-probability |
| Controller integration | Synthetic only | Metadata-bound 8-checkpoint/3-turn fixture emits 3 response edges with 0 duplicates; not model or product-quality evidence |
| Code quality gate | Passed | Ruff 0.15.16; 136/136 automated tests passed via `unittest` discovery |
| Accuracy/generalization | **Not established** | No full-corpus run, collected Hinglish recordings, real conversation replay, or official-test evaluation |
| Publication tooling | Guarded | Hash-verified HF publisher plus curated Kaggle/GitHub upload builders; verify each remote runtime separately |

The official test set remains deliberately sealed. The preview is real
development evidence, but validation was reused for threshold selection,
checkpoint selection, and bounded follow-up experiments. It is therefore
adaptive development evidence—not an independent estimate. See
[REPORT.md](REPORT.md) for the full reasoning and [MODEL_CARD.md](MODEL_CARD.md)
for the release contract.

The acoustic baseline is the first real-data development result: logistic
regression on ten waveform statistics, trained on all 2,939 train rows and
scored on all 326 validation rows in the single audited shard. At a threshold
selected on that same validation split for `FPR <= 0.02`, it has 3 FP, 21 TP,
163 TN, and 139 FN (`FPR 0.0181`, END recall `0.1313`, AUROC `0.7522`, AP
`0.7310`). The selected TinyTCN reaches 3 FP, 10 TP, 163 TN, and 150 FN at
threshold `0.7410008` on the same split (`FPR 0.0181`, END recall `0.0625`,
AUROC `0.7390`, AP `0.6881`, Brier `0.2133`). Its FPR bootstrap interval is
`[0, 0.0407]`, so this sample does not establish a population 2% interruption
bound. Neither result is independent-test or Hinglish evidence.

Because that baseline is the evidence winner, its 10-weight JSON artifact is
packaged beside the neural model under `reference_models/`; the Gradio Space
continues to demonstrate the ONNX TinyTCN and labels that choice explicitly.

## Why this design

Turn detection is asymmetric. A false `END` interrupts the user and can derail
an entire transaction; a false `HOLD` adds latency. Accuracy at an arbitrary
0.5 threshold therefore is not the primary target. The selection criterion is
**maximum END recall subject to a false-positive (false-interruption) budget**,
with FPR reported at 1%, 2%, and 5% operating points.

The production boundary is also explicit:

```text
16 kHz audio suffix
  -> 80-bin log-mel frontend
  -> causal depthwise-separable TinyTCN
  -> mask-aware attentive mean/std pooling
  -> END probability + optional filler auxiliary heads
  -> calibrated SPEAKING / HOLD / END controller
```

The selected preview has 151,812 parameters, six 128-channel blocks, and a
four-second suffix. Its convolutions are causal; final attentive pooling makes
this a suffix-level classifier, not a state-cached streaming network. The
repository retains a larger 396,164-parameter/eight-second configuration as a
proposed full-data experiment, not as the packaged model. The controller—not
the neural network—owns minimum silence, threshold relaxation, debounce, and
maximum wait.

No ASR transcript is required at inference. That keeps the path small and
avoids coupling endpoint latency to transcription, while the auxiliary
`midfiller` and `endfiller` objectives encourage the representation to notice
exactly the failure cases the challenge emphasizes. A Whisper-tiny teacher
configuration is included as an experiment, not assumed to be the best serving
model.

## Repository map

| Path | Purpose |
|---|---|
| `src/turn_detection/data/` | Lazy Parquet/HF ingestion, schema normalization, audio audit, transitive grouping, split validation |
| `src/turn_detection/models/` | Canonical log-mel frontend, TinyTCN student, Whisper teacher, attentive pooling |
| `src/turn_detection/training/` | Lazy dataloaders, masked multitask loss, threshold calibration, product metrics, trainer |
| `src/turn_detection/runtime/` | NumPy preprocessing, ONNX inference, turn controller, replay evaluation |
| `configs/` | Reproducible student, teacher, and smoke configurations |
| `scripts/` | Download, audit, split, train, evaluate, export, benchmark, package, publish |
| `data/collection/` | Deterministic 900-recording Hinglish assignment plan; no recordings are included |
| `reports/` | Measured one-shard audit and split reports |
| `tests/` | Unit and integration coverage across data, model, metrics, runtime, and prompts |
| `deployment/kaggle/` | Flat, standalone ONNX inference templates for Kaggle Models |
| `upload-ready/` | Generated Kaggle and GitHub upload folders; never a source input |

## Reproduce the pipeline

Python 3.10–3.12 and [`uv`](https://docs.astral.sh/uv/) are recommended.
These commands target a full source checkout. Raw audio, downloaded Parquet,
processed manifests, and per-example predictions are intentionally not
redistributed; recreate them with the pinned download/audit commands below.

```bash
uv sync --extra all
uv run pytest
```

The curated public repository already includes the selected ONNX artifact, so
the demo resolves it without an environment variable:

```bash
uv sync --extra demo
uv run python app.py
```

Place the Hugging Face token in the ignored `.env` file; never commit it:

```bash
cp .env.example .env
# edit .env and set HF_TOKEN
```

### 1. Download the pinned train snapshot

```bash
bash scripts/download_dataset.sh
```

The script pins revision
`e564e2ac567f774d1880aa1db6ce97afb8c519b7`, resumes safely, and refuses to
declare success until all 83 Parquet shards are present.

### 2. Audit and create manifests

```bash
uv run python scripts/audit_dataset.py \
  data/raw/smart-turn-data-v3.2-train \
  --output data/processed/manifest.jsonl \
  --report artifacts/data_audit.json \
  --fail-on-error

uv run python scripts/prepare_splits.py \
  --input data/processed/manifest.jsonl \
  --output data/processed/splits.jsonl \
  --report artifacts/split_report.json \
  --split train=0.9 --split validation=0.1 \
  --stratify endpoint,language,synthetic,dataset
```

The audit hashes encoded audio and creates privacy-preserving linkage keys for
conversation, speaker, voice, recording, prompt, record, and repeated normalized
text. Union-find turns overlapping keys into transitive components; the split
tool assigns whole components and fails validation if any group, audio hash, or
metadata key crosses splits.

On the available shard, however, upstream rows expose no usable speaker,
conversation, or TTS-voice identity: all 3,265 base linkage groups used by the
IID split contain one row. (The separate stress split deliberately regroups rows
by 12 source-dataset values.)
The measured split is therefore best-effort row-disjoint and duplicate-safe,
**not speaker-disjoint**. Its grouped bootstrap is effectively a row bootstrap;
speaker/voice leakage remains unknown until richer identity metadata or a new
consented speaker-disjoint set exists.

For a harsher domain-shift test, add `--holdout-field dataset`. Interpret that
result carefully: in the currently inspected shard, source is strongly
confounded with language and synthetic status. Leave-one-source-out helpers are
also available in `turn_detection.data.splits`.

### 3. Prove the training path, then run the real experiment

```bash
# Fast generated-feature integration check; never report as model quality.
uv run python scripts/train.py --config configs/smoke.json --smoke-test

# Reproduce the selected one-shard preview (development evidence only).
uv run python scripts/train.py \
  --config configs/partial_shard_warmstart_lr3e4_5ep.json

# Proposed primary full-data student experiment, once all shards are audited.
uv run python scripts/train.py --config configs/tiny_tcn.yaml
```

Run the interpretable partial-shard sanity baseline independently:

```bash
uv run python scripts/run_baselines.py audio \
  --manifest data/processed/partial-iid-splits.jsonl \
  --source-root . \
  --output artifacts/partial-baseline \
  --epochs 1000 --fpr-budget 0.02
```

The trainer writes the resolved configuration, epoch history, calibrated
threshold, and best checkpoint. Deterministic mode seeds model initialization,
data order, and training. Missing filler labels stay missing and are masked from
their auxiliary BCE losses.

### 4. Evaluate without touching the official test

```bash
uv run python scripts/evaluate.py \
  --checkpoint artifacts/partial-shard-warmstart-lr3e4-5ep/best.pt \
  --source data/processed/partial-iid-splits.jsonl \
  --source-root . \
  --split validation \
  --output reports/partial_tinytcn_metrics.json
```

Evaluation reports confusion counts, FPR/FNR, AUROC, average precision, Brier
score, ECE/reliability bins, operating points under FPR budgets, and
language/source/synthetic slices. Turn-level interruption rates are emitted only
when genuine turn/conversation IDs exist; this shard supports only a clearly
labelled clip-normalized per-audio-hour proxy. Final comparisons need real
speaker- or conversation-cluster confidence intervals.

The selected checkpoint was chosen by average precision after earlier runs had
already been inspected on this validation set. Its threshold was also selected
on the same 326 examples. Reproduction should yield the stored numbers, but they
must not be treated as a fresh holdout result.

### 5. Freeze, unseal once, and evaluate the official test

Only an experiment trained with exact `run.status: final` can be frozen. The
freeze manifest hashes the checkpoint, selected config, source tree,
preprocessing, controller policy, split manifest, threshold, and pinned test
identity. `configs/final.yaml` and `artifacts/final/*` below are intentionally
future-candidate placeholders; create them only after completing the full-data
experiment. Only then:

```bash
uv run python scripts/freeze_candidate.py \
  --checkpoint artifacts/final/best.pt \
  --config configs/final.yaml \
  --output artifacts/final/frozen_manifest.json

UNSEAL_OFFICIAL_TEST=I_HAVE_FROZEN_MODEL_AND_THRESHOLD \
FROZEN_MANIFEST=artifacts/final/frozen_manifest.json \
  bash scripts/download_test_dataset.sh

uv run python scripts/evaluate.py \
  --checkpoint artifacts/final/best.pt \
  --source data/raw/smart-turn-data-v3.2-test \
  --dataset-id pipecat-ai/smart-turn-data-v3.2-test \
  --revision 0500378e8ed6d38e37b016e24d261e8e6c6a6859 \
  --split test \
  --allow-sealed-test \
  --frozen-manifest artifacts/final/frozen_manifest.json \
  --output artifacts/final/test_metrics.json
```

Do not tune after reading the test result. A new model requires a new untouched
test set or a clearly labeled exploratory result.

### 6. Export and benchmark

```bash
uv run python scripts/export_onnx.py \
  --checkpoint artifacts/final/best.pt \
  --output artifacts/final/model.onnx

uv run python scripts/benchmark.py \
  --model artifacts/final/model.onnx \
  --metadata artifacts/final/model_metadata.json \
  --threads 1 --batch-size 1 \
  --output artifacts/final/cpu_benchmark.json
```

ONNX export checks PyTorch/ONNX numerical parity. Runtime preprocessing,
I/O names, activation semantics, threshold, parameter count, and evidence scope
live in `model_metadata.json`; hashes, shapes, checkpoint/config/split bindings,
parity, and quantization status live in `export_manifest.json`. Static INT8 is
supported with representative calibration features; quantized scores require
threshold recalibration before release.

The packaged preview is
`artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx`: 625,431 bytes,
151,812 parameters, and FP32 maximum absolute parity error `0.0`. On the
recorded Apple-arm/one-thread runs, ONNX neural-only p50/p95/p99 is
0.919/0.956/0.974 ms; four-second waveform-to-probability p50/p95/p99 is
1.572/1.629/1.651 ms. These are local implementation measurements, not a
deployment SLA.

### 7. Run the demo and exercise the controller

```bash
TURN_MODEL_PATH=artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \
  uv run python app.py

uv run python scripts/replay_stream.py \
  --input data/collection/controller_replay_fixture.jsonl \
  --output reports/controller_replay_integration.jsonl \
  --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \
  --evidence-scope synthetic_integration
```

The exported metadata binds the endpoint threshold and the complete controller
policy. `END` is latched until new speech or an explicit reset; only the first
END transition sets `emit_response=true`, preventing duplicate agent replies.
The bundled replay fixture is hand-authored integration data: its probabilities,
targets, and latency values are synthetic. It validates policy binding,
threshold relaxation, timeout, END latching, and edge emission only—it is not
model, conversation, latency, or product-quality evidence. Replace its input
with private, human-annotated VAD checkpoint logs for a real sequence study.

If weights are absent, the demo displays a conspicuous heuristic fallback. That
fallback validates only the UI and must never be cited as a model result. See
[docs/demo.md](docs/demo.md) for the pause-checkpoint contract.

### 8. Build and publish a guarded release

```bash
uv run python scripts/build_release.py \
  --model artifacts/final/model.onnx \
  --metadata artifacts/final/model_metadata.json \
  --metrics artifacts/final/test_metrics.json \
  --frozen-manifest artifacts/final/frozen_manifest.json

# Dry validation first.
uv run python scripts/publish_hf.py --username suvradeepp

# Creates/uploads the model repo and Gradio Space only after identity validation.
uv run python scripts/publish_hf.py --username suvradeepp --execute
```

For a personal account without Hugging Face PRO, the publisher requests the
free-account-compatible ZeroGPU tier (`--space-hardware zero-a10g`) only when it
must create a new Space. Hugging Face currently gates new personal Gradio CPU
Spaces behind PRO even when CPU Basic has no hourly charge. ZeroGPU eligibility
still requires a verified account in good standing and an available hosted
Space slot. Paid accounts may override this with `--space-hardware cpu-basic`.

The release builder refuses smoke checkpoints or a release without measured
test metrics unless `--allow-development-artifact` is passed explicitly. That
override must remain visibly development-only. For a reviewable partial-data
preview, pass that flag to the builder and `--allow-development-release` to both
publisher invocations. Development validation metrics are packaged as
`development_metrics.json`, never `test_metrics.json`.

The exact development-preview package command is:

```bash
uv run python scripts/build_release.py \
  --model artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \
  --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \
  --metrics reports/partial_tinytcn_metrics.json \
  --output release \
  --allow-development-artifact \
  --include-synthetic-controller-replay

uv run python scripts/publish_hf.py \
  --username suvradeepp \
  --release-dir release \
  --allow-development-release

uv run python scripts/build_upload_folders.py \
  --release-dir release \
  --output upload-ready \
  --allow-development-release
```

`upload-ready/kaggle-model/` is a flat drag-and-drop ONNX bundle.
`upload-ready/github-repository/` is a public-repository allowlist containing
source, CI, tests, aggregate evidence, ONNX weights, and the two checkpoints
needed to reproduce the selected warm start. It excludes credentials, caches,
raw/processed data, per-example predictions, stale releases, and publish
receipts. Keep the Kaggle variation private until the derived-weight rights
review described in `NOTICE` and `MODEL_CARD.md` is resolved.

Add `--execute` only after the dry validation succeeds. The publisher verifies
the authenticated owner and every packaged hash before creating or updating the
remote model repository and Space. Its receipt binds exact remote HEAD commits,
requires a visible build/start transition for a new Space upload, and records
the final `RUNNING` stage. Because the installed Hub API does not expose the
commit actually serving behind a Space runtime, the receipt explicitly records
`runtime_commit_bound=false` instead of overstating that guarantee.

The release hash-binds the 2.7 MB split manifest but intentionally does not
redistribute it because it contains upstream text and record identifiers. The
public package carries deterministic preparation code, the pinned revision,
manifest SHA-256, and aggregate reports; per-example rows stay private.

## Hinglish evaluation protocol

The upstream corpus is multilingual, but a `hin` or `eng` tag is not evidence
of within-utterance code-switching. This repository therefore includes a
separate, consent-first Shiprocket-domain protocol:

- 30 speakers × 30 assignments = 900 intended recordings;
- 450 `HOLD` and 450 `END` targets;
- 21/4/5 speaker-disjoint train/dev/test speakers;
- minimal pairs with natural 0.3–2.0 second internal pauses plus roleplay;
- three annotators answer whether responding at the checkpoint feels like an
  interruption; uncertain votes remain disagreement, not forced labels.

Only the assignments and protocol exist today—**no participant audio has been
collected**. Do not infer Hinglish performance until that benchmark is recorded
and evaluated. See [docs/data_collection_protocol.md](docs/data_collection_protocol.md)
and [DATA_CARD.md](DATA_CARD.md).

## What “good enough” means

A candidate is releaseable only after all of these gates pass:

1. all 83 train shards audited with zero unresolved hard validation errors;
2. group-safe validation and source/speaker stress tests completed;
3. threshold frozen under a predeclared false-interruption budget;
4. real Hinglish minimal-pair test collected with consent and held out by speaker;
5. official test opened once and reported without subsequent tuning;
6. preprocessing parity, ONNX parity, batch-1 CPU latency, and controller replay measured;
7. failure slices reviewed for language, source, synthetic/human, filler, duration,
   device/noise, gender/region where consented, and uncertainty;
8. upstream data/model redistribution terms reviewed before publishing weights.

The current repository passes export/runtime engineering gates and supplies
honest bounded development evidence. It does not pass the full-data, Hinglish,
official-test, sequence-replay, or rights-review gates.

## Documentation

- [Technical report](REPORT.md)
- [Model card](MODEL_CARD.md)
- [Data card](DATA_CARD.md)
- [Experiment registry](docs/experiment_registry.md)
- [Failure-analysis protocol](docs/failure_analysis.md)
- [Hinglish collection protocol](docs/data_collection_protocol.md)
- [Demo/runtime contract](docs/demo.md)

## License and responsible use

Repository-authored code is Apache-2.0; see [LICENSE](LICENSE) and
[NOTICE](NOTICE). The upstream
dataset card did not provide an explicit dataset license at the inspected
revision. This repository does not redistribute its audio. Apache-2.0 on the
code does **not** grant rights to upstream audio, recordings collected under the
local protocol, speaker likeness, customer data, or the development model
weights. The preview weights carry no license grant pending an upstream-rights
review; review data and derived-weight terms before use or redistribution.

This system decides conversational timing; it should not be used for speaker
identification, emotion inference, surveillance, or consequential decisions.