SOURCE_README.md

Tiny Hinglish Turn Detector

An audio-native, low-latency HOLD vs END classifier for voice agents, built for the Shiprocket Data Scientist challenge. The model is evaluated only at VAD pause checkpoints: HOLD means “keep listening”; END means “the agent may respond.”

The packaged artifact is a development preview: a 151,812-parameter causal TinyTCN using the most recent four seconds of audio. It exports to a 625,431-byte FP32 ONNX file and runs waveform-to-probability inference at 1.63 ms p95 on the measured Apple-arm CPU setup. It does not beat the acoustic logistic baseline on the current IID development split, and it has not been evaluated on the official test or on verified Hinglish speech. The repository is submission-grade engineering evidence, not a production-readiness claim.

Current evidence, at a glance

Item	Status	What is actually established
Upstream train snapshot	Partial locally	1 of 83 Parquet shards audited: 3,265 rows, 6.857 hours
Data integrity on that shard	Measured	3,265/3,265 valid; no exact-audio, ID, or label-conflict duplicates
Split integrity	Measured, best effort	IID and source-held-out manifests pass observed-key crossing checks; all 3,265 groups are singletons because speaker/conversation/voice IDs are absent
Interpretable development baseline	Measured, partial-shard only	10 acoustic statistics: AP 0.7310, AUROC 0.7522, END recall 0.1313 at FPR 0.0181
TinyTCN preview	Measured, partial-shard only	151,812 parameters: AP 0.6881, AUROC 0.7390, END recall 0.0625 at FPR 0.0181
Paired comparison	Baseline remains winner	TinyTCN minus baseline: AP −0.0429 (95% bootstrap CI −0.1195 to 0.0352); constrained recall −0.0688 (−0.1330 to −0.0122)
Domain-shift stress	Measured, confounded	Source-held-out AP 0.5500/0.5368 for TinyTCN/baseline; both near chance in ranking, with wide nine-source bootstrap intervals
Runtime/export	Measured	611 KiB FP32 ONNX; exact PyTorch/ONNX parity; ONNX p95 0.956 ms neural-only and 1.629 ms waveform-to-probability
Controller integration	Synthetic only	Metadata-bound 8-checkpoint/3-turn fixture emits 3 response edges with 0 duplicates; not model or product-quality evidence
Code quality gate	Passed	Ruff 0.15.16; 136/136 automated tests passed via unittest discovery
Accuracy/generalization	Not established	No full-corpus run, collected Hinglish recordings, real conversation replay, or official-test evaluation
Publication tooling	Guarded	Hash-verified HF publisher plus curated Kaggle/GitHub upload builders; verify each remote runtime separately

The official test set remains deliberately sealed. The preview is real development evidence, but validation was reused for threshold selection, checkpoint selection, and bounded follow-up experiments. It is therefore adaptive development evidence—not an independent estimate. See REPORT.md for the full reasoning and MODEL_CARD.md for the release contract.

The acoustic baseline is the first real-data development result: logistic regression on ten waveform statistics, trained on all 2,939 train rows and scored on all 326 validation rows in the single audited shard. At a threshold selected on that same validation split for FPR <= 0.02, it has 3 FP, 21 TP, 163 TN, and 139 FN (FPR 0.0181, END recall 0.1313, AUROC 0.7522, AP 0.7310). The selected TinyTCN reaches 3 FP, 10 TP, 163 TN, and 150 FN at threshold 0.7410008 on the same split (FPR 0.0181, END recall 0.0625, AUROC 0.7390, AP 0.6881, Brier 0.2133). Its FPR bootstrap interval is [0, 0.0407], so this sample does not establish a population 2% interruption bound. Neither result is independent-test or Hinglish evidence.

Because that baseline is the evidence winner, its 10-weight JSON artifact is packaged beside the neural model under reference_models/; the Gradio Space continues to demonstrate the ONNX TinyTCN and labels that choice explicitly.

Why this design

Turn detection is asymmetric. A false END interrupts the user and can derail an entire transaction; a false HOLD adds latency. Accuracy at an arbitrary 0.5 threshold therefore is not the primary target. The selection criterion is maximum END recall subject to a false-positive (false-interruption) budget, with FPR reported at 1%, 2%, and 5% operating points.

The production boundary is also explicit:

16 kHz audio suffix
  -> 80-bin log-mel frontend
  -> causal depthwise-separable TinyTCN
  -> mask-aware attentive mean/std pooling
  -> END probability + optional filler auxiliary heads
  -> calibrated SPEAKING / HOLD / END controller

The selected preview has 151,812 parameters, six 128-channel blocks, and a four-second suffix. Its convolutions are causal; final attentive pooling makes this a suffix-level classifier, not a state-cached streaming network. The repository retains a larger 396,164-parameter/eight-second configuration as a proposed full-data experiment, not as the packaged model. The controller—not the neural network—owns minimum silence, threshold relaxation, debounce, and maximum wait.

No ASR transcript is required at inference. That keeps the path small and avoids coupling endpoint latency to transcription, while the auxiliary midfiller and endfiller objectives encourage the representation to notice exactly the failure cases the challenge emphasizes. A Whisper-tiny teacher configuration is included as an experiment, not assumed to be the best serving model.

Repository map

Path	Purpose
src/turn_detection/data/	Lazy Parquet/HF ingestion, schema normalization, audio audit, transitive grouping, split validation
src/turn_detection/models/	Canonical log-mel frontend, TinyTCN student, Whisper teacher, attentive pooling
src/turn_detection/training/	Lazy dataloaders, masked multitask loss, threshold calibration, product metrics, trainer
src/turn_detection/runtime/	NumPy preprocessing, ONNX inference, turn controller, replay evaluation
configs/	Reproducible student, teacher, and smoke configurations
scripts/	Download, audit, split, train, evaluate, export, benchmark, package, publish
data/collection/	Deterministic 900-recording Hinglish assignment plan; no recordings are included
reports/	Measured one-shard audit and split reports
tests/	Unit and integration coverage across data, model, metrics, runtime, and prompts
deployment/kaggle/	Flat, standalone ONNX inference templates for Kaggle Models
upload-ready/	Generated Kaggle and GitHub upload folders; never a source input

Reproduce the pipeline

Python 3.10–3.12 and uv are recommended. These commands target a full source checkout. Raw audio, downloaded Parquet, processed manifests, and per-example predictions are intentionally not redistributed; recreate them with the pinned download/audit commands below.

uv sync --extra all
uv run pytest

The curated public repository already includes the selected ONNX artifact, so the demo resolves it without an environment variable:

uv sync --extra demo
uv run python app.py

Place the Hugging Face token in the ignored .env file; never commit it:

cp .env.example .env
# edit .env and set HF_TOKEN

1. Download the pinned train snapshot

bash scripts/download_dataset.sh

The script pins revision e564e2ac567f774d1880aa1db6ce97afb8c519b7, resumes safely, and refuses to declare success until all 83 Parquet shards are present.

2. Audit and create manifests

uv run python scripts/audit_dataset.py \
  data/raw/smart-turn-data-v3.2-train \
  --output data/processed/manifest.jsonl \
  --report artifacts/data_audit.json \
  --fail-on-error

uv run python scripts/prepare_splits.py \
  --input data/processed/manifest.jsonl \
  --output data/processed/splits.jsonl \
  --report artifacts/split_report.json \
  --split train=0.9 --split validation=0.1 \
  --stratify endpoint,language,synthetic,dataset

The audit hashes encoded audio and creates privacy-preserving linkage keys for conversation, speaker, voice, recording, prompt, record, and repeated normalized text. Union-find turns overlapping keys into transitive components; the split tool assigns whole components and fails validation if any group, audio hash, or metadata key crosses splits.

On the available shard, however, upstream rows expose no usable speaker, conversation, or TTS-voice identity: all 3,265 base linkage groups used by the IID split contain one row. (The separate stress split deliberately regroups rows by 12 source-dataset values.) The measured split is therefore best-effort row-disjoint and duplicate-safe, not speaker-disjoint. Its grouped bootstrap is effectively a row bootstrap; speaker/voice leakage remains unknown until richer identity metadata or a new consented speaker-disjoint set exists.

For a harsher domain-shift test, add --holdout-field dataset. Interpret that result carefully: in the currently inspected shard, source is strongly confounded with language and synthetic status. Leave-one-source-out helpers are also available in turn_detection.data.splits.

3. Prove the training path, then run the real experiment

# Fast generated-feature integration check; never report as model quality.
uv run python scripts/train.py --config configs/smoke.json --smoke-test

# Reproduce the selected one-shard preview (development evidence only).
uv run python scripts/train.py \
  --config configs/partial_shard_warmstart_lr3e4_5ep.json

# Proposed primary full-data student experiment, once all shards are audited.
uv run python scripts/train.py --config configs/tiny_tcn.yaml

Run the interpretable partial-shard sanity baseline independently:

uv run python scripts/run_baselines.py audio \
  --manifest data/processed/partial-iid-splits.jsonl \
  --source-root . \
  --output artifacts/partial-baseline \
  --epochs 1000 --fpr-budget 0.02

The trainer writes the resolved configuration, epoch history, calibrated threshold, and best checkpoint. Deterministic mode seeds model initialization, data order, and training. Missing filler labels stay missing and are masked from their auxiliary BCE losses.

4. Evaluate without touching the official test

uv run python scripts/evaluate.py \
  --checkpoint artifacts/partial-shard-warmstart-lr3e4-5ep/best.pt \
  --source data/processed/partial-iid-splits.jsonl \
  --source-root . \
  --split validation \
  --output reports/partial_tinytcn_metrics.json

Evaluation reports confusion counts, FPR/FNR, AUROC, average precision, Brier score, ECE/reliability bins, operating points under FPR budgets, and language/source/synthetic slices. Turn-level interruption rates are emitted only when genuine turn/conversation IDs exist; this shard supports only a clearly labelled clip-normalized per-audio-hour proxy. Final comparisons need real speaker- or conversation-cluster confidence intervals.

The selected checkpoint was chosen by average precision after earlier runs had already been inspected on this validation set. Its threshold was also selected on the same 326 examples. Reproduction should yield the stored numbers, but they must not be treated as a fresh holdout result.

5. Freeze, unseal once, and evaluate the official test

Only an experiment trained with exact run.status: final can be frozen. The freeze manifest hashes the checkpoint, selected config, source tree, preprocessing, controller policy, split manifest, threshold, and pinned test identity. configs/final.yaml and artifacts/final/* below are intentionally future-candidate placeholders; create them only after completing the full-data experiment. Only then:

uv run python scripts/freeze_candidate.py \
  --checkpoint artifacts/final/best.pt \
  --config configs/final.yaml \
  --output artifacts/final/frozen_manifest.json

UNSEAL_OFFICIAL_TEST=I_HAVE_FROZEN_MODEL_AND_THRESHOLD \
FROZEN_MANIFEST=artifacts/final/frozen_manifest.json \
  bash scripts/download_test_dataset.sh

uv run python scripts/evaluate.py \
  --checkpoint artifacts/final/best.pt \
  --source data/raw/smart-turn-data-v3.2-test \
  --dataset-id pipecat-ai/smart-turn-data-v3.2-test \
  --revision 0500378e8ed6d38e37b016e24d261e8e6c6a6859 \
  --split test \
  --allow-sealed-test \
  --frozen-manifest artifacts/final/frozen_manifest.json \
  --output artifacts/final/test_metrics.json

Do not tune after reading the test result. A new model requires a new untouched test set or a clearly labeled exploratory result.

6. Export and benchmark

uv run python scripts/export_onnx.py \
  --checkpoint artifacts/final/best.pt \
  --output artifacts/final/model.onnx

uv run python scripts/benchmark.py \
  --model artifacts/final/model.onnx \
  --metadata artifacts/final/model_metadata.json \
  --threads 1 --batch-size 1 \
  --output artifacts/final/cpu_benchmark.json

ONNX export checks PyTorch/ONNX numerical parity. Runtime preprocessing, I/O names, activation semantics, threshold, parameter count, and evidence scope live in model_metadata.json; hashes, shapes, checkpoint/config/split bindings, parity, and quantization status live in export_manifest.json. Static INT8 is supported with representative calibration features; quantized scores require threshold recalibration before release.

The packaged preview is artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx: 625,431 bytes, 151,812 parameters, and FP32 maximum absolute parity error 0.0. On the recorded Apple-arm/one-thread runs, ONNX neural-only p50/p95/p99 is 0.919/0.956/0.974 ms; four-second waveform-to-probability p50/p95/p99 is 1.572/1.629/1.651 ms. These are local implementation measurements, not a deployment SLA.

7. Run the demo and exercise the controller

TURN_MODEL_PATH=artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \
  uv run python app.py

uv run python scripts/replay_stream.py \
  --input data/collection/controller_replay_fixture.jsonl \
  --output reports/controller_replay_integration.jsonl \
  --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \
  --evidence-scope synthetic_integration

The exported metadata binds the endpoint threshold and the complete controller policy. END is latched until new speech or an explicit reset; only the first END transition sets emit_response=true, preventing duplicate agent replies. The bundled replay fixture is hand-authored integration data: its probabilities, targets, and latency values are synthetic. It validates policy binding, threshold relaxation, timeout, END latching, and edge emission only—it is not model, conversation, latency, or product-quality evidence. Replace its input with private, human-annotated VAD checkpoint logs for a real sequence study.

If weights are absent, the demo displays a conspicuous heuristic fallback. That fallback validates only the UI and must never be cited as a model result. See docs/demo.md for the pause-checkpoint contract.

8. Build and publish a guarded release

uv run python scripts/build_release.py \
  --model artifacts/final/model.onnx \
  --metadata artifacts/final/model_metadata.json \
  --metrics artifacts/final/test_metrics.json \
  --frozen-manifest artifacts/final/frozen_manifest.json

# Dry validation first.
uv run python scripts/publish_hf.py --username suvradeepp

# Creates/uploads the model repo and Gradio Space only after identity validation.
uv run python scripts/publish_hf.py --username suvradeepp --execute

For a personal account without Hugging Face PRO, the publisher requests the free-account-compatible ZeroGPU tier (--space-hardware zero-a10g) only when it must create a new Space. Hugging Face currently gates new personal Gradio CPU Spaces behind PRO even when CPU Basic has no hourly charge. ZeroGPU eligibility still requires a verified account in good standing and an available hosted Space slot. Paid accounts may override this with --space-hardware cpu-basic.

The release builder refuses smoke checkpoints or a release without measured test metrics unless --allow-development-artifact is passed explicitly. That override must remain visibly development-only. For a reviewable partial-data preview, pass that flag to the builder and --allow-development-release to both publisher invocations. Development validation metrics are packaged as development_metrics.json, never test_metrics.json.

The exact development-preview package command is:

uv run python scripts/build_release.py \
  --model artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \
  --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \
  --metrics reports/partial_tinytcn_metrics.json \
  --output release \
  --allow-development-artifact \
  --include-synthetic-controller-replay

uv run python scripts/publish_hf.py \
  --username suvradeepp \
  --release-dir release \
  --allow-development-release

uv run python scripts/build_upload_folders.py \
  --release-dir release \
  --output upload-ready \
  --allow-development-release

upload-ready/kaggle-model/ is a flat drag-and-drop ONNX bundle. upload-ready/github-repository/ is a public-repository allowlist containing source, CI, tests, aggregate evidence, ONNX weights, and the two checkpoints needed to reproduce the selected warm start. It excludes credentials, caches, raw/processed data, per-example predictions, stale releases, and publish receipts. Keep the Kaggle variation private until the derived-weight rights review described in NOTICE and MODEL_CARD.md is resolved.

Add --execute only after the dry validation succeeds. The publisher verifies the authenticated owner and every packaged hash before creating or updating the remote model repository and Space. Its receipt binds exact remote HEAD commits, requires a visible build/start transition for a new Space upload, and records the final RUNNING stage. Because the installed Hub API does not expose the commit actually serving behind a Space runtime, the receipt explicitly records runtime_commit_bound=false instead of overstating that guarantee.

The release hash-binds the 2.7 MB split manifest but intentionally does not redistribute it because it contains upstream text and record identifiers. The public package carries deterministic preparation code, the pinned revision, manifest SHA-256, and aggregate reports; per-example rows stay private.

Hinglish evaluation protocol

The upstream corpus is multilingual, but a hin or eng tag is not evidence of within-utterance code-switching. This repository therefore includes a separate, consent-first Shiprocket-domain protocol:

• 30 speakers × 30 assignments = 900 intended recordings;
• 450 HOLD and 450 END targets;
• 21/4/5 speaker-disjoint train/dev/test speakers;
• minimal pairs with natural 0.3–2.0 second internal pauses plus roleplay;
• three annotators answer whether responding at the checkpoint feels like an interruption; uncertain votes remain disagreement, not forced labels.

Only the assignments and protocol exist today—no participant audio has been collected. Do not infer Hinglish performance until that benchmark is recorded and evaluated. See docs/data_collection_protocol.md and DATA_CARD.md.

What “good enough” means

A candidate is releaseable only after all of these gates pass:

1. all 83 train shards audited with zero unresolved hard validation errors;
2. group-safe validation and source/speaker stress tests completed;
3. threshold frozen under a predeclared false-interruption budget;
4. real Hinglish minimal-pair test collected with consent and held out by speaker;
5. official test opened once and reported without subsequent tuning;
6. preprocessing parity, ONNX parity, batch-1 CPU latency, and controller replay measured;
7. failure slices reviewed for language, source, synthetic/human, filler, duration, device/noise, gender/region where consented, and uncertainty;
8. upstream data/model redistribution terms reviewed before publishing weights.

The current repository passes export/runtime engineering gates and supplies honest bounded development evidence. It does not pass the full-data, Hinglish, official-test, sequence-replay, or rights-review gates.

Documentation

• Technical report
• Model card
• Data card
• Experiment registry
• Failure-analysis protocol
• Hinglish collection protocol
• Demo/runtime contract

License and responsible use

Repository-authored code is Apache-2.0; see LICENSE and NOTICE. The upstream dataset card did not provide an explicit dataset license at the inspected revision. This repository does not redistribute its audio. Apache-2.0 on the code does not grant rights to upstream audio, recordings collected under the local protocol, speaker likeness, customer data, or the development model weights. The preview weights carry no license grant pending an upstream-rights review; review data and derived-weight terms before use or redistribution.

This system decides conversational timing; it should not be used for speaker identification, emotion inference, surveillance, or consequential decisions.