DATA_CARD.md

metadata

pretty_name: Tiny Hinglish Turn Detection Data Plan
language:
  - hi
  - en
task_categories:
  - audio-classification
license: other

Data card: Tiny Hinglish Turn Detection

This project uses two distinct data components:

1. an external multilingual Smart Turn corpus for representation training and benchmark compatibility; and
2. a proposed, consent-first Indian Hinglish hard-case benchmark for the actual deployment domain.

No upstream audio or locally recorded voice is distributed in this repository. Only data-preparation code, small machine-readable reports, and the local prompt assignment plan are included.

Component A: Smart Turn v3.2

Source and versions

Role	Hugging Face dataset	Pinned revision	Catalog scope	Local state
Train/development	pipecat-ai/smart-turn-data-v3.2-train	e564e2ac567f774d1880aa1db6ce97afb8c519b7	270,946 rows, 83 Parquet shards, about 41.4 GB	1 shard present and audited
Official test	pipecat-ai/smart-turn-data-v3.2-test	0500378e8ed6d38e37b016e24d261e8e6c6a6859	31,527 rows, 10 shards, about 4.84 GB	Sealed; not downloaded or inspected

Catalog sizes identify the expected snapshot; they are not local row-count claims. The download scripts verify the exact shard count before declaring a snapshot complete.

Intended task and label schema

The normalized schema is one row per labeled audio example:

Field	Type	Meaning
record_id	string	Upstream stable ID, or provenance-based fallback
audio	encoded bytes/path/array	Utterance or turn suffix audio
endpoint	required bool	true = END, false = HOLD
midfiller	nullable bool	Optional internal-filler auxiliary target
endfiller	nullable bool	Optional turn-final-filler auxiliary target
synthetic	nullable bool	Upstream synthetic-status tag
language	nullable string	Upstream language tag
dataset	nullable string	Upstream source/collection tag
spoken_text	nullable string	Transcript/text when supplied; never used at inference
source_file, source_row	provenance	Lazy resolution back to the raw row

Nullable filler fields remain null. Converting null to false would inject unobserved negative labels and bias the auxiliary tasks.

Local audit scope

The only locally audited file is:

data/raw/smart-turn-data-v3.2-train/data/train-00010-of-00083.parquet

Its evidence is stored in reports/partial_shard_audit.json.

Property	Count/value
Records	3,265
Valid records	3,265
Invalid records	0
Duration observed	24,685.5167 seconds (6.857 hours)
Minimum / maximum duration	0.36 / 29.56 seconds
Encoded bytes	492,503,246
Audio format	3,265 FLAC
endpoint=false / true	1,667 / 1,598
synthetic=false / true	553 / 2,712
midfiller=null / false / true	642 / 1,284 / 1,339
endfiller=null / false / true	642 / 1,718 / 905
Unique language tags	23
Exact-audio duplicate hashes	0
Duplicate record IDs	0
Conflicting-label audio hashes	0

Largest language counts are English 758, Spanish 206, French 161, Hindi 158, and Dutch 153. The presence of Hindi and English rows does not establish Hinglish code-switching within recordings.

The shard is about 83% synthetic. It may have been sharded by source or another non-random rule; these distributions must not be extrapolated to the other 82 files.

Validation and audio inspection

The audit is streaming and does not need to materialize the 41 GB corpus in memory. For each row it:

1. normalizes known aliases without conflating false and missing values;
2. validates the required endpoint and audio fields;
3. inspects encoded audio for hash, bytes, format, rate, channels, frames, bit depth, and duration;
4. retains all errors and warnings in the manifest;
5. summarizes label/language/source/audio distributions and duplicate conflicts.

--fail-on-error still writes the manifest/report for diagnosis, then returns a failure status if any row has hard errors.

Leakage-aware grouping

The pipeline creates an edge between rows sharing any available:

• exact encoded-audio SHA-256;
• conversation/session/call identifier;
• speaker/user/actor identifier;
• TTS voice identifier;
• source recording or clip identifier;
• prompt/template/script/parent identifier;
• upstream record identifier;
• sufficiently long normalized transcript/prompt;
• audio basename when no content hash is available.

Metadata values are canonicalized and SHA-256 hashed before being written as linkage keys. They are namespaced by source where appropriate. This reduces identifier exposure but is pseudonymization, not guaranteed anonymization.

Union-find computes transitive components across all keys. A deterministic hash of the component keys becomes group_id, independent of input row order. Split assignment moves entire groups and validates that no group, audio hash, or individual metadata key crosses boundaries.

The currently audited raw schema does not expose usable speaker, conversation, session, or TTS-voice identifiers. As a result, all 3,265 base linkage groups used by the IID split are singletons (largest group: one row; multi-row groups: zero). The source stress split separately groups rows by 12 dataset values. The grouping logic is ready for richer metadata, but the current artifact is only best-effort row-disjoint and exact-duplicate-safe—not speaker- or voice-disjoint. Unknown identity/template leakage remains possible, and a grouped bootstrap on this shard is numerically a row bootstrap.

Current split artifacts

The local-shard IID development split is deterministic with seed 42 and balances the endpoint, language, synthetic, and source marginals:

Split	Rows	END	HOLD	Synthetic	Human-tagged
Train	2,939	1,438	1,501	2,441	498
Validation	326	160	166	271	55

No crossing of the observed keys was detected; this does not close the missing speaker/voice linkage gap. Evidence: reports/partial_iid_split.json.

A source-held-out stress split has 2,617 train and 648 validation rows, also with zero detected crossings. It is strongly confounded:

• train: 2,614 synthetic / 3 human-tagged, spanning 23 languages;
• validation: 98 synthetic / 550 human-tagged, only English and Spanish.

It measures joint collection-domain shift, not the independent causal effect of an unseen source. Evidence: reports/partial_source_holdout_split.json.

Current experiment-use boundary

The 326-row IID validation split was reused for checkpoint inspection, model thresholds, and bounded follow-up experiments. Current model comparisons are therefore adaptive development evidence, not an untouched holdout. Because all IID groups are singletons, its grouped bootstrap is row bootstrap in effect.

On the 648-row source stress validation side, both the acoustic baseline and TinyTCN rank near chance (AUROC 0.5539 and 0.5523). The comparison bootstrap has only nine validation source groups and wide intervals. This supports a joint domain-shift warning, not an unseen-source generalization claim.

The official-test catalog identifies 31,527 rows, but no official-test audio has been downloaded, inspected, or evaluated. No test metric or Hinglish quality claim exists.

Known data limitations

• Only 1/83 train shards is locally characterized.
• The inspected shard is predominantly synthetic.
• The upstream label-generation and annotator-agreement process is not revalidated by this repository.
• Language tags do not measure code-switch position or Indian regional variety.
• Available source/domain fields are confounded with human/synthetic status and language.
• Exact hash checks do not catch acoustically identical audio with different encodings; future audits should add perceptual/audio fingerprints.
• Repeated short fillers are intentionally excluded from text grouping because grouping all “haan” examples would collapse unrelated turns; this leaves some template leakage risk.
• Transcripts can contain PII. Processed manifests are ignored by version control and should follow the same access/retention policy as raw data.
• Audio corruption checks validate decodability and headers, not semantic label correctness. A stratified listening audit is still required.

Component B: proposed Shiprocket Hinglish benchmark

Motivation

The target failures are natural Indian Hinglish holds: “haan…”, “matlab…”, addresses and identifiers broken across pauses, self-corrections, enumeration, and code-switching around logistics terms. Those behaviors cannot be certified from aggregate upstream language tags.

Planned composition

data/collection/assignments.jsonl contains 900 deterministic assignments generated with seed 20260823:

Dimension	Planned count
Speakers	30
Assignments per speaker	30
HOLD / END	450 / 450
Train	21 speakers, 630 examples
Development	4 speakers, 120 examples
Test	5 speakers, 150 examples

Scenarios cover address, COD, damaged parcels, delivery instructions, order ID, phone, pickup, reschedule, return, support, and tracking. Minimal-pair prompts change the pause location or completion while holding words/domain close. HOLD recordings must continue naturally after a 0.3–2.0 second <PAUSE> marker; the marker is an instruction and must not be spoken.

The assignments are not recordings. Zero participant audio examples have been collected, annotated, trained on, or evaluated.

Participant and acoustic sampling

Recruit adults across regions, gender identities, speech rates, and code-switch styles, without turning demographic coverage into an identity-inference task. Record across representative phone microphones, headsets, rooms, traffic/noise, far-field conditions, and codecs. Store only the attributes participants consent to disclose and only at a granularity justified by slice analysis.

Annotation

At every derived pause checkpoint, three independent annotators answer:

If the agent responded now, would it feel like an interruption?

Allowed votes are yes, no, and uncertain, with an optional short reason. For a soft endpoint target, use no_votes / valid_votes. Preserve uncertain votes and report disagreement rather than converting them into confident binary labels. Report raw agreement and a chance-corrected statistic. Adjudication must not use model predictions.

The speaker-disjoint split assignment is generated before model evaluation and must not be changed after viewing errors.

Consent, privacy, and governance

Before recording, each participant must receive and affirm:

• purpose, expected recording duration, and examples of use;
• that voice is biometric/personal data and participation is voluntary;
• whether audio, annotations, metadata, and derived weights may be redistributed;
• retention period, access controls, withdrawal/deletion contact and process;
• compensation and any intended commercial use.

Use random speaker IDs. Store identity/consent records separately from audio. Never solicit or record real names, customer phone numbers, addresses, order IDs, tracking IDs, or production support calls. Screen transcripts and a stratified audio sample for accidental PII before any release.

Raw recordings belong under ignored private storage. If redistribution rights are unclear, release only the protocol, non-identifying aggregate statistics, and artifacts whose terms have been reviewed.

Recommended full-data release checks

Before using either component in a public model claim:

1. verify all expected shards and pinned revisions;
2. audit 100% of rows and resolve hard errors;
3. report per-shard and full-corpus distributions to detect sharding effects;
4. add perceptual duplicate checks and a stratified manual listening sample;
5. freeze grouping and split policies;
6. report both IID and leave-one-source-out results with confounding warnings;
7. collect and lock the speaker-disjoint Hinglish test;
8. publish label definitions, agreement, exclusion counts, and slice denominators;
9. conduct PII, consent, retention, and redistribution reviews;
10. open the official upstream test once, after model and threshold freeze.

Licensing

The inspected upstream dataset card did not state an explicit dataset license. The license: other metadata on this card reflects mixed and unresolved data provenance; it is not a grant of permission. Repository-authored code is Apache-2.0, but that license does not apply automatically to upstream audio, participant recordings, annotations, speaker likeness, or derived artifacts.

Confirm rights with the dataset owner and legal/privacy reviewers before commercial use, redistribution, or public weight release.