docs/data_collection_protocol.md

Hinglish benchmark collection protocol

Purpose

This benchmark measures whether a voice agent would interrupt Indian Hinglish speakers at natural pauses. It is not a speaker-identification dataset. Do not collect names, phone numbers, real addresses, order IDs, or other customer data.

Consent and governance

Before recording, each adult participant must receive and affirm:

1. The recording purpose, expected duration, and examples of intended use.
2. That raw voice is biometric/personal data and participation is voluntary.
3. The chosen redistribution license—or an explicit statement that recordings remain private and only aggregate metrics/model updates will be published.
4. A deletion/contact process and retention period.

Store the signed consent record separately from audio. Use random speaker IDs in all manifests. The data/private/ directory is gitignored. This repository does not provide legal advice; the collector remains responsible for applicable law and organizational policy.

Recording

• Target 30 speakers with varied regions, genders, devices, and acoustic spaces.
• Generate 30 balanced assignments per speaker with scripts/generate_hinglish_protocol.py.
• Record 16 kHz or higher, mono preferred, lossless WAV/FLAC.
• For END, speak the full request naturally and stop.
• For HOLD, speak the entire request and take a natural 0.3–2.0 second pause at <PAUSE> before continuing. Do not stop the recording at the marker.
• Repeat failed recordings; do not silently relabel them.

The generated split is speaker-disjoint. Never move recordings between splits after looking at model predictions.

Annotation

At each derived pause checkpoint, three annotators independently answer:

If the agent responded now, would it feel like an interruption?

Options: yes, no, uncertain, plus a short optional reason. Store all votes. The soft endpoint target is no_votes / valid_votes; preserve uncertain as disagreement rather than coercing it into a confident binary label. Report raw agreement and a chance-corrected statistic.

Release gate

Before publishing any audio, confirm consent scope and license, remove accidental PII, run duplicate checks, and manually listen to a stratified quality sample. If redistribution rights are unclear, publish only the protocol, anonymized metadata, aggregate metrics, and trained artifacts whose terms have been reviewed.