File size: 2,434 Bytes
35d483e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # Hinglish benchmark collection protocol
## Purpose
This benchmark measures whether a voice agent would interrupt Indian Hinglish
speakers at natural pauses. It is not a speaker-identification dataset. Do not
collect names, phone numbers, real addresses, order IDs, or other customer data.
## Consent and governance
Before recording, each adult participant must receive and affirm:
1. The recording purpose, expected duration, and examples of intended use.
2. That raw voice is biometric/personal data and participation is voluntary.
3. The chosen redistribution license—or an explicit statement that recordings
remain private and only aggregate metrics/model updates will be published.
4. A deletion/contact process and retention period.
Store the signed consent record separately from audio. Use random speaker IDs in
all manifests. The `data/private/` directory is gitignored. This repository does
not provide legal advice; the collector remains responsible for applicable law
and organizational policy.
## Recording
- Target 30 speakers with varied regions, genders, devices, and acoustic spaces.
- Generate 30 balanced assignments per speaker with
`scripts/generate_hinglish_protocol.py`.
- Record 16 kHz or higher, mono preferred, lossless WAV/FLAC.
- For `END`, speak the full request naturally and stop.
- For `HOLD`, speak the entire request and take a natural 0.3–2.0 second pause at
`<PAUSE>` before continuing. Do not stop the recording at the marker.
- Repeat failed recordings; do not silently relabel them.
The generated split is speaker-disjoint. Never move recordings between splits
after looking at model predictions.
## Annotation
At each derived pause checkpoint, three annotators independently answer:
> If the agent responded now, would it feel like an interruption?
Options: `yes`, `no`, `uncertain`, plus a short optional reason. Store all votes.
The soft endpoint target is `no_votes / valid_votes`; preserve `uncertain` as
disagreement rather than coercing it into a confident binary label. Report raw
agreement and a chance-corrected statistic.
## Release gate
Before publishing any audio, confirm consent scope and license, remove accidental
PII, run duplicate checks, and manually listen to a stratified quality sample.
If redistribution rights are unclear, publish only the protocol, anonymized
metadata, aggregate metrics, and trained artifacts whose terms have been reviewed.
|