README.md

metadata

title: DECLARE User Study
emoji: 🎙️
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: false

DECLARE User Study

Blind comparison of three ASR conditions (BASELINE / VANILLA-FT / DECLARE) on real spoken FairSpeech-style commands. Single-file Flask app (app.py), matching the flat repo layout used elsewhere (RailVaani), with the frontend in templates/ + static/.

Participants record themselves speaking a prompted command via the browser's native microphone API; audio is routed blindly to one of three ASR conditions (Latin-square rotation per session); a rule-based intent/entity extractor summarises what the system understood; the participant judges intent-correctness and (if applicable) entity-correctness separately. After all trials, a validated 4-item UMUX usability questionnaire is shown. Model identity is never revealed during the session.

Folder structure

app.py                 -- everything: Flask routes, NLU, routing, logging, ASR
templates/index.html   -- page shell
static/style.css        -- styling
static/app.js            -- consent form, MediaRecorder, all fetch() calls
data/sample_commands.csv -- 38 stratified FairSpeech commands
checkpoints/            -- put your .nemo files here
requirements.txt
Dockerfile
README.md

Important constraint: session state lives in an in-memory Python dict in app.py (SESSIONS). This assumes a single worker process. Do not run behind multiple gunicorn/Flask workers without first moving session state to a shared store (Redis, a database) — otherwise a participant's session can "disappear" if a later request lands on a different worker.

Before deploying with real models

The app ships in MOCK mode by default so the full flow — consent, recording, routing, judgement, UMUX — can be tested without any model weights. To run real inference:

1. Uncomment the torch / nemo_toolkit[asr] lines in requirements.txt.
2. Add your VANILLA_FT and DECLARE .nemo checkpoints under checkpoints/.
3. Set Space secrets/variables if you don't want to hardcode paths:
   • BASELINE_MODEL_ID (default: stt_en_conformer_ctc_small_ls)
   • VANILLA_FT_CKPT
   • DECLARE_CKPT

Audio pipeline

Browser records via MediaRecorder (typically webm/opus) → app.py saves the raw upload → converts to 16kHz mono WAV via ffmpeg → runs ASR → deletes both the raw upload and the WAV file. The WAV deletion happens inside transcribe()'s own finally block (so it happens even if inference fails); the raw upload is deleted separately right after, in the route handler.

Results

• results/trial_log.csv — per-trial ASR output + intent/entity correctness
• results/participants.csv — demographics + consent (keyed by session ID)
• results/umux_responses.csv — usability questionnaire responses + score

HF Spaces' local disk is ephemeral — sync these to a private HF Dataset repo regularly if running more than a quick pilot.

Analysis

Join trial_log.csv back to its (unblinded) condition column and run:

• Friedman test across the three conditions (per-participant correctness rate, computed separately for intent_correct and entity_correct — filter or stratify entity_correct == "not_applicable" rows out of that specific test)
• Wilcoxon signed-rank, pairwise (DECLARE vs. VANILLA-FT is the key comparison)
• Disaggregate both by the demographic columns in participants.csv for the fairness question