README.md

---
title: DECLARE User Study
emoji: 🎙️
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: false
---

# DECLARE User Study

Blind comparison of three ASR conditions (BASELINE / VANILLA-FT / DECLARE) on
real spoken FairSpeech-style commands. Single-file Flask app (`app.py`),
matching the flat repo layout used elsewhere (RailVaani), with the frontend in
`templates/` + `static/`.

Participants record themselves speaking a prompted command via the browser's
native microphone API; audio is routed blindly to one of three ASR conditions
(Latin-square rotation per session); a rule-based intent/entity extractor
summarises what the system understood; the participant judges
intent-correctness and (if applicable) entity-correctness separately. After
all trials, a validated 4-item UMUX usability questionnaire is shown. Model
identity is never revealed during the session.

## Folder structure

```
app.py                 -- everything: Flask routes, NLU, routing, logging, ASR
templates/index.html   -- page shell
static/style.css        -- styling
static/app.js            -- consent form, MediaRecorder, all fetch() calls
data/sample_commands.csv -- 38 stratified FairSpeech commands
checkpoints/            -- put your .nemo files here
requirements.txt
Dockerfile
README.md
```

**Important constraint:** session state lives in an in-memory Python dict in
`app.py` (`SESSIONS`). This assumes a **single worker process**. Do not run
behind multiple gunicorn/Flask workers without first moving session state to
a shared store (Redis, a database) — otherwise a participant's session can
"disappear" if a later request lands on a different worker.

## Before deploying with real models

The app ships in **MOCK mode** by default so the full flow — consent,
recording, routing, judgement, UMUX — can be tested without any model weights.
To run real inference:

1. Uncomment the `torch` / `nemo_toolkit[asr]` lines in `requirements.txt`.
2. Add your `VANILLA_FT` and `DECLARE` `.nemo` checkpoints under `checkpoints/`.
3. Set Space secrets/variables if you don't want to hardcode paths:
   - `BASELINE_MODEL_ID` (default: `stt_en_conformer_ctc_small_ls`)
   - `VANILLA_FT_CKPT`
   - `DECLARE_CKPT`

## Audio pipeline

Browser records via `MediaRecorder` (typically webm/opus) → `app.py` saves the
raw upload → converts to 16kHz mono WAV via `ffmpeg` → runs ASR → **deletes
both the raw upload and the WAV file**. The WAV deletion happens inside
`transcribe()`'s own `finally` block (so it happens even if inference fails);
the raw upload is deleted separately right after, in the route handler.

## Results

- `results/trial_log.csv` — per-trial ASR output + intent/entity correctness
- `results/participants.csv` — demographics + consent (keyed by session ID)
- `results/umux_responses.csv` — usability questionnaire responses + score

**HF Spaces' local disk is ephemeral** — sync these to a private HF Dataset
repo regularly if running more than a quick pilot.

## Analysis

Join `trial_log.csv` back to its (unblinded) `condition` column and run:

- Friedman test across the three conditions (per-participant correctness rate,
  computed separately for `intent_correct` and `entity_correct` — filter or
  stratify `entity_correct == "not_applicable"` rows out of that specific test)
- Wilcoxon signed-rank, pairwise (DECLARE vs. VANILLA-FT is the key comparison)
- Disaggregate both by the demographic columns in `participants.csv` for the
  fairness question