hassaanik/AI_Voice_Form_Filler

Voice Form Assistant — Web App

A Flask web app that fills out forms through natural conversation. Pick a form, choose to fill it manually or talk to a voice assistant, and watch it get filled in and validated live.

Fully local: Whisper for transcription, a local LLM via Ollama for understanding, Piper for spoken replies. No cloud AI APIs, no API keys.

Full documentation: see documentation.html for architecture, every concept explained, the testing approach, and a log of real bugs found and fixed while building this. This README is just the practical quick-start.

The 4 forms

Form	Collects
Job Application	Contact details, experience, availability, relocation
School Admission	Student and guardian details, grade applying for
Patient Intake	Patient details, reason for visit, insurance, appointment date
Support Request	Contact details, product/issue, priority level

Adding a fifth form means adding one entry to forms.py — no route, template, or dialogue logic changes needed.

Setup

pip install -r requirements.txt

You also need:

1. Ollama, running locally with a model pulled:

   ollama pull llama3.2
   ollama serve
   

2. A Piper voice (optional). Download both files for a voice from huggingface.co/rhasspy/piper-voices — e.g. en_US-lessac-medium.onnx and en_US-lessac-medium.onnx.json — into the voices/ folder.

   If you skip this, the app still works — it falls back to the browser's own built-in speech synthesis instead.

Run it

python app.py

Open http://localhost:5000.

The voice assistant is click-to-talk: click the mic, speak, click again when you're done.

Run the tests

python tests/test_validators.py
python tests/test_app.py

Both run with the AI calls mocked — no Ollama, Whisper model, or Piper voice required. They verify the app's logic and wiring; they don't verify real transcription or synthesis quality, which only running the actual app can do.

Project structure

app.py                  Flask routes, sessions, the voice API
forms.py                Registry of all 4 forms
form_schema.py           FormField definition + Whisper context hints
validators.py            One validator per field type
state_manager.py         Tracks a form's filled/missing fields per turn
extractor.py              Calls the local LLM to extract fields from speech
dialogue_manager.py       Decides what the assistant says next
stt.py                    Whisper transcription
tts.py                    Piper speech synthesis
templates/                Jinja2 pages (form select, manual form, voice UI, success)
static/                   style.css + voice_assistant.js
tests/                    test_validators.py, test_app.py
voices/                   Piper voice files go here

Known limitations

• In-memory sessions — fine for a local demo, resets on server restart.
• Click-to-talk rather than fully hands-free (a deliberate reliability tradeoff for demo contexts).
• No accounts or real persistence — submissions exist to demonstrate the flow, not as a production backend.
• Requires Ollama and (optionally) Piper running locally.

See documentation.html for the reasoning behind each of these.