What this project actually is
A Flask web app that lets someone fill out any of four different forms either by typing, or by having a real spoken conversation with an AI assistant that listens, understands, validates, and fills the form in for them — live, in the browser.
Every form offers two paths that collect exactly the same data through exactly the same validation rules: a normal manual form, and a voice assistant. The voice path is the actual point of the project — it demonstrates a full local AI pipeline doing something concrete and demoable: turning a spoken sentence into a validated, structured field value, live, with no cloud APIs involved anywhere.
The project exists in two layers that matter for understanding it: a form-agnostic dialogue engine (extraction, validation, state tracking, conversation flow) that has no idea what form it's filling out, and a thin web layer (Flask routes, browser mic/speaker, session handling) that adapts that engine to run in a browser instead of a terminal. That separation is what let four different forms exist with zero duplicated logic.
How a spoken sentence becomes a filled-in field
One full turn of the voice conversation, start to finish.
Browser records your voice
Click the mic, speak, click again. The browser's own microphone access
(getUserMedia) records a short audio clip client-side —
the server never touches your microphone directly.
Audio is uploaded to the server
The recorded clip is sent as a multipart file upload. Flask saves it to a temp file and hands it to the transcription step.
Speech becomes text
A local Whisper model transcribes the clip. Silence/noise is filtered before decoding, and the model is given a short hint about what kind of answer to expect (an email, a date, a yes/no) based on which field was just asked — both measurably improve accuracy on short spoken answers.
The LLM figures out what you meant
The transcript is sent to a local LLM with a strict instruction: read this and return only JSON containing any of the form's field values mentioned. Nothing is guessed — a field is only filled if it was actually said.
The value is validated and remembered
Every extracted value passes through a type-specific validator (real email, valid date, sensible number...) before being accepted into the form's running state.
The assistant decides what to say next
A strict priority check, every turn: was the last answer invalid → ask again; is a required field still missing → ask for it; was something already-confirmed just corrected → re-show the summary; is everything filled → confirm; did they confirm → done.
The reply is spoken back
The response text is synthesized into audio server-side and sent back as base64 WAV. If no local voice is configured, the frontend automatically falls back to the browser's built-in speech synthesis instead — the assistant always talks, one way or another.
The field tracker updates live
The whole state of the form — what's filled, what's current, what's still pending — comes back in the same JSON response and re-renders instantly next to the conversation. This is the moment that actually demonstrates the pipeline working, without anyone needing to understand Whisper, Ollama, or Piper at all.
dialogue_manager.py
has no idea whether its input came from a browser mic or a keyboard, and no idea whether its
output gets spoken or printed — it only ever handles plain text in, plain text out. That's what
let the exact same dialogue engine run as a CLI tool first, then get wrapped in a web layer
later, without a single line of the actual conversation logic changing.
Four forms, one engine, zero duplicated logic
Every form is a plain data declaration in forms.py —
nothing else in the app knows or cares which one is active.
| Form | Collects | Fields | Field types used |
|---|---|---|---|
| job_application | Contact details, experience, availability, relocation willingness | 8 (6 required) | string, email, phone, integer, date, boolean |
| school_admission | Student and guardian details, grade applying for | 8 (6 required) | string, date, email, phone, boolean |
| medical_intake | Patient details, reason for visit, insurance, appointment date | 8 (6 required) | string, date, phone, email, boolean |
| support_request | Contact details, product/issue, priority level | 7 (5 required) | string, email, phone |
What adding a fifth form actually requires
NEW_FORM: List[FormField] = [
FormField("field_name", "human label", "string", True, "What should I ask?"),
# ...
]
FORMS["new_form_id"] = FormDefinition(
id="new_form_id", title="...", description="...", icon="...", fields=NEW_FORM
)That's it. No route changes, no template changes, no dialogue logic changes. Every page, every API endpoint, and the entire voice pipeline operate on "whatever fields this form declares" — this is the single design decision that made four forms cost barely more than one.
File-by-file walkthrough
Nine Python modules, each responsible for exactly one part of the pipeline.
app.py — the web layer
The only file that knows it's a website. Owns every Flask route, in-memory session storage
(VOICE_SESSIONS, SUBMISSIONS), and the
JSON contract the frontend talks to. Two route groups:
- Page routes — form selection, mode choice, the manual form (validated server-side
with the exact same
validators.pythe voice path uses), and the success page. - Voice API —
POST /api/voice/<form_id>/startcreates aDialogueManagerfor that form and returns the opening question;POST /api/voice/<session_id>/turnhandles every subsequent turn: transcribe → extract → validate → decide → synthesize, all in one request.
Two helper functions do a lot of the real work:
_synthesize_safe() never lets a missing Piper voice crash a
response, and _progress() builds the live field-tracker data sent
to the frontend every turn.
The AI pipeline files
stt.py — speech to text
A thin wrapper around faster-whisper.
Lazily loads the model once and reuses it (loading is the expensive part — every request
after the first is fast). transcribe() takes any audio file
faster-whisper's decoder handles — including the browser's webm/opus recordings directly,
via the bundled PyAV decoder, no separate ffmpeg install needed.
extractor.py — structured extraction
This is the "understanding" step. Instead of hoping the LLM's free-text reply happens to contain the right information, the prompt explicitly forces JSON-only output matching the active form's schema — the same idea as "function calling" in hosted LLM APIs, done manually via prompting since this runs through a local Ollama model.
try:
raw_response = call_llm(prompt)
except requests.exceptions.RequestException:
return {} # Ollama down or unreachable -> "nothing extracted", not a crash
try:
return json.loads(_strip_json_fences(raw_response))
except json.JSONDecodeError:
return {} # malformed response -> same graceful fallbackAlso owns is_llm_reachable() — a cheap
connectivity check the web app uses to show a clear warning banner if Ollama isn't running,
instead of the conversation just silently never advancing.
tts.py — text to speech
Mirrors stt.py's structure: lazy-loaded
Piper voice, cached after first use. synthesize_to_wav_bytes()
renders straight to an in-memory buffer — no temp file ever touches disk for the outgoing
audio, since it only needs to become base64 in a JSON response.
Schema, state, and dialogue
form_schema.py + forms.py — the form as data
FormField is a small dataclass — name,
type, required, the question to ask. forms.py is a plain registry
mapping form ids to a list of these. Nothing here is a route or a template — it's pure data,
which is exactly why every other layer of the app can be generic.
validators.py — one function per data type
Six validators (string, email, phone, integer, date, boolean),
each returning the identical shape: (is_valid, cleaned_value,
error_message). That consistency is what lets a single dispatch table call the right
one without a long if/elif chain, and it's the exact same logic used by both the voice path
and the manual form — validating a spoken email and a typed email go through the same code.
state_manager.py — the conversation's memory
FormState tracks every field's value,
which optional fields were explicitly skipped, and — critically —
update() reports which fields actually changed value
this turn, not just which were mentioned. That distinction is what makes corrections work:
restating an answer doesn't trigger anything, but a genuine change re-opens a confirmation
that was already given.
dialogue_manager.py — orchestration
A small finite state machine, checked in strict priority order every turn (see the architecture diagram above, step 6). Also owns the optional natural LLM-phrasing layer — templates are always the source of truth for what gets said; the LLM, when enabled, only ever reword them, and any phrasing failure falls straight back to the plain template.
Templates, styling, and the live conversation UI
Server-rendered Jinja2 pages, one focused piece of JavaScript for the voice screen, no frontend framework or build step.
Pages (Jinja2 templates)
base.html — shared shell + step indicator ·
index.html — form selection cards ·
choose_mode.html — manual vs. voice ·
manual_form.html — inputs generated from the schema, one loop,
any form · voice_assistant.html — the conversation screen ·
success.html — submission summary
Design system
Two-tone by intent: dark ink surfaces for anything "live" (the voice conversation, the recording state), calm paper surfaces for anything structural (forms, lists, summaries). Amber marks in-progress/attention state, teal marks completion — color carries real meaning throughout, not decoration.
The signature screen: the live field tracker
Split layout on voice_assistant.html: conversation on the left,
a running list of every field on the right. Each turn's JSON response includes a
progress array — every field's current status
(pending / current /
filled / skipped) and value if any —
and renderTracker() in voice_assistant.js
redraws it after every single turn.
This is deliberately the centerpiece of the whole UI. It's the one thing that makes "the AI understood what I said and filled in the right field" viscerally obvious to someone watching, without them needing to know anything about the three models running underneath.
Graceful degradation, twice, in the same file
voice_assistant.js's speak() function
plays real Piper audio when it's available; when it isn't, it silently calls the browser's own
speechSynthesis API instead. Separately, an
llm-banner element shows a clear warning if Ollama becomes
unreachable mid-conversation — added specifically after watching a real conversation
silently "get stuck" with no visible explanation of why.
Both are the same underlying philosophy applied at the UI layer: a missing optional dependency should degrade visibly and gracefully, never fail silently and never crash the experience.
Every core concept, explained
Converting spoken audio into text using a model that runs entirely on your own machine (faster-whisper) instead of a cloud API. Matters here for both privacy and for the "100% local, no API keys" pitch — nothing you say ever leaves the machine running the Flask server.
Forcing an LLM to return data matching a schema instead of free-flowing text, by being extremely explicit in the prompt about the exact output format and providing a concrete example. The same idea as "function calling" in hosted APIs, implemented manually here since the model runs through Ollama rather than a service with native tool-calling support.
Piper, a fast local neural TTS engine, converts the assistant's text reply into audio server-side. Chosen specifically for being fast enough to feel conversational rather than optimizing purely for voice quality.
A system that's always in exactly one of a fixed set of states, with clear rules for what happens next. The whole conversation is one: currently-asking, invalid-answer, confirming, or done — checked in the same strict priority order every single turn, which is what makes the flow predictable and debuggable rather than a tangle of conditionals.
Declaring the shape of your data once, as pure data, and writing every other layer of the system to act generically on that description rather than hardcoding knowledge about any specific form. This single decision is why four forms exist for barely more effort than one.
Every dependency that can fail — Ollama being down, a Piper voice not being installed, a malformed LLM response, an empty transcription — is designed to degrade to a visible, safe fallback rather than crash the request. This shows up at least five separate times across this project, each one added after actually watching it fail during development.
The frontend and backend agree on a fixed shape for every voice API
response (response_text, audio_base64,
progress, is_complete,
llm_available...). Keeping this contract explicit and stable is
what let the frontend be built and reasoned about independently of the Python behind it.
The MediaRecorder API records the mic
client-side; an <audio> element (or
speechSynthesis) plays the reply client-side. The server never
needs direct hardware audio access at all — a meaningful simplification over the CLI version
of this same pipeline, which needed sounddevice and a real local
microphone on whatever machine ran it.
Testing philosophy: mock the model, test the wiring
79 checks across two files, all passing, none of them requiring Ollama, a downloaded Whisper model, a Piper voice, or a browser.
Every test replaces the actual AI calls (extract_fields,
transcribe, synthesize_to_wav_bytes)
with small, deterministic fakes — the same pattern used consistently across this whole
project. This isolates the thing actually being tested (the routing, the session handling,
the state machine, the validation) from the thing that can't be tested this way (real model
output quality, which only a human listening to real audio can judge).
tests/test_app.py uses Flask's real test
client against real routes — including a full multi-turn conversation carried through an
actual multipart audio upload, all four forms confirmed to independently produce a working
opening question, and the manual-submission flow followed all the way through to actually
rendering the success page (not just checking a redirect status — see the bug log below for
exactly why that distinction mattered).
Real bugs found and fixed
Documented deliberately, not swept under the rug — finding and fixing these is the actual engineering work this project demonstrates.
The extraction call never actually passed the active form's schema, so it silently always extracted against a hardcoded default form regardless of which one was in use. Invisible with only one form ever built — exactly the class of bug that only surfaces once a second real case exists to compare against.
Fixed by requiring schema explicitly rather than defaulting it, and covered by a regression test that checks all four forms independently.
A connection failure to Ollama was raised uncaught, unlike every other failure mode in this project, which all degrade to "nothing extracted" instead of crashing.
Wrapped the LLM call in the same defensive pattern already used for malformed JSON responses.
The template used submission.values — but Python dicts have a
real built-in method called .values(), so Jinja silently
resolved that instead of the intended dictionary key. The first version of the test suite
didn't catch it either, because it only checked the redirect status without ever actually
rendering the page.
Renamed the key to answers everywhere to eliminate
the whole class of collision, and added a test that actually renders the success page.
A zip combined this web app with an earlier CLI-only version of the same pipeline. They
share filenames (dialogue_manager.py,
state_manager.py...) with incompatible contents, and
separately, Flask's templates//static/
folder convention had been flattened during zipping — breaking every page and every
static asset at once.
Rebuilt the correct structure from the same files (content was untouched), verified by extracting the corrected zip fresh and re-running the full test suite against that exact extraction.
Once bug #2 was fixed, an unreachable LLM correctly stopped crashing things — but now it failed too quietly: the conversation just kept re-asking the same question with no visible explanation anywhere in the UI.
Added is_llm_reachable(), threaded into every
voice API response as llm_available, and a clear red banner in
the UI the moment it's false.
Setup & running
Full detail lives in README.md — this is the short version.
pip install -r requirements.txt
# Terminal 2 -- local LLM, used for understanding speech
ollama pull llama3.2
ollama serve
# voices/en_US-lessac-medium.onnx + .onnx.json
# from huggingface.co/rhasspy/piper-voices (optional --
# falls back to the browser's own voice if skipped)
python app.py
# -> http://localhost:5000Known limitations
Stated plainly, the same way every fallback in this project is — say these before anyone else points them out.
- In-memory sessions. Fine for a local demo; restarting the server drops in-progress
conversations. Moving to Redis or a database wouldn't require changing
DialogueManagerorFormStateat all. - Click-to-talk, not fully hands-free. A deliberate tradeoff for demo reliability over the fully automatic voice-activity-detection built in an earlier CLI-only version of this pipeline — a misfiring auto-stop mid-pitch is a worse failure mode than one extra click.
- No accounts or real persistence. Submissions live in memory to demonstrate the flow works, not as a production form backend.
- Depends on Ollama and Piper configured locally. Not something a shipped product would ask an end user to do — in a real deployment these would run server-side, invisible to the user, exactly as they're already architected here (the browser never talks to either directly).
Pitch & resume notes
- The strongest live-demo moment is the field tracker filling in in real time as you talk — lead with that, not an explanation of the tech underneath.
- "How hard would it be to add our form?" is answered directly by
forms.py— pull it up and show how short one form declaration is. - Built a fully local, three-model voice pipeline — Whisper for transcription, a local LLM for structured extraction via prompt-based function calling, Piper for synthesis — with zero paid API dependency anywhere in the stack.
- Designed for graceful degradation as a first-class concern, not an afterthought: every external dependency (the LLM, the voice model, the transcription) has a visible, tested fallback path rather than a crash.
- Found and fixed five real bugs through actual testing and real usage, not just written-and-assumed-correct code — including a subtle Jinja2/Python dict-method collision that crashed every single form submission before it was caught.