HANDOFF.md

Fabella Handoff — Explanation + Read-Aloud Pipeline on Modal

Context

Fabella is a Gradio children's-storytelling app in /home/khoi/fabella, now pivoted to Track I · Backyard AI ("useful for someone the maker actually knows"). It solves a specific real problem parents face: how do I explain a hard thing to my kid in their own language?

The parent describes a situation in a sentence or two. Fabella drafts a short, kind, age-appropriate explanation in an Opener / Body / Closer / follow-up shape. A second small model checks the draft against a 6-criterion rubric before the parent sees it. After generation, the parent can optionally click Read aloud to synthesize the explanation with VoxCPM2.

Live URLs:

• HF Space: https://build-small-hackathon-fabella.hf.space
• Modal drafter (Gemma 4 E4B-IT): https://khoitruong071510--fabella-serve-drafter.modal.run
• Modal judge (Nemotron-3 Nano 4B): https://khoitruong071510--fabella-serve-judge.modal.run
• Modal TTS (VoxCPM2): https://khoitruong071510--fabella-serve-tts.modal.run
• Modal app: https://modal.com/apps/khoitruong071510/main/deployed/fabella
• HF Space repo: https://huggingface.co/spaces/build-small-hackathon/Fabella

Current Architecture

                    HF Space (CPU, custom HTML+CSS+JS)
                                   |
                                   |  POST /gradio_api/call/make_explanation
                                   |  -> SSE stream with 4-section string
                                   v
                         app.py (gradio.Server / FastAPI)
                                   |
              +--------------------+--------------------+--------------------+
              |                                         |                    |
              v                                         v                    v
   Modal drafter (A10G, Gemma 4 E4B-IT)     Modal judge (A10G, Nemotron-3)  Modal TTS (A10G, VoxCPM2)
   --tool-call-parser gemma4                (no tool-calling flags)         FastAPI /synthesize
   ReAct via LangChain                      Pydantic JSON verdict          audio/wav when requested
   validate_explanation tool                one invoke + one repair retry
   middleware jumps to "end" on OK

Key design decisions:

• HF OAuth + bucket per-user JSON history. Signed-in users are keyed by Hugging Face username; unsigned users use a browser localStorage session id. Minimal chat history and parent preferences persist in SQLite at /models/fabella-data/history.sqlite3 on the mounted HF bucket. This is intentionally not PostgreSQL. Use external Postgres later if multi-replica writes or richer queries are needed.

• Drafter on LangGraph. The drafter is a create_agent ReAct loop with one tool (validate_explanation) and a custom middleware that jumps to end after a successful validation or after a hard cap of two tool calls. State machine, conditional edges, tool-call plumbing — that's LangGraph's job, and it works.

• Judge on direct OpenAI + Pydantic. The judge task is bounded — one rubric, one draft, one structured verdict — so it doesn't need an agent loop. judge.py calls the judge endpoint through the OpenAI-compatible client, first tries vLLM/OpenAI response_format with JudgeVerdict.model_json_schema(), then falls back to prompt-only JSON plus one repair retry. Cross-field consistency (ok ⇔ verdict) is enforced in code, not in the prompt.

• Three separate Modal web_servers. Drafter and judge run on A10G with min_containers=0 and a 2-minute scaledown_window so they scale to zero when idle. TTS runs separately on L4 with the same scale-to-zero policy.

• TTS only runs on demand. VoxCPM2 is a separate FastAPI wrapper, not vLLM. The HF Space make_audio API posts explanation text to /synthesize, receives audio/wav, and returns a base64 data URL to the browser. It cold-starts on demand after an idle period.

• The judge has NO tool-calling flags on the server side. Its prompt asks for raw JSON in content; the Pydantic parser does the rest. This dodges Nemotron-3-Nano's chat-template tool-dialect (<tool_call>...</tool_call> markers that vLLM's hermes parser doesn't recognize) entirely.

• The drafter DOES use tool calling. vLLM is launched with --enable-auto-tool-choice --tool-call-parser gemma4; the server parses Gemma 4's <|tool_call|>...<tool_call|> markers into OpenAI-spec tool_calls JSON, which the client reads off response.choices[0].message.tool_calls directly.

• HF Space runs a custom gradio.Server (FastAPI subclass). No default Gradio chrome. Storybook design, single hand-coded page.

• API contract for the frontend: the @app.api endpoint returns one string with sections joined by U+001F (Unit Separator) — Opener, Body, Closer, Follow-up. The frontend splits on that. (Gradio Server @app.api has no output components, so tuples get dropped — single string is the simplest workaround.)

File Map

File	Purpose
app.py	gradio.Server (FastAPI subclass) app, custom HTML+CSS+JS, make_explanation API, make_audio TTS proxy, HF OAuth-aware history APIs, per-user JSON files in the HF Bucket, no-op @spaces.GPU placeholder for HF runtime
agent.py	LangChain ReAct agent. build_agent(llm, req, judge_llm=None) returns (agent, user_prompt). make_validate_tool builds a closure that calls judge_explanation() if a judge is given, else falls back to a rule check. FabellaAgentMiddleware.before_model jumps to end once validation passes or after max_tool_calls=2. extract_explanation(messages) parses Opener/Body/Closer/follow-up sections from the validated tool-call draft.
judge.py	Direct OpenAI-compatible + Pydantic-validated judge. judge_explanation(llm, draft, req_age, req_tone, child_name, situation) -> JudgeVerdict. First tries vLLM/OpenAI response_format with JudgeVerdict.model_json_schema(), then falls back to prompt-only JSON plus one repair retry before raising JudgeFailed. Tolerant of markdown fences and pretty-printed JSON.
schema.py	ExplainRequest dataclass + JudgeVerdict Pydantic model + JudgeFailed exception.
safety.py	Input sanitization, profanity block, sanitize_situation, explain_to_words(tone), age_bucket(age).
llm.py	FabellaVLLM BaseChatModel wrapping vLLM's OpenAI-compatible API. bind_tools builds OpenAI-spec tools=[...], _generate passes it and reads message.tool_calls from the response. Replay of prior AIMessage.tool_calls and ToolMessage results into next-turn messages uses the OpenAI chat-completions shape.
modal_app.py	Modal deployment: download_drafter + download_judge + download_tts; serve_drafter (port 8000), serve_judge (port 8001), serve_tts (port 8002). Drafter/judge on A10G; TTS on L4.
modal_app_gemma.py	(removed) Legacy: a previous-session single-model Modal deploy, kept for reference. Not the live deploy.

What Changed This Session

The most recent session (pivot to Backyard AI) changed:

New files

• judge.py — Pydantic-validated judge with repair retry
• modal_app_gemma.py — (removed) kept as reference for the prior single-model deploy

Substantially rewritten

• agent.py — story-generation agent replaced with explanation-generation agent. make_validate_tool now optionally takes judge_llm and routes through judge_explanation(). The drafter's output format changed from "Title: / body" to "Opener: / Body: / Closer: / (optional) If they ask more:". extract_explanation parses these four sections.
• schema.py — StoryRequest replaced with ExplainRequest (situation, age, child_name, tone, seed). Added JudgeVerdict (Pydantic) and JudgeFailed.
• safety.py — sanitize_situation, explain_to_words(tone). Legacy theme/moral/length functions kept for compat.
• app.py — frontend redesigned for the "explain a hard thing" use case. New form fields: situation textarea, age slider (5-12), child_name (optional), tone segmented control (gentle / matter-of-fact / playful), example chips. Output is the four sections in a book-page layout with the new "Opener" / "The explanation" / "Closer" / "If they ask another question" tags. Added Read aloud with make_audio proxy to VoxCPM2.
• modal_app.py — three web_servers in one Modal app. Drafter uses --tool-call-parser gemma4; judge uses no tool flags; TTS runs VoxCPM2 behind FastAPI /synthesize.
• llm.py — defaults updated to point at the drafter endpoint (gemma-4 model name).

Removed earlier

• multi_agent.py — earlier multi-agent design (3 parallel drafters + judge) was reverted
• nemotron3_tool_parser.py — custom XML tool parser for the (also removed) 30B Nemotron path
• prompts.py, generator.py, mock.py, real.py — legacy files

Non-obvious gotchas

• No-op @spaces.GPU in app.py. HF Spaces runtime scans for at least one @spaces.GPU function at import and raises RUNTIME_ERROR: No @spaces.GPU function detected if none exists. The placeholder is a 1-second no-op. Do not delete it.
• sys.path hack in every module. Each file does sys.path.insert(0, os.path.dirname(...)) so imports work when run as python app.py from the package root. Don't refactor to relative imports.
• Pydantic disallows _-prefixed fields. In llm.py, the runtime-mutable state (OpenAI client, tools, call counter) is declared with PrivateAttr, not Field.
• Three Modal endpoints, three env vars. HF Space reads MODAL_DRAFTER_URL, MODAL_JUDGE_URL, and MODAL_TTS_URL. The old MODAL_VLLM_URL is dead — delete it if it's still there.
• The drafter uses native tool calling via vLLM. vLLM is started with --enable-auto-tool-choice --tool-call-parser gemma4; the server parses Gemma 4's native <|tool_call|>call:name{args}<tool_call|> markers into OpenAI-spec tool_calls JSON. The client passes real tools=[{type:"function", function:{name, description, parameters:JSON-schema}}] on each request and reads response.choices[0].message.tool_calls directly. If the model emits no tool call, content is returned as the final answer.
• The judge does NOT use tool calling. The chat template for Nemotron-3-Nano-4B emits tool calls in a custom XML dialect inside <tool_call>...</tool_call> markers that vLLM's built-in parsers don't recognize. The judge server runs with no tool-calling flags; the judge prompt asks for raw JSON in content, and judge.py parses that with Pydantic.
• Pydantic judge schema in schema.py. JudgeVerdict has five fields: ok (bool), issues (list[str], each capped at 200 chars), score (float in [0, 1]), verdict (Literal["approve", "revise"]), reasoning (str, capped at 300 chars). Cross-field consistency (ok ⇔ verdict) is enforced in judge_explanation() — the model is asked to agree, and the code normalizes if it doesn't.
• Judge retry-on-failure. If the first response isn't parseable JSON, judge_explanation() retries once with a REPAIR_PROMPT that shows the previous bad response. If both fail, JudgeFailed is raised and the validate tool falls back to the deterministic rule check. The deployed judge path intentionally bypasses LangChain message invocation; LangGraph stays only in the drafter loop.
• Middleware @hook_config(can_jump_to=["end"]) is required. Without it, LangGraph never creates the conditional edge and the early-exit silently does nothing.
• Modal uses CUDA devel image. The nvidia/cuda:12.9.0-devel-ubuntu22.04 base provides nvcc, which vLLM/FlashInfer need. debian_slim crashes during vLLM startup.
• Drafter flag --language-model-only is required. Gemma 4's multimodal processor pulls heavy deps and crashes the vLLM server on text-only requests. This flag tells vLLM to skip processor init. The judge (Nemotron-Nano-4B) is text-only and does NOT need this flag.
• Gemma 4 E4B is multimodal — it can take audio input. This matters in two ways:
  1. --language-model-only is correct today because Fabella's drafter only ever receives text. If you later add a feature where the parent records a 30s voice memo and the drafter transcribes it (Whisper-style), the vLLM flag will need to change to support audio inputs. The model supports it natively.
  2. The audio side of Gemma 4 is a separate path from the VoxCPM2 TTS endpoint. They are independent: VoxCPM2 reads text and produces 48 kHz audio; Gemma 4 could (if enabled) read audio and produce text. Don't conflate them when debugging.
• Critical-path LLMs scale to zero. Drafter and judge use min_containers=0 with a 2-minute scaledown_window, so the first generation after idle pays a Modal/vLLM cold start but the demo does not bill continuously while nobody is using it. TTS follows the same policy on L4.
• TTS runs on L4. VoxCPM2 is ~2B and fits smaller GPUs, so serve_tts uses gpu="L4" plus min_containers=0 instead of A10G. If L4 availability or latency is bad, switch back to A10G or try Modal GPU fallbacks.
• VoxCPM2 TTS is not vLLM. serve_tts writes a generated FastAPI server into the container and runs uvicorn --app-dir /root. It returns audio/wav from /synthesize; app.py::make_audio converts that to a base64 data URL for the browser.
• Do not switch to nanovllm-voxcpm for this demo. It is faster, but it needs flash-attn, changes the API (target_text, streamed MP3), and is not worth the integration risk with one day left and a tight GPU budget. Keep the stable official VoxCPM2 server.
• section_sep is U+001F (Unit Separator). The @app.api endpoint returns Opener, Body, Closer, Follow-up joined by \x1f. The frontend splits on it. Don't use \n — body text can contain newlines legitimately.
• FABELLA_MODEL_PATH env var is no longer consulted on the deployed path. Modal's download_drafter hardcodes google/gemma-4-E4B-it (Apache 2.0, not gated). Do not swap to gemma-3-4b-it (gated — would break the no-API-key rule).

Deployment Commands

# Modal: download weights (run once per model)
.venv/bin/modal run modal_app.py::download_drafter
.venv/bin/modal run modal_app.py::download_judge
.venv/bin/modal run modal_app.py::download_tts

# Modal: deploy (rebuilds the image, rolls out all web_servers)
.venv/bin/modal deploy modal_app.py

# HF Space: env vars
hf spaces variables add build-small-hackathon/Fabella \
    --env MODAL_DRAFTER_URL=https://khoitruong071510--fabella-serve-drafter.modal.run
hf spaces variables add build-small-hackathon/Fabella \
    --env MODAL_JUDGE_URL=https://khoitruong071510--fabella-serve-judge.modal.run
hf spaces variables add build-small-hackathon/Fabella \
    --env MODAL_TTS_URL=https://khoitruong071510--fabella-serve-tts.modal.run

# HF Space: upload code
hf upload build-small-hackathon/Fabella app.py    --type space
hf upload build-small-hackathon/Fabella agent.py  --type space
hf upload build-small-hackathon/Fabella judge.py  --type space
hf upload build-small-hackathon/Fabella llm.py    --type space
hf upload build-small-hackathon/Fabella schema.py --type space
hf upload build-small-hackathon/Fabella safety.py --type space
hf upload build-small-hackathon/Fabella requirements.txt --type space

# HF Space: restart to pick up new code
hf spaces restart build-small-hackathon/Fabella

Cost

• Drafter: 1× A10G while active, $0.80/hr, 2-minute scaledown
• Judge: 1× A10G while active, $0.80/hr, 2-minute scaledown
• TTS: 1× L4 while active, min_containers=0, only used after Read aloud
• At idle: $0/hr (scaledown)
• Typical demo session: a few minutes warm = ~$0.03-0.05

Known Issues / Open Questions

1. Cold start latency — First request after 2 min idle triggers vLLM cold start (~2 min per container for model load + torch.compile
   • CUDA graph capture). Both containers cold-start in sequence on the first request of a new session. Could add min_containers=1 to each Modal serve() to keep warm (costs ~$1.60/hr idle).
2. No test suite — No automated tests exist. Manual smoke-tests are in this handoff (search "Live test" or "Smoke-test").
3. Judge occasionally emits unparseable thinking-traces. The judge_explanation() repair prompt fixes this most of the time. When both attempts fail, the validate tool falls back to the rule check, so the system never hard-errors. The model is a reasoning model; a --default-chat-template-kwargs '{"enable_thinking": false}' flag could be added to the judge server to make outputs shorter, but the retry handles it well enough.
4. Drafter at temperature 0.9 — produces creative variety but the judge sometimes rejects a perfectly good draft on style grounds. The seed UI control lets parents re-roll for variety.

Suggested Skills

• hf-cli — Manage HF Space: variables, logs, uploads, restarts
• find-docs — For Modal, vLLM, Gradio, LangChain, Pydantic API questions (use ctx7 CLI)
• diagnose — If runtime errors occur (vLLM startup, agent failures, judge parsing)
• agent-browser — For end-to-end testing of the live HF Space
• handoff — If handing off again after further work

Next Steps (if continuing)

1. Add a min_containers=1 warmup to both Modal serves for zero cold-start latency
2. Add basic test suite: judge parsing (valid / repair / fallback), validate tool, explanation extraction, end-to-end agent with stubs
3. Stream the explanation token-by-token as the drafter writes it (the API contract would change from one-shot to SSE)
4. Cache common patterns (the same situation often comes up — "moving", "new baby", "death of grandparent") so warm requests skip the LLM
5. Polish the HF Space card README to match the new Backyard AI framing before the hackathon submission deadline