IDEATION.md

Build Small Hackathon — Ideation & Decisions

Working notes from ideation. Project: AI Prof (primary submission). Hackathon: https://huggingface.co/build-small-hackathon · Field guide: https://build-small-hackathon-field-guide.hf.space/

Hackathon constraints (the rules that shape everything)

• Model size: ≤32B parameters PER MODEL (not aggregate). You can freely combine multiple small models as long as each one individually stays under the cap. ("not just active params")
• No sponsor exclusivity — a sponsor's model just has to be a core part of the experience; you may mix in other providers' models.
• Platform: must be built in Gradio, hosted as a Hugging Face Space.
• Deliverables: working Space + demo video + social media post.
• Timeline: hack window June 5–15, 2026.
• Credits: OpenAI $100 is for Codex (their coding agent), not API/inference. Modal $250, HF $20.

Tracks

• Backyard AI — solve a genuine problem for someone you personally know. Judged on specificity, actual user adoption, and appropriate model fit. ← our track.
• Thousand Token Wood (TTW) — delightful/unconventional, AI must be load-bearing (game, story, art).

Prize surface (it stacks — one app can hit many)

Main tracks ($18k): 1st–4th per track + Community Choice ($2k). Sponsor awards:

• OpenBMB $10k — build with MiniCPM (incl. vision MiniCPM-V / omni MiniCPM-o).
• OpenAI $10k — won via Codex-attributed commits in the repo/Space (build-tool track, model-agnostic).
• NVIDIA — two RTX 5080 GPUs; build with Nemotron. One for "best space", one for community engagement (likes).
• Modal $20k credits — use Modal for dev/runtime, note in README. Special awards ($8k): Bonus Quest Champion $2k · Off-Brand (best custom UI via gr.Server) $1.5k · Tiny Titan (best app on ≤4B model) $1.5k · Best Demo $1k · Best Agent $1k · Judges' Wildcard $1k. Six merit badges: Off the Grid (no cloud APIs) · Well-Tuned (publish fine-tune) · Off-Brand (custom UI) · Llama Champion (llama.cpp runtime) · Sharing is Caring (publish agent trace) · Field Notes (build blog).

Decision: build the AI Prof (Backyard AI)

Problem (real): a specific classmate finds that having the slides isn't enough — they're static and lack the in-class explanation. Test with real, anonymized lecture slides from our class.

Core loop: upload lecture PDF → model reads each slide as an image → explains it like a TA, streamed in real time → classmate can interject with a question at any moment.

Two-model architecture (stacks OpenBMB + NVIDIA, cleanly justified)

• MiniCPM-V (~4.1B, GGUF) = the eyes. Reads each slide as an image (diagrams, equations, layout — not just scraped text). Core → satisfies OpenBMB. Runs via llama.cpp → Llama Champion badge.
• Nemotron 3 Nano (9B, or 30B-A3B MoE = only ~3.6B active → fast decode) = the brain. Turns the slide reading into the spoken explanation and answers interjections. Reasoning/agentic → NVIDIA fit.
• Per-model cap means no param-budget tension between the two. Division of labor (see/explain) survives the "core part of the experience" test → not sponsor-stuffing.

Prize map for this one app

Backyard placement · OpenBMB $10k · NVIDIA RTX 5080 · OpenAI $10k (Codex commits) · Modal $20k (self-host) · Off-Brand $1.5k+badge (custom whiteboard UI) · Llama Champion · Off the Grid · Sharing is Caring (publish a teaching-session trace) · Field Notes (blog) · Best Demo. → realistically 6+ surfaces.

Scope discipline (vertical slice order)

1. Slide upload → MiniCPM-V reads → Nemotron explains, streamed in a simple custom UI. (Submittable alone.)
2. Text interjection (pause, ask, resume).
3. Whiteboard — model emits Mermaid / Excalidraw JSON (structured; small models do this reliably). Avoid freeform tldraw generation — that's the day-eating trap.
4. Voice / TTS — only with slack.

Real-time architecture

Goal: feel like a live lecture — explanation streams as if the prof is talking through each slide, smooth slide-to-slide, interruptible mid-sentence.

• Token streaming: stream Nemotron output to the UI (Gradio generators) so it appears as it's generated.
• Complete index before teaching: process the full deck before starting the lecture. The professor needs a global map to choose slides intelligently and answer interjections. Show preparation progress in Gradio.
• Two-stage per slide, cached: (a) MiniCPM-V → a structured "slide reading" (text + diagram desc + equations), computed once and cached; (b) Nemotron → the explanation. Interjections reuse cached (a), never re-run vision.
• Interjection = interrupt + branch: need a cancellable generation (async cancel token / threading.Event). On input: stop current stream, answer using the cached slide reading + history as context, then resume.
• Streaming TTS (optional): chunk explanation into sentences, TTS each as generated, play sequentially. Barge-in (interrupt-on-speech) is hard mode — defer.
• Session state: { slides[], current_index, cached_slide_readings{}, conversation_history }.
• Hosting tradeoff for low latency: free HF Space hardware likely too slow for 2 models in real time. Lean: Modal GPU as inference backend (uses the Modal track) + Gradio Space as frontend. Self-hosting (not an external inference API) still supports the Off the Grid badge.

Voice (STT + TTS) — makes it feel like class, but it's the deepest rabbit hole

Two more tiny, on-HF models (free under the ≤32B-per-model cap; FAQ uses "a 7B speech model" as its example). Keep both open + self-hosted (not ElevenLabs/Deepgram/OpenAI API) → protects the Off the Grid badge.

Pipeline: TTS (out) speaks Nemotron's streamed explanation; STT (in) transcribes the spoken interjection into text for Nemotron; VAD is the referee that detects when the classmate starts talking → triggers barge-in.

• STT (interjections) — pick: Whisper, or Moonshine for speed.
  • faster-whisper / distil-whisper (large-v3 ~1.5B, or base/small for latency) — accurate, OpenAI open weights.
  • Moonshine (~tiny) — built for real-time/on-device, faster time-to-text on short clips.
  • Interjections are short → small model is fine; time-to-text, not accuracy, is the bottleneck. Start with faster-whisper-base, switch to Moonshine only if laggy.
• TTS (narration) — pick: Kokoro, fallback Piper.
  • Kokoro-82M (hexgrad/Kokoro-82M) — 82M, good quality, fast time-to-first-audio, streamable. Sweet spot.
  • Piper — even lower latency, CPU-friendly, slightly more robotic. Use if speech layer isn't on GPU.
  • Stream sentence-by-sentence: synthesize + play each sentence as Nemotron emits it (audio ~1 sentence behind gen).
• Barge-in / turn-taking — use FastRTC (fastrtc, the Gradio/HF WebRTC stack). Gives low-latency mic+playback over WebRTC, built-in VAD turn detection (ReplyOnPause), and the hook to cancel playback + generation the instant the user speaks. Avoids hand-rolling silence detection; reuses our cancellable-generation design.
  • Loop: narrate (TTS) → Silero VAD hears speech → kill TTS + cancel Nemotron → buffer to end-of-speech → STT → answer using the cached slide reading as context → TTS answer → resume narration.
• Latency budget (what "feels live" needs): minimize time-to-first-audio. STT small (100–300ms) + Nemotron MoE fast prefill + Kokoro first chunk (tens of ms). Run STT/TTS/VAD on the same Modal GPU as Nemotron (or CPU for Piper+Silero) to avoid network hops.
• Scope ladder (don't let voice eat the week):
  1. TTS narration only — Prof talks, classmate types to interject. Low risk, already feels real-time.
  2. Push-to-talk interject — hold key / tap to ask aloud → STT. No VAD/barge-in. ~90% of magic, ~30% of work.
  3. Full-duplex barge-in via FastRTC + VAD — only once 1–2 are solid. The 2→3 jump is where time goes.
• Model count after voice: MiniCPM-V (vision) + Nemotron (brain) + Whisper/Moonshine (STT) + Kokoro/Piper (TTS)
  • Silero VAD. Multi-model is explicitly blessed; more "appropriate model fit" surface, but more integration.

Agent loop, tools & slide grounding

The brain (Nemotron) runs as a tool-using agent driving the lecture — professor-esque: decides which slide to be on, when to draw vs. just talk, when to jump back. (Strengthens the Best Agent award; the tool-call sequence is the publishable agent trace → Sharing is Caring badge.)

Tools (mutate UI / session state):

• goto_slide(i) / next_slide() / prev_slide() — navigation; lets it jump back to a referenced slide.
• look_closer(question) — on-demand real-time MiniCPM-V call on the current slide image for detail.
• draw(mermaid | excalidraw_json) — render on the whiteboard surface.
• clear_whiteboard()
• highlight_region(bbox) — optional, later.
• narration = the free-text part of the response → streamed to TTS.

Control flow: orchestrator loop. Each turn the agent gets { current slide reading, deck outline, recent history, trigger } where trigger = "continue lecture" | "user asked: ". It returns narration + optional tool calls; the orchestrator executes tools (swap displayed slide, render whiteboard) and streams narration. Reserve heavy reasoning for decision points (between slides), not during narration, or latency balloons.

Grounding — preprocess (breadth) + real-time (depth), both:

• On upload (once; complete before lecture begins):
  • render each slide → image (serves both display and vision),
  • MiniCPM-V → cached structured slide reading (title, bullets, equations, diagram desc, key concepts),
  • PDF text-layer extraction (exact text ground-truth; vision misreads text) to complement vision,
  • build a deck outline / index (slide → title/concepts) — this is what lets the agent plan and pick a slide; it can't navigate to a slide it has never seen.
• During lecture (look_closer): targeted MiniCPM-V look at the actual slide with the specific question in context — handles the long tail (specific visual questions, detail the cached summary glossed over).
• Why both: preprocess = global map + fast/cheap narration + navigation; real-time = accuracy on specific asks.

UI: show the REAL slide, never a summary.

• Main surface = the actual rendered slide image / PDF page, synced to the agent's current_slide.
• Whiteboard = a separate adjacent canvas (Mermaid/Excalidraw render) so drawings read as the prof's annotations, not edits to the original slide. (Region-highlight overlay on the slide can come later.)
• Plus: live caption / transcript + mic / interject control.

The detailed deployment, deck-cache, teaching-beat, interruption, speech, and whiteboard decisions are in ARCHITECTURE.md.

Fine-tuning (after core pipeline — not before)

Chosen direction: teaching-style SFT / guided learning — tune the brain (Nemotron) to explain like a good TA: analogies, checks for understanding, concise, no preamble. Bootstrap dataset = (slide reading → ideal explanation) pairs generated with a strong model + a little hand-curation; QLoRA via TRL / Unsloth on Modal; publish the adapter to HF → Well-Tuned badge (+ feeds Bonus Quest Champion). Put a before/after note in the model card. Strictly a step-2 enhancement — only once the live pipeline works and there's a clear objective.

Backburner ideas (TTW — only if a 2nd submission is feasible)

• Emotion-driven TRPG / choose-your-own-adventure: free text → LLM updates structured NPC emotional state (trust/fear/affection…) → state steers the story. Best Agent + Sharing is Caring (emotion-delta traces).
• AI Garlic Phone: message/drawing passed down a chain of model personas, mutating; needs a reveal payoff. If a drawing is passed, pulls in vision = OpenBMB again.
• Sketch-to-story adventure: you draw your action, MiniCPM-V interprets the doodle, the world reacts. Fuses vision + emotion mechanic. Strong demo, only-AI-can-do-this.
• Model-vs-model battle → traces for training (browser-brawl-style, different domain): most technically impressive (Best Agent + Well-Tuned + Sharing is Caring) but research-shaped; the training loop can eat the week.

Key model references

• MiniCPM-V-4 (4.1B, multimodal): https://huggingface.co/openbmb/MiniCPM-V-4
• MiniCPM-V-4 GGUF (llama.cpp): https://huggingface.co/openbmb/MiniCPM-V-4-gguf
• Nemotron 3 Nano collection: https://huggingface.co/collections/nvidia/nvidia-nemotron-v3
• Nemotron 3 Nano 4B: https://huggingface.co/blog/nvidia/nemotron-3-nano-4b
• Nemotron 3 Nano 30B-A3B: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16