README.md

metadata

title: Tutori — Your Whiteboard Tutor
emoji: ✏️
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
fullWidth: true
header: mini
pinned: false
license: apache-2.0
short_description: Voice tutor that sketches on a whiteboard while it talks
tags:
  - build-small-hackathon
  - backyard ai
  - off the grid
  - off-brand
  - sharing is caring
  - agent
  - education
  - speech
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - achievement:offbrand
  - achievement:sharing
  - achievement:fieldnotes
models:
  - google/gemma-4-12B-it
  - bosonai/higgs-audio-v2-generation-3B-base
  - openbmb/MiniCPM5-1B
  - openai/whisper-large-v3-turbo
  - nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
  - ProCreations/tutori-board-nemotron
  - ProCreations/tutori-board-gemma
datasets:
  - ProCreations/tutori-whiteboard-lessons

✏️ Tutori — your whiteboard tutor

🎬 Demo

(demo video file — narrated by Tutori's own voice; music synthesized from scratch.)

Speak a question. Tutori researches it, then teaches you out loud while sketching the idea on a whiteboard — in real time, stroke by stroke, in sync with its voice.

Built for the HF Build Small Hackathon: every model runs on this Space itself via ZeroGPU. No cloud APIs, no keys.

Why Backyard AI 🏡

I built Tutori for my parents. They're quite behind on today's technology — especially AI — and the firehose of new models and jargon is impenetrable from the outside. With Tutori they can just ask: "what is Gemma?", "what happened in AI this month?" — and get a patient, spoken explanation, drawn out on a whiteboard, at their pace, with the research done for them. They've actually been using it, and they find it genuinely useful for finally keeping up.

The stack (Σ 16.9B params — well under the 32B cap)

Role	Model	Params
🧠 Teacher + vision	google/gemma-4-12B-it	12B
🧭 Research planner + study coach	openbmb/MiniCPM5-1B	1B
🗣️ Expressive voice	bosonai/higgs-audio-v2-generation-3B-base	3B
👂 Speech recognition	openai/whisper-large-v3-turbo	0.8B

Gemma teaches. MiniCPM plans and coaches. Every turn, MiniCPM5 1B decides whether the question needs fresh facts and writes the search queries — the agentic step that turns a chatbot into a researcher. Gemma 4 then teaches from what was found, Higgs speaks it, Whisper listens. And after every lesson, MiniCPM comes back as the study coach: it updates the learner's profile and writes three personalized follow-up questions that land on the sticky notes under the chat — tap one and the lesson continues where your curiosity points.

Engineering notes (the honest kind)

• We tried to ship Nemotron ASR as the ears — three separate times. NeMo in the main process crashes ZeroGPU's forked workers ("GPU task aborted"); lazy-loading inside the worker costs a fresh worker its whole turn (we re-measured on June 12 with a dedicated probe Space: 57.1 s just to restore the 0.6B streaming model, against a 59 s turn budget, paid every turn because workers are disposable); and a CPU sidecar measured RTF ≈ 24. So Whisper turbo keeps the ears — it preloads with everything else and transcribes in about a second.
• Higgs Audio v3 TTS ships only for the SGLang-Omni serving stack (needs a persistent GPU), so we use v2 — same family, natively in transformers.
• The live whiteboard now has a deterministic diagram specialist between Gemma and the renderer. When the lesson lands on a known teaching family (rockets to orbit, gradient descent, Pythagorean theorem, neural networks, photosynthesis, supply/demand, binary search, recursion, rainbows, water cycle), Tutori compiles the board from hand-authored diagram ops instead of asking a language model to freehand coordinates. Unknown topics still use the model's drawing, then pass through the same no-overlap layout engine.
• We LoRA-fine-tuned two dedicated whiteboard artists on a purpose-built dataset (tutori-whiteboard-lessons: 7,109 gold lesson steps, 8 diagram families, 78 topics, every one validated to render with zero overlapping elements). Both are integrated behind a TUTORI_BOARD_MODEL flag — and the flag is off. The honest scorecard:
  • Nemotron 3 Nano 4B artist (eval loss 0.021): can't run here — Nemotron-H's Mamba-2 layers need the fused mamba-ssm Triton kernels, which ZeroGPU's fresh-per-turn workers can't JIT inside the 59 s turn budget (the pure-PyTorch fallback OOMs at prefill instead). The same constraint broke Nemotron's earlier role as research planner — which is how MiniCPM5 1B got the job, and it turned out to be excellent at it.
  • Gemma 4 12B artist (eval loss 0.024, rides the already-loaded teacher as a LoRA): this one DID run live on ZeroGPU, and drew textbook diagrams on eval topics — correct loss-curve axes, properly labelled hypotenuses. But in real use it lost to the boring pipeline: the artist never sees the researched facts (only the topic and narration), so current-events lessons got improvised geometry, and sharing one model for narration + rendering serialized the turn past the GPU window. We kept the better teacher. On a persistent GPU (TUTORI_REAL=1 locally) both artists work — flip the flag and they draw. ZeroGPU giveth, ZeroGPU taketh away.

How a turn works

1. You talk (or type). Whisper turbo transcribes you on-device.
2. MiniCPM plans. MiniCPM5 1B decides whether the question needs fresh facts; if so it writes the search queries and Tutori pulls snippets + page text from the web (DuckDuckGo, keyless). Timeless topics skip straight to teaching.
3. It teaches in steps. Gemma 4 emits a JSON lesson script — each step is a sentence to say plus whiteboard ops to draw (boxes, arrows, curves, axes, highlights in a 100×75 coordinate space). A semantic board layer upgrades high-confidence topics into deterministic textbook diagrams before the layout pass.
4. Steps stream. The moment the first step's JSON closes, Higgs Audio voices it and ships it to your browser — Tutori starts talking while the rest of the lesson is still being generated.
5. The board draws itself in sync. A hand-drawn canvas renderer animates each stroke across exactly the duration of that step's audio — and when a lesson is drawing-heavy, the agent expands the whiteboard to take over the page, nudging the rest of the UI aside until the next lesson.
6. The coach debriefs. MiniCPM5 updates your learner profile and swaps the sticky notes for three follow-up questions tailored to what you just learned.

The smart context system

• Learner profile — Tutori keeps gentle notes (level, goals, what clicked, what confused you) that it updates every turn and folds into the next lesson. Stored only in your browser (BrowserState), fully inspectable and erasable in the UI.
• Pace dial — 1 (total beginner, tiny steps, analogies) → 5 (expert, dense). Injected straight into the teaching prompt.
• It can see your drawings — sketch on the board with the pen tools and hit “🖐 Ask Tutori about the board”: Gemma 4’s vision reads your strokes.

Running it

• On Spaces (the real thing): select ZeroGPU hardware. All four models load at startup and get packed by ZeroGPU; each turn runs in a single @spaces.GPU(duration=59) generator call, sized so even logged-out visitors (120s/day ZeroGPU quota) get a full lesson. Sign in to Hugging Face for much more daily GPU time.
• Locally (no GPU needed): pip install gradio soundfile numpy && python app.py runs a mock engine so you can play with the whiteboard, voice flow, and UI. Set TUTORI_REAL=1 on a CUDA machine to use the real models.

Submission

• Space: you're looking at it 🙂
• Demo: the video at the top of this card (file)
• Social Media Post: https://x.com/SSHTheDev/status/2065159474671653005

Merit badges claimed

• 🔌 Off the Grid — zero cloud model APIs; the only network egress is optional keyless web search, and you can switch that off too.
• 🎨 Off-Brand — a full "teacher's studio" design system over Gradio: chalkboard scene, hand-chalked header, paper cards with washi tape, sticky-note suggestion chips, marker-cap toolbar, self-hosted fonts (zero external requests) — plus the custom hand-drawn whiteboard renderer (vanilla canvas, ~600 lines of sketchy-stroke drawing) and an agent-controlled expanding board.
• 🤝 Sharing is Caring — verbatim agent traces from live sessions are published in traces/: the Nemotron planner's search queries, every spoken sentence, and every whiteboard op with coordinates, timestamped.
• 📓 Field Notes — a full write-up of what we built and learned, published on the hackathon org blog: Building Tutori, a Whiteboard Tutor That Draws While It Talks (source: docs/field-notes.md).
• 🎯 Well-Tuned — we LoRA-fine-tuned TWO whiteboard artists (Nemotron 3 Nano 4B and Gemma 4 12B) on a purpose-built, programmatically validated dataset (tutori-whiteboard-lessons) and published all three. Both run flag-gated (TUTORI_BOARD_MODEL=nemotron|gemma); the Gemma artist was A/B-tested live on this very Space and honestly retired — full scorecard in the engineering notes above.