README.md

---
title: Tutori — Your Whiteboard Tutor
emoji: ✏️
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
fullWidth: true
header: mini
pinned: false
license: apache-2.0
short_description: Voice tutor that sketches on a whiteboard while it talks
tags:
  - build-small-hackathon
  - backyard ai
  - off the grid
  - off-brand
  - sharing is caring
  - agent
  - education
  - speech
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - achievement:offgrid
  - achievement:offbrand
  - achievement:sharing
  - achievement:fieldnotes
models:
  - google/gemma-4-12B-it
  - bosonai/higgs-audio-v2-generation-3B-base
  - openbmb/MiniCPM5-1B
  - openai/whisper-large-v3-turbo
  - nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
  - ProCreations/tutori-board-nemotron
  - ProCreations/tutori-board-gemma
datasets:
  - ProCreations/tutori-whiteboard-lessons
---

# ✏️ Tutori — your whiteboard tutor

## 🎬 Demo

<video src="https://huggingface.co/spaces/build-small-hackathon/tutori/resolve/main/tutori_demo.mp4" controls width="100%"></video>

*([demo video file](./tutori_demo.mp4) — narrated by Tutori's own voice; music synthesized from scratch.)*

**Speak a question. Tutori researches it, then teaches you out loud while
sketching the idea on a whiteboard — in real time, stroke by stroke, in sync
with its voice.**

Built for the [HF Build Small Hackathon](https://huggingface.co/build-small-hackathon):
every model runs **on this Space itself** via ZeroGPU. No cloud APIs, no keys.

## Why Backyard AI 🏡

I built Tutori for my parents. They're quite behind on today's technology —
especially AI — and the firehose of new models and jargon is impenetrable
from the outside. With Tutori they can just **ask**: "what is Gemma?",
"what happened in AI this month?" — and get a patient, spoken explanation,
drawn out on a whiteboard, at their pace, with the research done for them.
They've actually been using it, and they find it genuinely useful for
finally keeping up.

## The stack (Σ 16.9B params — well under the 32B cap)

| Role | Model | Params |
|---|---|---|
| 🧠 Teacher + vision | [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) | 12B |
| 🧭 Research planner + study coach | [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) | 1B |
| 🗣️ Expressive voice | [bosonai/higgs-audio-v2-generation-3B-base](https://huggingface.co/bosonai/higgs-audio-v2-generation-3B-base) | 3B |
| 👂 Speech recognition | [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | 0.8B |

**Gemma teaches. MiniCPM plans and coaches.** Every turn, MiniCPM5 1B decides
whether the question needs fresh facts and writes the search queries — the
agentic step that turns a chatbot into a researcher. Gemma 4 then teaches
from what was found, Higgs speaks it, Whisper listens. And after every
lesson, MiniCPM comes back as the **study coach**: it updates the learner's
profile and writes three personalized follow-up questions that land on the
sticky notes under the chat — tap one and the lesson continues where your
curiosity points.

### Engineering notes (the honest kind)

- We tried to ship **Nemotron ASR** as the ears — three separate times. NeMo
  in the main process crashes ZeroGPU's forked workers ("GPU task aborted");
  lazy-loading inside the worker costs a fresh worker its whole turn (we
  re-measured on June 12 with a dedicated probe Space: **57.1 s just to
  restore the 0.6B streaming model**, against a 59 s turn budget, paid every
  turn because workers are disposable); and a CPU sidecar measured RTF ≈ 24.
  So Whisper turbo keeps the ears — it preloads with everything else and
  transcribes in about a second.
- Higgs Audio **v3** TTS ships only for the SGLang-Omni serving stack (needs
  a persistent GPU), so we use v2 — same family, natively in transformers.
- The live whiteboard now has a deterministic **diagram specialist** between
  Gemma and the renderer. When the lesson lands on a known teaching family
  (rockets to orbit, gradient descent, Pythagorean theorem, neural networks,
  photosynthesis, supply/demand, binary search, recursion, rainbows, water
  cycle), Tutori compiles the board from hand-authored diagram ops instead of
  asking a language model to freehand coordinates. Unknown topics still use
  the model's drawing, then pass through the same no-overlap layout engine.
- We **LoRA-fine-tuned two dedicated whiteboard artists** on a purpose-built
  dataset ([tutori-whiteboard-lessons](https://huggingface.co/datasets/ProCreations/tutori-whiteboard-lessons):
  7,109 gold lesson steps, 8 diagram families, 78 topics, every one validated
  to render with zero overlapping elements). Both are integrated behind a
  `TUTORI_BOARD_MODEL` flag — and the flag is **off**. The honest scorecard:
  - [**Nemotron 3 Nano 4B artist**](https://huggingface.co/ProCreations/tutori-board-nemotron)
    (eval loss 0.021): can't run here — Nemotron-H's Mamba-2 layers need the
    fused `mamba-ssm` Triton kernels, which ZeroGPU's fresh-per-turn workers
    can't JIT inside the 59 s turn budget (the pure-PyTorch fallback OOMs at
    prefill instead). The same constraint broke Nemotron's earlier role as
    research planner — which is how MiniCPM5 1B got the job, and it turned
    out to be excellent at it.
  - [**Gemma 4 12B artist**](https://huggingface.co/ProCreations/tutori-board-gemma)
    (eval loss 0.024, rides the already-loaded teacher as a LoRA): this one
    DID run live on ZeroGPU, and drew textbook diagrams on eval topics —
    correct loss-curve axes, properly labelled hypotenuses. But in real use
    it lost to the boring pipeline: the artist never sees the researched
    facts (only the topic and narration), so current-events lessons got
    improvised geometry, and sharing one model for narration + rendering
    serialized the turn past the GPU window. We kept the better teacher.
  On a persistent GPU (`TUTORI_REAL=1` locally) both artists work — flip the
  flag and they draw. ZeroGPU giveth, ZeroGPU taketh away.

## How a turn works

1. **You talk** (or type). Whisper turbo transcribes you on-device.
2. **MiniCPM plans.** MiniCPM5 1B decides whether the question needs fresh
   facts; if so it writes the search queries and Tutori pulls snippets +
   page text from the web (DuckDuckGo, keyless). Timeless topics skip
   straight to teaching.
3. **It teaches in steps.** Gemma 4 emits a JSON lesson script — each step is
   a sentence to *say* plus whiteboard ops to *draw* (boxes, arrows, curves,
   axes, highlights in a 100×75 coordinate space). A semantic board layer
   upgrades high-confidence topics into deterministic textbook diagrams before
   the layout pass.
4. **Steps stream.** The moment the first step's JSON closes, Higgs Audio
   voices it and ships it to your browser — Tutori starts talking while the
   rest of the lesson is still being generated.
5. **The board draws itself in sync.** A hand-drawn canvas renderer animates
   each stroke across exactly the duration of that step's audio — and when a
   lesson is drawing-heavy, the agent expands the whiteboard to take over the
   page, nudging the rest of the UI aside until the next lesson.
6. **The coach debriefs.** MiniCPM5 updates your learner profile and swaps
   the sticky notes for three follow-up questions tailored to what you just
   learned.

## The smart context system

- **Learner profile** — Tutori keeps gentle notes (level, goals, what clicked,
  what confused you) that it updates every turn and folds into the next
  lesson. Stored **only in your browser** (`BrowserState`), fully inspectable
  and erasable in the UI.
- **Pace dial** — 1 (total beginner, tiny steps, analogies) → 5 (expert,
  dense). Injected straight into the teaching prompt.
- **It can see your drawings** — sketch on the board with the pen tools and
  hit “🖐 Ask Tutori about the board”: Gemma 4’s vision reads your strokes.

## Running it

- **On Spaces (the real thing):** select **ZeroGPU** hardware. All four
  models load at startup and get packed by ZeroGPU; each turn runs in a
  single `@spaces.GPU(duration=59)` generator call, sized so even
  **logged-out visitors** (120s/day ZeroGPU quota) get a full lesson. Sign in
  to Hugging Face for much more daily GPU time.
- **Locally (no GPU needed):** `pip install gradio soundfile numpy && python app.py`
  runs a mock engine so you can play with the whiteboard, voice flow, and UI.
  Set `TUTORI_REAL=1` on a CUDA machine to use the real models.

## Submission

- **Space:** you're looking at it 🙂
- **Demo:** the video at the top of this card ([file](./tutori_demo.mp4))
- **Social Media Post:** https://x.com/SSHTheDev/status/2065159474671653005

## Merit badges claimed

- 🔌 **Off the Grid** — zero cloud model APIs; the only network egress is
  optional keyless web search, and you can switch that off too.
- 🎨 **Off-Brand** — a full "teacher's studio" design system over Gradio:
  chalkboard scene, hand-chalked header, paper cards with washi tape,
  sticky-note suggestion chips, marker-cap toolbar, self-hosted fonts
  (zero external requests) — plus the custom hand-drawn whiteboard renderer
  (vanilla canvas, ~600 lines of sketchy-stroke drawing) and an
  agent-controlled expanding board.
- 🤝 **Sharing is Caring** — verbatim agent traces from live sessions are
  published in [`traces/`](./traces): the Nemotron planner's search queries,
  every spoken sentence, and every whiteboard op with coordinates, timestamped.
- 📓 **Field Notes** — a full write-up of what we built and learned,
  published on the hackathon org blog:
  [Building Tutori, a Whiteboard Tutor That Draws While It Talks](https://huggingface.co/blog/build-small-hackathon/tutori)
  (source: [docs/field-notes.md](./docs/field-notes.md)).
- 🎯 **Well-Tuned** — we LoRA-fine-tuned TWO whiteboard artists
  ([Nemotron 3 Nano 4B](https://huggingface.co/ProCreations/tutori-board-nemotron)
  and [Gemma 4 12B](https://huggingface.co/ProCreations/tutori-board-gemma))
  on a purpose-built, programmatically validated dataset
  ([tutori-whiteboard-lessons](https://huggingface.co/datasets/ProCreations/tutori-whiteboard-lessons))
  and published all three. Both run flag-gated (`TUTORI_BOARD_MODEL=nemotron|gemma`);
  the Gemma artist was A/B-tested live on this very Space and honestly
  retired — full scorecard in the engineering notes above.