README.md

---
title: Pocket Tutor
emoji: 🎒
colorFrom: green
colorTo: yellow
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
python_version: "3.12"
short_description: Photo-first homework coaching for students and parents
pinned: false
tags:
  - track:backyard
  - sponsor:openbmb
  - sponsor:openai
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
---

# Pocket Tutor

**Pocket Tutor** is a private, multimodal homework coach for students and parents. It accepts a worksheet photo, a typed question, or microphone input, then returns a compact tutoring plan: what the problem is asking, known information, a strategy, worked steps, a quality check, the next hint, and a parent support note.

The app runs on a dedicated OpenBMB MiniCPM-V vision-language model inside the Hugging Face Space runtime. There is no external answer API, so the tutoring flow stays local to the Space and follows the same production output structure used during fine-tuning.

Pocket Tutor is a tutoring tool, not a cheating tool. Requests for active tests or exams should be redirected toward study guidance instead of direct answers.

### Links

- **Demo**: [Video](Demo.mp4)
- **Social Post**: [LinkedIn](https://www.linkedin.com/posts/awilliams1988_buildsmall-huggingface-ai-ugcPost-7470803652820025346-SFt0)
- **GitHub Repo**: [awilliams88/pocket-tutor](https://github.com/awilliams88/pocket-tutor)
- **Hugging Face Space**: [build-small-hackathon/pocket-tutor](https://huggingface.co/spaces/build-small-hackathon/pocket-tutor)
- **Fine-tuned Model**: [build-small-hackathon/pocket-tutor-minicpmv-socratic](https://huggingface.co/build-small-hackathon/pocket-tutor-minicpmv-socratic)

## Hackathon Alignment

| Requirement | Pocket Tutor implementation |
|---|---|
| Gradio Space in `build-small-hackathon` | `build-small-hackathon/pocket-tutor` |
| Track | Backyard AI |
| Sponsor focus | OpenBMB MiniCPM-V multimodal tutoring |
| Merit targets | Off-Brand, Best Agent, Well-Tuned, Off the Grid |
| Multimodal input | Image upload/webcam, typed question, microphone transcript |
| Fine-tuning | Modal QLoRA adapter trained on app-format Socratic tutoring examples |

## What It Does

Write or upload a homework prompt, then let Pocket Tutor break it into the same structured response used by the model and UI:

| Section | Description |
|---|---|
| **Problem Read** | What the problem is asking |
| **Knowns** | Relevant facts extracted from the prompt or image |
| **Strategy** | The most useful method for solving it |
| **Worked Steps** | Compact step-by-step reasoning |
| **Check** | A quick correctness check |
| **Next Hint** | A Socratic follow-up hint |
| **Parent Note** | A short note for a parent or helper |

The app supports:

- worksheet photo input
- typed homework questions
- microphone-based question capture
- short, structured tutoring answers
- refusal of active test or exam solving

## Fine-Tuned Model

The inference engine uses a QLoRA-adapted version of [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6), trained specifically for the Pocket Tutor output format.

**Why fine-tune instead of only prompting?**
The base model is general-purpose. Fine-tuning teaches it the exact tutoring voice, the section layout, the refusal style, and the compact Socratic coaching pattern expected by the app. That makes the output more consistent than prompt-only steering.

**Training details:**
- Method: QLoRA with 4-bit NF4 quantization
- Hardware: Modal A10G
- Dataset: 16 structured tutoring examples plus 7 schema-aligned follow-up coaching examples
- Output format: the seven current production UI sections, not an older seed format
- Coverage: math, science, reading, writing, statistics, blurry image recovery, parent notes, and cheat-refusal examples
- Validation: Modal smoke test passes across all structured and follow-up examples
- Runtime pairing: the same MiniCPM-V base model used by the Space

The fine-tuned LoRA adapter is published at [`build-small-hackathon/pocket-tutor-minicpmv-socratic`](https://huggingface.co/build-small-hackathon/pocket-tutor-minicpmv-socratic) and is loaded automatically on top of the base model at startup.

## Inference Architecture

```
User Input (photo, text, or mic transcript)
        │
        ▼
┌─────────────────────┐
│    Gradio UI        │  ui/layout.py — dark classroom dashboard
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│  Analyzer           │  core/analyzer.py — prompt construction & dispatch
└──────────┬──────────┘
           │
     ┌─────┴──────┐
     ▼            ▼
┌─────────┐  ┌──────────┐
│Inference│  │  Parser  │  core/inference.py — model execution
│ Engine  │  │  Engine  │  core/parser.py — section splitting
└────┬────┘  └──────────┘
     │
     └── Space runtime: MiniCPM-V base model + LoRA adapter via `PeftModel`
```

**Inference priority:**
1. Dedicated multimodal model on the Space runtime with the fine-tuned adapter.
2. If model execution fails, return a deterministic tutoring scaffold instead of silently failing.
3. Do not route homework text to a separate answer API.

## Local Development

**Setup:**
```bash
./run.sh setup
```

**Run locally:**
```bash
./run.sh app
```

**Quality checks**:
```bash
./run.sh verify
```

**Smoke tests**:
```bash
modal run modal/smoke.py
python scripts/smoke_space.py
```

The Modal smoke test loads the production base model and LoRA adapter on Modal and checks every training and follow-up example against the seven-section UI contract. To smoke test only one prompt, run:

```bash
SMOKE_ALL_EXAMPLES=0 modal run modal/smoke.py
```

The Space smoke test calls the deployed Gradio API endpoint and verifies that all seven user-facing sections are generated by the model instead of falling back to default placeholder text.

## Codebase

### Root
| File | Purpose |
|---|---|
| `app.py` | Hugging Face Spaces entry point |

### `env/` - App infrastructure
| File | Purpose |
|---|---|
| `env/config.py` | Central constants - model IDs, repo URLs, limits |
| `env/runtime.py` | Env var loader and asyncio cleanup patch |

### `core/` - Business logic
| File | Purpose |
|---|---|
| `core/analyzer.py` | Tutoring orchestrator and model dispatch |
| `core/inference.py` | Lazy model loader and local inference |
| `core/parser.py` | Output section splitter |

### `ui/` - Presentation
| File | Purpose |
|---|---|
| `ui/layout.py` | Gradio layout, components, and event wiring |
| `ui/styles.py` | Custom chalkboard CSS and component styling |

### `modal/` - Remote fine-tuning
| File | Purpose |
|---|---|
| `modal/tune.py` | QLoRA fine-tuning orchestrator |
| `modal/dataset.py` | Tutoring dataset and prompt builders |
| `modal/smoke.py` | Modal smoke test for the fine-tuned adapter and section contract |
| `modal/CARD.md` | Hugging Face model card for the LoRA adapter |

### `scripts/` - Deployment checks
| File | Purpose |
|---|---|
| `scripts/smoke_space.py` | Live Space API smoke test |

## Tech Stack

- **Model**: `openbmb/MiniCPM-V-4.6` + custom LoRA adapter (`build-small-hackathon/pocket-tutor-minicpmv-socratic`)
- **Fine-tuning**: QLoRA via `peft` + `trl` SFTTrainer on Modal A10G
- **Inference**: `transformers` + `peft` + `accelerate`
- **UI**: Gradio 6 with custom CSS
- **Hosting**: Hugging Face Spaces
- **Sponsor**: [OpenBMB](https://github.com/OpenBMB)

## Safety

Pocket Tutor is designed to teach reasoning, not help students cheat. It should redirect active test and exam requests toward study guidance and worked practice.

## Training Data

The Modal dataset includes math, science, reading, writing, statistics, blurry image recovery, parent-note coaching, schema-aligned follow-up turns, and explicit cheating refusal examples. The training target is the current production UI format, not an earlier seed format.