Video2Guide / README.md
Claude
Claude Opus 4.8
Add AI slide-to-transcript alignment for untimestamped images
e646192
|
Raw History Blame Contribute Delete
8.54 kB
---
title: The Describatron
emoji: πŸ€–
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
---
# The Describatron
Generate WCAG 2.2 AA-ready HTML study guides from instructional videos β€” or from
a set of slide images.
Takes a lecture plus a transcript (VTT/SRT) and produces a flowing, accessible
HTML document: visual content shown as images, narration aligned to each segment,
on-screen text captured, instructor talking-head footage filtered out, and
optional Claude-powered enrichment (scene descriptions, alt text, on-screen-text
cleanup, LaTeX math, section titles).
## Two ways in
- **From a video** (`/`) β€” upload a video + transcript; the app detects slide
changes, extracts a representative frame per scene, and filters out
talking-head shots.
- **From slide images** (`/images`) β€” already have slide exports? Upload the
images plus (optionally) a transcript. Name each image with an `HH-MM-SS`
timestamp (e.g. `00-05-30.png`) to anchor it exactly; if the names have no
timestamps, the AI reads the transcript and places each slide where it's
discussed (or, with no API key, spreads them evenly). No transcript at all β†’
a slides-only guide ordered by filename. No video upload, no scene detection.
Both paths feed the same downstream pipeline and editor.
## What it does (video path)
1. **Scene preview** β€” see extracted frames before committing; deselect any you don't need (duplicates, transitions, stray shots).
2. **Scene detection** β€” finds moments where visual content shifts (new slide, diagram, animation step). Frame extraction is deterministic (frame-accurate seeking).
3. **Instructor filtering** β€” drops frames that are mostly a talking head (face detection). Tune the threshold in the sidebar.
4. **On-screen text** β€” Tesseract OCR, or Claude reads the slide directly.
5. **Transcript alignment** β€” maps narration to each visual segment by timestamp.
6. **LLM enrichment** *(optional, requires Anthropic API key)* β€” Claude reads each frame and can produce: a **scene description** (an open paragraph under the slide that doubles as the long-form image description), WCAG-grade **alt text**, cleaned **on-screen text**, LaTeX **math equations**, and **section titles**. It can also **segment the transcript into topics** across slides.
7. **HTML generation** β€” an accessible document with proper landmarks, headings, `lang`, skip links, collapsible transcript, and MathJax-rendered equations.
## The editor and outputs
Choosing **Editor** output opens an in-browser editor to refine everything before
publishing. It supports:
- Editing titles, scene descriptions, alt text, transcript, on-screen text, and math.
- **Per-element include toggles** (and document-wide bulk toggles) β€” choose exactly
what carries into the published guide, per section: scene description, alt-text,
on-screen text, transcript, audio.
- **Save edits (JSON)** β€” a self-contained file that embeds the slides. Anyone can
reopen it at **`/edit`** and keep editing β€” no images to ship, no re-run.
- **Download** as a single self-contained **HTML** file, semantic **Markdown**, or a
**ZIP** bundle (HTML + `static/` assets).
## Accessibility
This tool produces a **WCAG 2.2 AA-ready scaffold**, not a certified-compliant
document. Auto-generated descriptions/alt-text are drafts flagged for human review
in the editor. Heading hierarchy, landmarks, `lang`, skip links, and reading order
are generated correctly out of the box.
The intended pattern is the **scene description**: an open paragraph under each
slide that both sighted and screen-reader users read in the page body. When a
description is present, the image's `alt` is emptied (decorative) so the meaning
isn't announced twice. The editor lets you confirm or rewrite each one, and choose
per-section what carries into the published guide. See `DATA_HANDLING.md` for what
data is sent where when the AI features are on.
## Quick start
Requires Python 3.10–3.13 and system packages `ffmpeg` + `tesseract-ocr`.
```bash
pip install -r requirements.txt
uvicorn app.main:app --reload
```
Then open http://localhost:8000.
### Environment variables
| Variable | Default | Description |
|---|---|---|
| `MAX_VIDEO_MB` | `500` | Video upload size cap in MB. Set to `0` to disable. |
| `MAX_TRANSCRIPT_MB` | `5` | Transcript upload size cap in MB. Set to `0` to disable. |
| `MAX_IMAGE_MB` | `25` | Per-image upload cap (images mode). Set to `0` to disable. |
| `MAX_EDIT_MB` | `150` | Cap for a self-contained edits JSON uploaded to `/edit`. |
| `ANTHROPIC_API_KEY` | *(unset)* | Enables LLM enrichment features. Can also be entered per-job in the UI. |
| `ANTHROPIC_BASE_URL` | *(unset)* | Route Anthropic requests to a custom compatible endpoint. Optional; see below. |
| `PARLEY_BASE_URL` | `https://parley.api.mit.edu` | Endpoint used for MIT Parley keys. |
| `ANTHROPIC_MODEL` | `claude-haiku-4-5-20251001` | Model ID for all LLM calls. Set this if your endpoint names models differently β€” it **must be vision-capable** or slide analysis returns blank/hallucinated descriptions. |
### API keys and MIT Parley
You can paste **either** a normal Anthropic key (`sk-ant-…`) **or** an MIT Parley
key (`sk-parley-…`) into the API-key field (or set it as `ANTHROPIC_API_KEY`).
Parley keys are automatically routed to the Parley proxy (`PARLEY_BASE_URL`);
Anthropic keys go to Anthropic. No configuration needed β€” the routing is by key
prefix. To point normal keys at some other Anthropic-compatible proxy, set
`ANTHROPIC_BASE_URL` (this never overrides Parley-key routing).
```bash
# Example: no upload limits, LLM features on
MAX_VIDEO_MB=0 ANTHROPIC_API_KEY=sk-ant-... uvicorn app.main:app --reload
```
## Project layout
```
app/
main.py FastAPI app: / (video), /images, /edit, job queue, preview + asset endpoints
jobs.py In-memory job registry (SSE progress, TTL, quota)
pipeline/
orchestrator.py Pipeline entry point (run_pipeline); video_path optional
transcript.py VTT/SRT parsing (incl. hourless timestamps) + segment alignment
scenes.py Scene detection + frame-accurate extraction
image_input.py Images mode: HH-MM-SS filename parsing β†’ SceneFrame list
filter.py Instructor-frame filtering (Haar + YuNet)
ocr.py Tesseract OCR
llm_extract.py Claude extraction (scene description, alt text, OCR, math, titles, topics)
html_gen.py WCAG HTML rendering (guide, single, review) + metadata round-trip + bundling
metadata.py Shared Pydantic schema (segments, include flags, inlined assets)
templates/
index.html Video upload form + scene picker UI
images.html Images-mode upload page
edit.html Open a shared self-contained edits JSON
guide.html Published study guide template
review.html In-browser editor template
_doc_styles.html Shared CSS for guide and review outputs
tests/
```
## Deploy to Hugging Face Spaces
This repo ships a Dockerfile compatible with [Hugging Face Spaces](https://huggingface.co/spaces) (free CPU tier).
1. Create a new Space β€” pick **Docker** as the SDK and **Blank** as the template.
2. Push this repo to the Space's git remote:
```bash
git remote add space https://huggingface.co/spaces/<your-user>/<space-name>
git push space <branch>:main
```
3. Spaces reads the YAML frontmatter at the top of this README, builds the Dockerfile, and exposes the app on port `7860`.
Notes:
- Uploads land in `/tmp` inside the container (ephemeral β€” one-shot processing per visitor).
- The free CPU tier handles a 20-minute lecture in a few minutes.
- The Space sleeps after inactivity and wakes on the next request (~30 s cold start).
- Do **not** set `X-Frame-Options` or CSP headers at the app level β€” HF Spaces proxies the app through its own infrastructure and those headers will break the Space. Apply them at the reverse-proxy / CDN layer instead.
## Status
Beta. The pipeline runs end-to-end on common inputs. Known gaps:
- A transcript file is required (there's an optional Whisper extra: `pip install -e .[asr]`).
- OCR is English-only by default; Claude-based on-screen text handles other cases better.
- Face detection uses OpenCV Haar cascades + YuNet (fast, good enough for talking heads).
- The `/edit` shared editor has no alt-frame switching or live re-OCR (those need a live job); it edits all text, descriptions, toggles, and math.