--- title: The Describatron emoji: 🤖 colorFrom: indigo colorTo: purple sdk: docker app_port: 7860 pinned: false --- # The Describatron Generate WCAG 2.2 AA-ready HTML study guides from instructional videos — or from a set of slide images. Takes a lecture plus a transcript (VTT/SRT) and produces a flowing, accessible HTML document: visual content shown as images, narration aligned to each segment, on-screen text captured, instructor talking-head footage filtered out, and optional Claude-powered enrichment (scene descriptions, alt text, on-screen-text cleanup, LaTeX math, section titles). ## Two ways in - **From a video** (`/`) — upload a video + transcript; the app detects slide changes, extracts a representative frame per scene, and filters out talking-head shots. - **From slide images** (`/images`) — already have slide exports? Upload the images plus (optionally) a transcript. Name each image with an `HH-MM-SS` timestamp (e.g. `00-05-30.png`) to anchor it exactly; if the names have no timestamps, the AI reads the transcript and places each slide where it's discussed (or, with no API key, spreads them evenly). No transcript at all → a slides-only guide ordered by filename. No video upload, no scene detection. Both paths feed the same downstream pipeline and editor. ## What it does (video path) 1. **Scene preview** — see extracted frames before committing; deselect any you don't need (duplicates, transitions, stray shots). 2. **Scene detection** — finds moments where visual content shifts (new slide, diagram, animation step). Frame extraction is deterministic (frame-accurate seeking). 3. **Instructor filtering** — drops frames that are mostly a talking head (face detection). Tune the threshold in the sidebar. 4. **On-screen text** — Tesseract OCR, or Claude reads the slide directly. 5. **Transcript alignment** — maps narration to each visual segment by timestamp. 6. **LLM enrichment** *(optional, requires Anthropic API key)* — Claude reads each frame and can produce: a **scene description** (an open paragraph under the slide that doubles as the long-form image description), WCAG-grade **alt text**, cleaned **on-screen text**, LaTeX **math equations**, and **section titles**. It can also **segment the transcript into topics** across slides. 7. **HTML generation** — an accessible document with proper landmarks, headings, `lang`, skip links, collapsible transcript, and MathJax-rendered equations. ## The editor and outputs Choosing **Editor** output opens an in-browser editor to refine everything before publishing. It supports: - Editing titles, scene descriptions, alt text, transcript, on-screen text, and math. - **Per-element include toggles** (and document-wide bulk toggles) — choose exactly what carries into the published guide, per section: scene description, alt-text, on-screen text, transcript, audio. - **Save edits (JSON)** — a self-contained file that embeds the slides. Anyone can reopen it at **`/edit`** and keep editing — no images to ship, no re-run. - **Download** as a single self-contained **HTML** file, semantic **Markdown**, or a **ZIP** bundle (HTML + `static/` assets). ## Accessibility This tool produces a **WCAG 2.2 AA-ready scaffold**, not a certified-compliant document. Auto-generated descriptions/alt-text are drafts flagged for human review in the editor. Heading hierarchy, landmarks, `lang`, skip links, and reading order are generated correctly out of the box. The intended pattern is the **scene description**: an open paragraph under each slide that both sighted and screen-reader users read in the page body. When a description is present, the image's `alt` is emptied (decorative) so the meaning isn't announced twice. The editor lets you confirm or rewrite each one, and choose per-section what carries into the published guide. See `DATA_HANDLING.md` for what data is sent where when the AI features are on. ## Quick start Requires Python 3.10–3.13 and system packages `ffmpeg` + `tesseract-ocr`. ```bash pip install -r requirements.txt uvicorn app.main:app --reload ``` Then open http://localhost:8000. ### Environment variables | Variable | Default | Description | |---|---|---| | `MAX_VIDEO_MB` | `500` | Video upload size cap in MB. Set to `0` to disable. | | `MAX_TRANSCRIPT_MB` | `5` | Transcript upload size cap in MB. Set to `0` to disable. | | `MAX_IMAGE_MB` | `25` | Per-image upload cap (images mode). Set to `0` to disable. | | `MAX_EDIT_MB` | `150` | Cap for a self-contained edits JSON uploaded to `/edit`. | | `ANTHROPIC_API_KEY` | *(unset)* | Enables LLM enrichment features. Can also be entered per-job in the UI. | | `ANTHROPIC_BASE_URL` | *(unset)* | Route Anthropic requests to a custom compatible endpoint. Optional; see below. | | `PARLEY_BASE_URL` | `https://parley.api.mit.edu` | Endpoint used for MIT Parley keys. | | `ANTHROPIC_MODEL` | `claude-haiku-4-5-20251001` | Model ID for all LLM calls. Set this if your endpoint names models differently — it **must be vision-capable** or slide analysis returns blank/hallucinated descriptions. | ### API keys and MIT Parley You can paste **either** a normal Anthropic key (`sk-ant-…`) **or** an MIT Parley key (`sk-parley-…`) into the API-key field (or set it as `ANTHROPIC_API_KEY`). Parley keys are automatically routed to the Parley proxy (`PARLEY_BASE_URL`); Anthropic keys go to Anthropic. No configuration needed — the routing is by key prefix. To point normal keys at some other Anthropic-compatible proxy, set `ANTHROPIC_BASE_URL` (this never overrides Parley-key routing). ```bash # Example: no upload limits, LLM features on MAX_VIDEO_MB=0 ANTHROPIC_API_KEY=sk-ant-... uvicorn app.main:app --reload ``` ## Project layout ``` app/ main.py FastAPI app: / (video), /images, /edit, job queue, preview + asset endpoints jobs.py In-memory job registry (SSE progress, TTL, quota) pipeline/ orchestrator.py Pipeline entry point (run_pipeline); video_path optional transcript.py VTT/SRT parsing (incl. hourless timestamps) + segment alignment scenes.py Scene detection + frame-accurate extraction image_input.py Images mode: HH-MM-SS filename parsing → SceneFrame list filter.py Instructor-frame filtering (Haar + YuNet) ocr.py Tesseract OCR llm_extract.py Claude extraction (scene description, alt text, OCR, math, titles, topics) html_gen.py WCAG HTML rendering (guide, single, review) + metadata round-trip + bundling metadata.py Shared Pydantic schema (segments, include flags, inlined assets) templates/ index.html Video upload form + scene picker UI images.html Images-mode upload page edit.html Open a shared self-contained edits JSON guide.html Published study guide template review.html In-browser editor template _doc_styles.html Shared CSS for guide and review outputs tests/ ``` ## Deploy to Hugging Face Spaces This repo ships a Dockerfile compatible with [Hugging Face Spaces](https://huggingface.co/spaces) (free CPU tier). 1. Create a new Space — pick **Docker** as the SDK and **Blank** as the template. 2. Push this repo to the Space's git remote: ```bash git remote add space https://huggingface.co/spaces// git push space :main ``` 3. Spaces reads the YAML frontmatter at the top of this README, builds the Dockerfile, and exposes the app on port `7860`. Notes: - Uploads land in `/tmp` inside the container (ephemeral — one-shot processing per visitor). - The free CPU tier handles a 20-minute lecture in a few minutes. - The Space sleeps after inactivity and wakes on the next request (~30 s cold start). - Do **not** set `X-Frame-Options` or CSP headers at the app level — HF Spaces proxies the app through its own infrastructure and those headers will break the Space. Apply them at the reverse-proxy / CDN layer instead. ## Status Beta. The pipeline runs end-to-end on common inputs. Known gaps: - A transcript file is required (there's an optional Whisper extra: `pip install -e .[asr]`). - OCR is English-only by default; Claude-based on-screen text handles other cases better. - Face detection uses OpenCV Haar cascades + YuNet (fast, good enough for talking heads). - The `/edit` shared editor has no alt-frame switching or live re-OCR (those need a live job); it edits all text, descriptions, toggles, and math.