Spaces:
Sleeping
Sleeping
File size: 8,538 Bytes
2128d4b 2f32611 2128d4b 2f32611 b6f361a 637c2d1 e646192 637c2d1 b6f361a 637c2d1 b6f361a 71844d0 b6f361a 71844d0 637c2d1 71844d0 897f74e 5125240 897f74e 71844d0 b6f361a 637c2d1 b6f361a 637c2d1 71844d0 b6f361a 637c2d1 b6f361a 637c2d1 71844d0 b6f361a 2128d4b 71844d0 2128d4b 71844d0 2128d4b 71844d0 2128d4b 71844d0 2128d4b b6f361a 71844d0 637c2d1 71844d0 637c2d1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | ---
title: The Describatron
emoji: π€
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
---
# The Describatron
Generate WCAG 2.2 AA-ready HTML study guides from instructional videos β or from
a set of slide images.
Takes a lecture plus a transcript (VTT/SRT) and produces a flowing, accessible
HTML document: visual content shown as images, narration aligned to each segment,
on-screen text captured, instructor talking-head footage filtered out, and
optional Claude-powered enrichment (scene descriptions, alt text, on-screen-text
cleanup, LaTeX math, section titles).
## Two ways in
- **From a video** (`/`) β upload a video + transcript; the app detects slide
changes, extracts a representative frame per scene, and filters out
talking-head shots.
- **From slide images** (`/images`) β already have slide exports? Upload the
images plus (optionally) a transcript. Name each image with an `HH-MM-SS`
timestamp (e.g. `00-05-30.png`) to anchor it exactly; if the names have no
timestamps, the AI reads the transcript and places each slide where it's
discussed (or, with no API key, spreads them evenly). No transcript at all β
a slides-only guide ordered by filename. No video upload, no scene detection.
Both paths feed the same downstream pipeline and editor.
## What it does (video path)
1. **Scene preview** β see extracted frames before committing; deselect any you don't need (duplicates, transitions, stray shots).
2. **Scene detection** β finds moments where visual content shifts (new slide, diagram, animation step). Frame extraction is deterministic (frame-accurate seeking).
3. **Instructor filtering** β drops frames that are mostly a talking head (face detection). Tune the threshold in the sidebar.
4. **On-screen text** β Tesseract OCR, or Claude reads the slide directly.
5. **Transcript alignment** β maps narration to each visual segment by timestamp.
6. **LLM enrichment** *(optional, requires Anthropic API key)* β Claude reads each frame and can produce: a **scene description** (an open paragraph under the slide that doubles as the long-form image description), WCAG-grade **alt text**, cleaned **on-screen text**, LaTeX **math equations**, and **section titles**. It can also **segment the transcript into topics** across slides.
7. **HTML generation** β an accessible document with proper landmarks, headings, `lang`, skip links, collapsible transcript, and MathJax-rendered equations.
## The editor and outputs
Choosing **Editor** output opens an in-browser editor to refine everything before
publishing. It supports:
- Editing titles, scene descriptions, alt text, transcript, on-screen text, and math.
- **Per-element include toggles** (and document-wide bulk toggles) β choose exactly
what carries into the published guide, per section: scene description, alt-text,
on-screen text, transcript, audio.
- **Save edits (JSON)** β a self-contained file that embeds the slides. Anyone can
reopen it at **`/edit`** and keep editing β no images to ship, no re-run.
- **Download** as a single self-contained **HTML** file, semantic **Markdown**, or a
**ZIP** bundle (HTML + `static/` assets).
## Accessibility
This tool produces a **WCAG 2.2 AA-ready scaffold**, not a certified-compliant
document. Auto-generated descriptions/alt-text are drafts flagged for human review
in the editor. Heading hierarchy, landmarks, `lang`, skip links, and reading order
are generated correctly out of the box.
The intended pattern is the **scene description**: an open paragraph under each
slide that both sighted and screen-reader users read in the page body. When a
description is present, the image's `alt` is emptied (decorative) so the meaning
isn't announced twice. The editor lets you confirm or rewrite each one, and choose
per-section what carries into the published guide. See `DATA_HANDLING.md` for what
data is sent where when the AI features are on.
## Quick start
Requires Python 3.10β3.13 and system packages `ffmpeg` + `tesseract-ocr`.
```bash
pip install -r requirements.txt
uvicorn app.main:app --reload
```
Then open http://localhost:8000.
### Environment variables
| Variable | Default | Description |
|---|---|---|
| `MAX_VIDEO_MB` | `500` | Video upload size cap in MB. Set to `0` to disable. |
| `MAX_TRANSCRIPT_MB` | `5` | Transcript upload size cap in MB. Set to `0` to disable. |
| `MAX_IMAGE_MB` | `25` | Per-image upload cap (images mode). Set to `0` to disable. |
| `MAX_EDIT_MB` | `150` | Cap for a self-contained edits JSON uploaded to `/edit`. |
| `ANTHROPIC_API_KEY` | *(unset)* | Enables LLM enrichment features. Can also be entered per-job in the UI. |
| `ANTHROPIC_BASE_URL` | *(unset)* | Route Anthropic requests to a custom compatible endpoint. Optional; see below. |
| `PARLEY_BASE_URL` | `https://parley.api.mit.edu` | Endpoint used for MIT Parley keys. |
| `ANTHROPIC_MODEL` | `claude-haiku-4-5-20251001` | Model ID for all LLM calls. Set this if your endpoint names models differently β it **must be vision-capable** or slide analysis returns blank/hallucinated descriptions. |
### API keys and MIT Parley
You can paste **either** a normal Anthropic key (`sk-ant-β¦`) **or** an MIT Parley
key (`sk-parley-β¦`) into the API-key field (or set it as `ANTHROPIC_API_KEY`).
Parley keys are automatically routed to the Parley proxy (`PARLEY_BASE_URL`);
Anthropic keys go to Anthropic. No configuration needed β the routing is by key
prefix. To point normal keys at some other Anthropic-compatible proxy, set
`ANTHROPIC_BASE_URL` (this never overrides Parley-key routing).
```bash
# Example: no upload limits, LLM features on
MAX_VIDEO_MB=0 ANTHROPIC_API_KEY=sk-ant-... uvicorn app.main:app --reload
```
## Project layout
```
app/
main.py FastAPI app: / (video), /images, /edit, job queue, preview + asset endpoints
jobs.py In-memory job registry (SSE progress, TTL, quota)
pipeline/
orchestrator.py Pipeline entry point (run_pipeline); video_path optional
transcript.py VTT/SRT parsing (incl. hourless timestamps) + segment alignment
scenes.py Scene detection + frame-accurate extraction
image_input.py Images mode: HH-MM-SS filename parsing β SceneFrame list
filter.py Instructor-frame filtering (Haar + YuNet)
ocr.py Tesseract OCR
llm_extract.py Claude extraction (scene description, alt text, OCR, math, titles, topics)
html_gen.py WCAG HTML rendering (guide, single, review) + metadata round-trip + bundling
metadata.py Shared Pydantic schema (segments, include flags, inlined assets)
templates/
index.html Video upload form + scene picker UI
images.html Images-mode upload page
edit.html Open a shared self-contained edits JSON
guide.html Published study guide template
review.html In-browser editor template
_doc_styles.html Shared CSS for guide and review outputs
tests/
```
## Deploy to Hugging Face Spaces
This repo ships a Dockerfile compatible with [Hugging Face Spaces](https://huggingface.co/spaces) (free CPU tier).
1. Create a new Space β pick **Docker** as the SDK and **Blank** as the template.
2. Push this repo to the Space's git remote:
```bash
git remote add space https://huggingface.co/spaces/<your-user>/<space-name>
git push space <branch>:main
```
3. Spaces reads the YAML frontmatter at the top of this README, builds the Dockerfile, and exposes the app on port `7860`.
Notes:
- Uploads land in `/tmp` inside the container (ephemeral β one-shot processing per visitor).
- The free CPU tier handles a 20-minute lecture in a few minutes.
- The Space sleeps after inactivity and wakes on the next request (~30 s cold start).
- Do **not** set `X-Frame-Options` or CSP headers at the app level β HF Spaces proxies the app through its own infrastructure and those headers will break the Space. Apply them at the reverse-proxy / CDN layer instead.
## Status
Beta. The pipeline runs end-to-end on common inputs. Known gaps:
- A transcript file is required (there's an optional Whisper extra: `pip install -e .[asr]`).
- OCR is English-only by default; Claude-based on-screen text handles other cases better.
- Face detection uses OpenCV Haar cascades + YuNet (fast, good enough for talking heads).
- The `/edit` shared editor has no alt-frame switching or live re-OCR (those need a live job); it edits all text, descriptions, toggles, and math.
|