Spaces:
Running
Running
|
Download README.md from joemartis/Video2Guide: direct link, hf CLI and curl.
- Browser
- Download file 8.54 kB
-
https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/README.md
- Command line
-
hf download hf://spaces/joemartis/Video2Guide/README.md
-
curl -L -o README.md https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/README.md
8.54 kB
| title: The Describatron | |
| emoji: π€ | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # The Describatron | |
| Generate WCAG 2.2 AA-ready HTML study guides from instructional videos β or from | |
| a set of slide images. | |
| Takes a lecture plus a transcript (VTT/SRT) and produces a flowing, accessible | |
| HTML document: visual content shown as images, narration aligned to each segment, | |
| on-screen text captured, instructor talking-head footage filtered out, and | |
| optional Claude-powered enrichment (scene descriptions, alt text, on-screen-text | |
| cleanup, LaTeX math, section titles). | |
| ## Two ways in | |
| - **From a video** (`/`) β upload a video + transcript; the app detects slide | |
| changes, extracts a representative frame per scene, and filters out | |
| talking-head shots. | |
| - **From slide images** (`/images`) β already have slide exports? Upload the | |
| images plus (optionally) a transcript. Name each image with an `HH-MM-SS` | |
| timestamp (e.g. `00-05-30.png`) to anchor it exactly; if the names have no | |
| timestamps, the AI reads the transcript and places each slide where it's | |
| discussed (or, with no API key, spreads them evenly). No transcript at all β | |
| a slides-only guide ordered by filename. No video upload, no scene detection. | |
| Both paths feed the same downstream pipeline and editor. | |
| ## What it does (video path) | |
| 1. **Scene preview** β see extracted frames before committing; deselect any you don't need (duplicates, transitions, stray shots). | |
| 2. **Scene detection** β finds moments where visual content shifts (new slide, diagram, animation step). Frame extraction is deterministic (frame-accurate seeking). | |
| 3. **Instructor filtering** β drops frames that are mostly a talking head (face detection). Tune the threshold in the sidebar. | |
| 4. **On-screen text** β Tesseract OCR, or Claude reads the slide directly. | |
| 5. **Transcript alignment** β maps narration to each visual segment by timestamp. | |
| 6. **LLM enrichment** *(optional, requires Anthropic API key)* β Claude reads each frame and can produce: a **scene description** (an open paragraph under the slide that doubles as the long-form image description), WCAG-grade **alt text**, cleaned **on-screen text**, LaTeX **math equations**, and **section titles**. It can also **segment the transcript into topics** across slides. | |
| 7. **HTML generation** β an accessible document with proper landmarks, headings, `lang`, skip links, collapsible transcript, and MathJax-rendered equations. | |
| ## The editor and outputs | |
| Choosing **Editor** output opens an in-browser editor to refine everything before | |
| publishing. It supports: | |
| - Editing titles, scene descriptions, alt text, transcript, on-screen text, and math. | |
| - **Per-element include toggles** (and document-wide bulk toggles) β choose exactly | |
| what carries into the published guide, per section: scene description, alt-text, | |
| on-screen text, transcript, audio. | |
| - **Save edits (JSON)** β a self-contained file that embeds the slides. Anyone can | |
| reopen it at **`/edit`** and keep editing β no images to ship, no re-run. | |
| - **Download** as a single self-contained **HTML** file, semantic **Markdown**, or a | |
| **ZIP** bundle (HTML + `static/` assets). | |
| ## Accessibility | |
| This tool produces a **WCAG 2.2 AA-ready scaffold**, not a certified-compliant | |
| document. Auto-generated descriptions/alt-text are drafts flagged for human review | |
| in the editor. Heading hierarchy, landmarks, `lang`, skip links, and reading order | |
| are generated correctly out of the box. | |
| The intended pattern is the **scene description**: an open paragraph under each | |
| slide that both sighted and screen-reader users read in the page body. When a | |
| description is present, the image's `alt` is emptied (decorative) so the meaning | |
| isn't announced twice. The editor lets you confirm or rewrite each one, and choose | |
| per-section what carries into the published guide. See `DATA_HANDLING.md` for what | |
| data is sent where when the AI features are on. | |
| ## Quick start | |
| Requires Python 3.10β3.13 and system packages `ffmpeg` + `tesseract-ocr`. | |
| ```bash | |
| pip install -r requirements.txt | |
| uvicorn app.main:app --reload | |
| ``` | |
| Then open http://localhost:8000. | |
| ### Environment variables | |
| | Variable | Default | Description | | |
| |---|---|---| | |
| | `MAX_VIDEO_MB` | `500` | Video upload size cap in MB. Set to `0` to disable. | | |
| | `MAX_TRANSCRIPT_MB` | `5` | Transcript upload size cap in MB. Set to `0` to disable. | | |
| | `MAX_IMAGE_MB` | `25` | Per-image upload cap (images mode). Set to `0` to disable. | | |
| | `MAX_EDIT_MB` | `150` | Cap for a self-contained edits JSON uploaded to `/edit`. | | |
| | `ANTHROPIC_API_KEY` | *(unset)* | Enables LLM enrichment features. Can also be entered per-job in the UI. | | |
| | `ANTHROPIC_BASE_URL` | *(unset)* | Route Anthropic requests to a custom compatible endpoint. Optional; see below. | | |
| | `PARLEY_BASE_URL` | `https://parley.api.mit.edu` | Endpoint used for MIT Parley keys. | | |
| | `ANTHROPIC_MODEL` | `claude-haiku-4-5-20251001` | Model ID for all LLM calls. Set this if your endpoint names models differently β it **must be vision-capable** or slide analysis returns blank/hallucinated descriptions. | | |
| ### API keys and MIT Parley | |
| You can paste **either** a normal Anthropic key (`sk-ant-β¦`) **or** an MIT Parley | |
| key (`sk-parley-β¦`) into the API-key field (or set it as `ANTHROPIC_API_KEY`). | |
| Parley keys are automatically routed to the Parley proxy (`PARLEY_BASE_URL`); | |
| Anthropic keys go to Anthropic. No configuration needed β the routing is by key | |
| prefix. To point normal keys at some other Anthropic-compatible proxy, set | |
| `ANTHROPIC_BASE_URL` (this never overrides Parley-key routing). | |
| ```bash | |
| # Example: no upload limits, LLM features on | |
| MAX_VIDEO_MB=0 ANTHROPIC_API_KEY=sk-ant-... uvicorn app.main:app --reload | |
| ``` | |
| ## Project layout | |
| ``` | |
| app/ | |
| main.py FastAPI app: / (video), /images, /edit, job queue, preview + asset endpoints | |
| jobs.py In-memory job registry (SSE progress, TTL, quota) | |
| pipeline/ | |
| orchestrator.py Pipeline entry point (run_pipeline); video_path optional | |
| transcript.py VTT/SRT parsing (incl. hourless timestamps) + segment alignment | |
| scenes.py Scene detection + frame-accurate extraction | |
| image_input.py Images mode: HH-MM-SS filename parsing β SceneFrame list | |
| filter.py Instructor-frame filtering (Haar + YuNet) | |
| ocr.py Tesseract OCR | |
| llm_extract.py Claude extraction (scene description, alt text, OCR, math, titles, topics) | |
| html_gen.py WCAG HTML rendering (guide, single, review) + metadata round-trip + bundling | |
| metadata.py Shared Pydantic schema (segments, include flags, inlined assets) | |
| templates/ | |
| index.html Video upload form + scene picker UI | |
| images.html Images-mode upload page | |
| edit.html Open a shared self-contained edits JSON | |
| guide.html Published study guide template | |
| review.html In-browser editor template | |
| _doc_styles.html Shared CSS for guide and review outputs | |
| tests/ | |
| ``` | |
| ## Deploy to Hugging Face Spaces | |
| This repo ships a Dockerfile compatible with [Hugging Face Spaces](https://huggingface.co/spaces) (free CPU tier). | |
| 1. Create a new Space β pick **Docker** as the SDK and **Blank** as the template. | |
| 2. Push this repo to the Space's git remote: | |
| ```bash | |
| git remote add space https://huggingface.co/spaces/<your-user>/<space-name> | |
| git push space <branch>:main | |
| ``` | |
| 3. Spaces reads the YAML frontmatter at the top of this README, builds the Dockerfile, and exposes the app on port `7860`. | |
| Notes: | |
| - Uploads land in `/tmp` inside the container (ephemeral β one-shot processing per visitor). | |
| - The free CPU tier handles a 20-minute lecture in a few minutes. | |
| - The Space sleeps after inactivity and wakes on the next request (~30 s cold start). | |
| - Do **not** set `X-Frame-Options` or CSP headers at the app level β HF Spaces proxies the app through its own infrastructure and those headers will break the Space. Apply them at the reverse-proxy / CDN layer instead. | |
| ## Status | |
| Beta. The pipeline runs end-to-end on common inputs. Known gaps: | |
| - A transcript file is required (there's an optional Whisper extra: `pip install -e .[asr]`). | |
| - OCR is English-only by default; Claude-based on-screen text handles other cases better. | |
| - Face detection uses OpenCV Haar cascades + YuNet (fast, good enough for talking heads). | |
| - The `/edit` shared editor has no alt-frame switching or live re-OCR (those need a live job); it edits all text, descriptions, toggles, and math. | |