Video2Guide / README.md
Claude
Claude Opus 4.8
Add AI slide-to-transcript alignment for untimestamped images
e646192
|
Raw History Blame Contribute Delete
8.54 kB
metadata
title: The Describatron
emoji: πŸ€–
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false

The Describatron

Generate WCAG 2.2 AA-ready HTML study guides from instructional videos β€” or from a set of slide images.

Takes a lecture plus a transcript (VTT/SRT) and produces a flowing, accessible HTML document: visual content shown as images, narration aligned to each segment, on-screen text captured, instructor talking-head footage filtered out, and optional Claude-powered enrichment (scene descriptions, alt text, on-screen-text cleanup, LaTeX math, section titles).

Two ways in

  • From a video (/) β€” upload a video + transcript; the app detects slide changes, extracts a representative frame per scene, and filters out talking-head shots.
  • From slide images (/images) β€” already have slide exports? Upload the images plus (optionally) a transcript. Name each image with an HH-MM-SS timestamp (e.g. 00-05-30.png) to anchor it exactly; if the names have no timestamps, the AI reads the transcript and places each slide where it's discussed (or, with no API key, spreads them evenly). No transcript at all β†’ a slides-only guide ordered by filename. No video upload, no scene detection.

Both paths feed the same downstream pipeline and editor.

What it does (video path)

  1. Scene preview β€” see extracted frames before committing; deselect any you don't need (duplicates, transitions, stray shots).
  2. Scene detection β€” finds moments where visual content shifts (new slide, diagram, animation step). Frame extraction is deterministic (frame-accurate seeking).
  3. Instructor filtering β€” drops frames that are mostly a talking head (face detection). Tune the threshold in the sidebar.
  4. On-screen text β€” Tesseract OCR, or Claude reads the slide directly.
  5. Transcript alignment β€” maps narration to each visual segment by timestamp.
  6. LLM enrichment (optional, requires Anthropic API key) β€” Claude reads each frame and can produce: a scene description (an open paragraph under the slide that doubles as the long-form image description), WCAG-grade alt text, cleaned on-screen text, LaTeX math equations, and section titles. It can also segment the transcript into topics across slides.
  7. HTML generation β€” an accessible document with proper landmarks, headings, lang, skip links, collapsible transcript, and MathJax-rendered equations.

The editor and outputs

Choosing Editor output opens an in-browser editor to refine everything before publishing. It supports:

  • Editing titles, scene descriptions, alt text, transcript, on-screen text, and math.
  • Per-element include toggles (and document-wide bulk toggles) β€” choose exactly what carries into the published guide, per section: scene description, alt-text, on-screen text, transcript, audio.
  • Save edits (JSON) β€” a self-contained file that embeds the slides. Anyone can reopen it at /edit and keep editing β€” no images to ship, no re-run.
  • Download as a single self-contained HTML file, semantic Markdown, or a ZIP bundle (HTML + static/ assets).

Accessibility

This tool produces a WCAG 2.2 AA-ready scaffold, not a certified-compliant document. Auto-generated descriptions/alt-text are drafts flagged for human review in the editor. Heading hierarchy, landmarks, lang, skip links, and reading order are generated correctly out of the box.

The intended pattern is the scene description: an open paragraph under each slide that both sighted and screen-reader users read in the page body. When a description is present, the image's alt is emptied (decorative) so the meaning isn't announced twice. The editor lets you confirm or rewrite each one, and choose per-section what carries into the published guide. See DATA_HANDLING.md for what data is sent where when the AI features are on.

Quick start

Requires Python 3.10–3.13 and system packages ffmpeg + tesseract-ocr.

pip install -r requirements.txt
uvicorn app.main:app --reload

Then open http://localhost:8000.

Environment variables

Variable Default Description
MAX_VIDEO_MB 500 Video upload size cap in MB. Set to 0 to disable.
MAX_TRANSCRIPT_MB 5 Transcript upload size cap in MB. Set to 0 to disable.
MAX_IMAGE_MB 25 Per-image upload cap (images mode). Set to 0 to disable.
MAX_EDIT_MB 150 Cap for a self-contained edits JSON uploaded to /edit.
ANTHROPIC_API_KEY (unset) Enables LLM enrichment features. Can also be entered per-job in the UI.
ANTHROPIC_BASE_URL (unset) Route Anthropic requests to a custom compatible endpoint. Optional; see below.
PARLEY_BASE_URL https://parley.api.mit.edu Endpoint used for MIT Parley keys.
ANTHROPIC_MODEL claude-haiku-4-5-20251001 Model ID for all LLM calls. Set this if your endpoint names models differently β€” it must be vision-capable or slide analysis returns blank/hallucinated descriptions.

API keys and MIT Parley

You can paste either a normal Anthropic key (sk-ant-…) or an MIT Parley key (sk-parley-…) into the API-key field (or set it as ANTHROPIC_API_KEY). Parley keys are automatically routed to the Parley proxy (PARLEY_BASE_URL); Anthropic keys go to Anthropic. No configuration needed β€” the routing is by key prefix. To point normal keys at some other Anthropic-compatible proxy, set ANTHROPIC_BASE_URL (this never overrides Parley-key routing).

# Example: no upload limits, LLM features on
MAX_VIDEO_MB=0 ANTHROPIC_API_KEY=sk-ant-... uvicorn app.main:app --reload

Project layout

app/
  main.py              FastAPI app: / (video), /images, /edit, job queue, preview + asset endpoints
  jobs.py              In-memory job registry (SSE progress, TTL, quota)
  pipeline/
    orchestrator.py    Pipeline entry point (run_pipeline); video_path optional
    transcript.py      VTT/SRT parsing (incl. hourless timestamps) + segment alignment
    scenes.py          Scene detection + frame-accurate extraction
    image_input.py     Images mode: HH-MM-SS filename parsing β†’ SceneFrame list
    filter.py          Instructor-frame filtering (Haar + YuNet)
    ocr.py             Tesseract OCR
    llm_extract.py     Claude extraction (scene description, alt text, OCR, math, titles, topics)
    html_gen.py        WCAG HTML rendering (guide, single, review) + metadata round-trip + bundling
    metadata.py        Shared Pydantic schema (segments, include flags, inlined assets)
  templates/
    index.html         Video upload form + scene picker UI
    images.html        Images-mode upload page
    edit.html          Open a shared self-contained edits JSON
    guide.html         Published study guide template
    review.html        In-browser editor template
    _doc_styles.html   Shared CSS for guide and review outputs
tests/

Deploy to Hugging Face Spaces

This repo ships a Dockerfile compatible with Hugging Face Spaces (free CPU tier).

  1. Create a new Space β€” pick Docker as the SDK and Blank as the template.
  2. Push this repo to the Space's git remote:
    git remote add space https://huggingface.co/spaces/<your-user>/<space-name>
    git push space <branch>:main
    
  3. Spaces reads the YAML frontmatter at the top of this README, builds the Dockerfile, and exposes the app on port 7860.

Notes:

  • Uploads land in /tmp inside the container (ephemeral β€” one-shot processing per visitor).
  • The free CPU tier handles a 20-minute lecture in a few minutes.
  • The Space sleeps after inactivity and wakes on the next request (~30 s cold start).
  • Do not set X-Frame-Options or CSP headers at the app level β€” HF Spaces proxies the app through its own infrastructure and those headers will break the Space. Apply them at the reverse-proxy / CDN layer instead.

Status

Beta. The pipeline runs end-to-end on common inputs. Known gaps:

  • A transcript file is required (there's an optional Whisper extra: pip install -e .[asr]).
  • OCR is English-only by default; Claude-based on-screen text handles other cases better.
  • Face detection uses OpenCV Haar cascades + YuNet (fast, good enough for talking heads).
  • The /edit shared editor has no alt-frame switching or live re-OCR (those need a live job); it edits all text, descriptions, toggles, and math.