Spaces:
Running
Download PROMPTS.md from joemartis/Video2Guide: direct link, hf CLI and curl.
- Browser
- Download file 12.8 kB
-
https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/PROMPTS.md
- Command line
-
hf download hf://spaces/joemartis/Video2Guide/PROMPTS.md
-
curl -L -o PROMPTS.md https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/PROMPTS.md
LLM prompts
Every prompt The Describatron sends to Claude, reproduced verbatim, with the
context each one runs in. All of them live in
app/pipeline/llm_extract.py β if you change a
prompt there, update this file too.
There are four distinct calls:
- Per-slide extraction β alt text, scene description, on-screen text, math, section title (one call per slide/scene).
- Topic segmentation β splits the transcript into chapters.
- Slide-to-transcript alignment β places un-timestamped slide images on the timeline by content.
Notes that apply throughout:
- The app does not use the structured-output beta. It asks for raw JSON and parses leniently, because the MIT Parley proxy doesn't enforce a response format. Each prompt therefore ends with an explicit "return only JSON" shape directive.
- System prompts are byte-stable within a run, so Anthropic prompt caching
(
cache_control: ephemeral) kicks in after the first call. - Only the enabled features contribute text β a toggle that's off adds nothing to the prompt or the response schema, so no output tokens are spent on it.
1. Per-slide extraction
One call per slide/scene. The system prompt is assembled from a fixed
preamble, one instruction block per enabled feature, a fixed postamble, and a
JSON shape directive (_build_system_prompt). The user message carries the
slide image plus the narration (and optionally Tesseract's OCR).
System prompt
Preamble (_SYSTEM_PREAMBLE), always present:
You analyze single frames from instructional video lectures and return structured JSON.
For each frame you receive together with the spoken narration that accompanies it, extract:
Feature blocks (_FIELD_PROMPTS) β only enabled features are included, and
they're numbered 1., 2., β¦ in this order:
section_title
section_title β a short, descriptive title for this segment (β€8 words). Capture the main idea, not the slide chrome. Sentence case, no trailing punctuation.
description (the scene description / long-form image description)
description β a detailed visual description of the slide, written as one or two open paragraphs rendered below the figure in the study guide.
- Doubles as the long-form image description: BOTH sighted and non-sighted readers will encounter it as body text, so write it to be useful to either.
- Open with what's visually shown ("A gallery of purple flower stem macro photographs is shown."), then explain what the visual contributes to the lecturer's point ("These demonstrate different perspectives and processing techniques and illustrate how visual experimentation leads to new clarity.").
- Plain prose. No markdown, no bullet lists, no leading "Image of" / "Slide showing". 2β4 sentences per paragraph.
- When the slide carries displayed math or on-screen text essential to the argument, include the meaning of it inline ("β¦and the equation (E = mc^2) is displayed."). Never invent content not in the frame or narration.
- If the frame is purely an instructor talking-head, say so plainly in one short sentence.
alt_text
alt_text β a WCAG-grade alt-text draft for the frame.
- Convey the PURPOSE of the visual in the lecture's argument, not a transcription of slide text. Bad: "Slide containing 'Combining AI models'." Good: "Title slide introducing the section on combining specialized AI models."
- One or two sentences, β€200 characters. Plain prose, no markdown, no leading "Image of"/"Slide showing".
- If the frame is purely an instructor talking-head with no slide content, say so plainly.
- Never invent details that aren't visible in the frame or supported by the narration.
ocr_text (Claude reads the slide directly, instead of Tesseract)
ocr_text β a verbatim transcription of all text VISIBLE ON THE SLIDE.
- Include headings, bullets, callouts, captions, axis labels, and equations exactly as they appear. Use plain text (no markdown).
- Preserve line breaks the slide actually uses; one blank line between blocks.
- Skip purely decorative elements (logos, page numbers).
- Render math expressions as LaTeX inline (e.g. "(x^2 + y^2)"), not as ASCII approximations.
- Empty string if the frame has no on-screen text.
math_equations
math_equations β equations VISIBLY DISPLAYED on the slide.
- latex: a clean LaTeX expression suitable for MathJax (no $ delimiters; no \begin{equation}).
- aria_label: a plain-English screen-reader description of the equation.
- Skip equations that are only spoken, not on screen, or simple inline labels.
- Empty list if the slide has no displayed equation.
Postamble (_SYSTEM_POSTAMBLE), always present:
Return ONLY the structured fields. Do not invent content that isn't on the slide or in the narration.
JSON shape directive β appended, listing exactly the enabled keys. With all features on it reads:
Respond with ONLY a single JSON object and nothing else β no prose, no explanation, no markdown code fences. Use exactly these keys:
{ "section_title": "<string>", "description": "<string>", "alt_text": "<string>", "ocr_text": "<string>", "math_equations": [{"latex": "<string>", "aria_label": "<string>"}] }Use an empty string for a text field with no content and [] for math_equations when the slide has no displayed equation.
User message
An image block (the slide, base64-encoded JPEG) followed by a text block
(_build_user_content):
Spoken narration during this segment: {narration for the segment, or
(no narration)}OCR text from the slide (may contain errors): {Tesseract OCR, or
(no OCR text)}
The OCR block is omitted entirely when the ocr_text feature is on β Claude
is reading the slide itself, so sending Tesseract's version too would only waste
input tokens.
2. Topic segmentation
One call. Splits the transcript into sequential chapters. The user message is
the transcript formatted with per-cue [mm:ss] timestamps.
System prompt
Base (_TOPIC_SYSTEM_PROMPT):
You segment lecture transcripts into topical sections.
Given a transcript with per-cue timestamps, return 3β15 sequential, non-overlapping topics that together cover the entire duration. Each topic must capture a single coherent subject the lecturer discusses β prefer fewer, meaningful sections over many short ones. Topic boundaries should fall at natural conversational shifts (new subject, transitions like "now let's talk about", a clear pause in flow).
For each topic, provide:
- start_seconds, end_seconds: float timestamps in seconds, matching the transcript timestamps. The first topic must start at the transcript's first cue and the last topic must end at the final cue's end.
- title: β€8 words, sentence case, no trailing punctuation. Describe the topic substantively ("How heat pumps work in cold climates"), not the slide chrome ("Slide 2").
- summary: one-sentence (β€30 words) plain-prose recap of the topic.
Return ONLY the structured topics. Do not invent content; if the transcript is too short or off-topic for clean segmentation, return a single topic covering the full range.
With slide thumbnails β when "segment with images" is on, the base prompt is
extended (_TOPIC_SYSTEM_PROMPT_WITH_IMAGES) and slide thumbnails + their start
timestamps are attached to the user message:
You are also given thumbnails of each kept slide with its start timestamp. Use them to spot visual chapter markers (splash slides, title cards, big layout shifts) that the narration alone would miss, and to confirm boundaries when the spoken transition is subtle. A slide whose title differs sharply from its neighbours is a strong signal of a new topic; minor visual changes within the same topic (e.g. animation build-ups) are not.
IMPORTANT β talking-head frames: some thumbnails show the instructor on camera rather than slide content (a face filling much of the frame, a webcam shot, a podium close-up, an empty stage). These carry no topical information β they are an artifact of cutaways during continuous narration. Treat them as transparent: do NOT start, end, or place a topic boundary on the basis of a talking-head frame, and do NOT let one displace a real slide as the anchor for a topic. The only time a talking-head frame should influence segmentation is if it is unambiguously the entire visual content of a stretch with no slide ever shown (e.g. a pure Q&A section); in that case it can be its own topic. Otherwise, set boundaries from the surrounding slide frames and the narration.
Genre hint β an optional block prepended to the system prompt based on the
chosen video type (_TOPIC_VIDEO_TYPE_HINTS). general adds nothing; the others:
lecture
This is a classroom lecture. Lectures typically follow: introduction / agenda β background or motivation β main concept(s) with sub-points β worked examples β summary or take-aways β Q&A or wrap-up. Prefer topic boundaries that match these conventional sections; merge adjacent slides that elaborate on the same concept rather than splitting on every new example.
lab
This is a lab walkthrough or experiment procedure. Lab videos typically follow: introduction / purpose β background or theory β materials and equipment β step-by-step procedure β observations or measurements β analysis / discussion β conclusion. Treat the procedure as one topic unless the lecturer explicitly marks a major phase change ('now we move to data collection'); individual procedural steps are not separate topics.
tutorial
This is a how-to tutorial or screencast. Tutorials typically follow: introduction / goal β prerequisites or setup β step-by-step instructions (often the bulk of the runtime) β verification of the expected result β troubleshooting or extensions β wrap-up. Group consecutive steps that build a single feature into one topic; break at major phases (setup vs. main flow vs. testing).
interview
This is an interview, panel, or Q&A. Topic boundaries usually coincide with a new question from the host or a clear pivot in subject matter; the answer that follows belongs to its question.
talk
This is a talk or presentation (conference, keynote, TED-style). Talks typically follow: hook / opening story β thesis or central question β 2β4 supporting points each with evidence β call to action or memorable closer. Topic boundaries should fall on the major argument transitions, not on every supporting anecdote.
JSON shape directive (_TOPIC_JSON_DIRECTIVE), appended:
Respond with ONLY a single JSON object and nothing else β no prose, no markdown code fences. Shape:
{ "topics": [ {"start_seconds": <float>, "end_seconds": <float>, "title": "<string>", "summary": "<string>"} ] }
3. Slide-to-transcript alignment
One call. Used in images-input mode when a transcript is present but the slide
filenames carry no HH-MM-SS timestamps. Each slide is OCR'd locally
(Tesseract) and this call decides where in the narration each slide begins β so
it's text-only, no images, which means it works through any
Anthropic-compatible endpoint including MIT Parley. On any failure the caller
falls back to spreading the slides evenly across the transcript.
System prompt
You align lecture slides to a transcript. You are given a transcript with per-cue [mm:ss] timestamps, and a list of N slides IN THE ORDER they were shown, each with the text that appears on it. For each slide, determine the time (in SECONDS from the start) at which the lecturer begins discussing that slide β match the slide's text to the narration that introduces it. The slides advance in order, so the start times MUST be non-decreasing. The first slide starts at the transcript's first cue.
Respond with ONLY a JSON object and nothing else β no prose, no markdown fences:
{"slide_starts": [<number>, ...]}with EXACTLY {N} numbers, one per slide, in slide order.
User message
Transcript (mm:ss timestamps are cue starts):
{transcript, one line per cue:
[mm:ss] text}Slides, in the order shown:
Slide 1: {OCR text of slide 1, or
(no visible text)}Slide 2: {OCR text of slide 2}
β¦
The model's returned start times are then clamped to the transcript span, forced non-decreasing, and the first slide is pinned to the first cue before they're used β so a stray or out-of-order value can't produce an inverted segment.