Spaces:
Running
Running
|
Download PROMPTS.md from joemartis/Video2Guide: direct link, hf CLI and curl.
- Browser
- Download file 12.8 kB
-
https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/PROMPTS.md
- Command line
-
hf download hf://spaces/joemartis/Video2Guide/PROMPTS.md
-
curl -L -o PROMPTS.md https://huggingface.co/spaces/joemartis/Video2Guide/resolve/main/PROMPTS.md
12.8 kB
| # LLM prompts | |
| Every prompt The Describatron sends to Claude, reproduced verbatim, with the | |
| context each one runs in. All of them live in | |
| [`app/pipeline/llm_extract.py`](app/pipeline/llm_extract.py) β if you change a | |
| prompt there, update this file too. | |
| There are **four** distinct calls: | |
| 1. [Per-slide extraction](#1-per-slide-extraction) β alt text, scene | |
| description, on-screen text, math, section title (one call per slide/scene). | |
| 2. [Topic segmentation](#2-topic-segmentation) β splits the transcript into | |
| chapters. | |
| 3. [Slide-to-transcript alignment](#3-slide-to-transcript-alignment) β places | |
| un-timestamped slide images on the timeline by content. | |
| Notes that apply throughout: | |
| - The app does **not** use the structured-output beta. It asks for raw JSON and | |
| parses leniently, because the MIT Parley proxy doesn't enforce a response | |
| format. Each prompt therefore ends with an explicit "return only JSON" shape | |
| directive. | |
| - System prompts are byte-stable within a run, so Anthropic **prompt caching** | |
| (`cache_control: ephemeral`) kicks in after the first call. | |
| - Only the enabled features contribute text β a toggle that's off adds nothing | |
| to the prompt or the response schema, so no output tokens are spent on it. | |
| --- | |
| ## 1. Per-slide extraction | |
| One call per slide/scene. The **system prompt** is assembled from a fixed | |
| preamble, one instruction block per enabled feature, a fixed postamble, and a | |
| JSON shape directive (`_build_system_prompt`). The **user message** carries the | |
| slide image plus the narration (and optionally Tesseract's OCR). | |
| ### System prompt | |
| **Preamble** (`_SYSTEM_PREAMBLE`), always present: | |
| > You analyze single frames from instructional video lectures and return structured JSON. | |
| > | |
| > For each frame you receive together with the spoken narration that accompanies it, extract: | |
| **Feature blocks** (`_FIELD_PROMPTS`) β only enabled features are included, and | |
| they're numbered `1.`, `2.`, β¦ in this order: | |
| **section_title** | |
| > section_title β a short, descriptive title for this segment (β€8 words). Capture the main idea, not the slide chrome. Sentence case, no trailing punctuation. | |
| **description** (the scene description / long-form image description) | |
| > description β a detailed visual description of the slide, written as one or two open paragraphs rendered below the figure in the study guide. | |
| > - Doubles as the long-form image description: BOTH sighted and non-sighted readers will encounter it as body text, so write it to be useful to either. | |
| > - Open with what's visually shown ("A gallery of purple flower stem macro photographs is shown."), then explain what the visual contributes to the lecturer's point ("These demonstrate different perspectives and processing techniques and illustrate how visual experimentation leads to new clarity."). | |
| > - Plain prose. No markdown, no bullet lists, no leading "Image of" / "Slide showing". 2β4 sentences per paragraph. | |
| > - When the slide carries displayed math or on-screen text essential to the argument, include the meaning of it inline ("β¦and the equation \(E = mc^2\) is displayed."). Never invent content not in the frame or narration. | |
| > - If the frame is purely an instructor talking-head, say so plainly in one short sentence. | |
| **alt_text** | |
| > alt_text β a WCAG-grade alt-text draft for the frame. | |
| > - Convey the PURPOSE of the visual in the lecture's argument, not a transcription of slide text. Bad: "Slide containing 'Combining AI models'." Good: "Title slide introducing the section on combining specialized AI models." | |
| > - One or two sentences, β€200 characters. Plain prose, no markdown, no leading "Image of"/"Slide showing". | |
| > - If the frame is purely an instructor talking-head with no slide content, say so plainly. | |
| > - Never invent details that aren't visible in the frame or supported by the narration. | |
| **ocr_text** (Claude reads the slide directly, instead of Tesseract) | |
| > ocr_text β a verbatim transcription of all text VISIBLE ON THE SLIDE. | |
| > - Include headings, bullets, callouts, captions, axis labels, and equations exactly as they appear. Use plain text (no markdown). | |
| > - Preserve line breaks the slide actually uses; one blank line between blocks. | |
| > - Skip purely decorative elements (logos, page numbers). | |
| > - Render math expressions as LaTeX inline (e.g. "\(x^2 + y^2\)"), not as ASCII approximations. | |
| > - Empty string if the frame has no on-screen text. | |
| **math_equations** | |
| > math_equations β equations VISIBLY DISPLAYED on the slide. | |
| > - latex: a clean LaTeX expression suitable for MathJax (no $ delimiters; no \begin{equation}). | |
| > - aria_label: a plain-English screen-reader description of the equation. | |
| > - Skip equations that are only spoken, not on screen, or simple inline labels. | |
| > - Empty list if the slide has no displayed equation. | |
| **Postamble** (`_SYSTEM_POSTAMBLE`), always present: | |
| > Return ONLY the structured fields. Do not invent content that isn't on the slide or in the narration. | |
| **JSON shape directive** β appended, listing exactly the enabled keys. With all | |
| features on it reads: | |
| > Respond with ONLY a single JSON object and nothing else β no prose, no explanation, no markdown code fences. Use exactly these keys: | |
| > ```json | |
| > { | |
| > "section_title": "<string>", | |
| > "description": "<string>", | |
| > "alt_text": "<string>", | |
| > "ocr_text": "<string>", | |
| > "math_equations": [{"latex": "<string>", "aria_label": "<string>"}] | |
| > } | |
| > ``` | |
| > Use an empty string for a text field with no content and [] for math_equations when the slide has no displayed equation. | |
| ### User message | |
| An image block (the slide, base64-encoded JPEG) followed by a text block | |
| (`_build_user_content`): | |
| > Spoken narration during this segment: | |
| > {narration for the segment, or `(no narration)`} | |
| > | |
| > OCR text from the slide (may contain errors): | |
| > {Tesseract OCR, or `(no OCR text)`} | |
| The **OCR block is omitted entirely** when the `ocr_text` feature is on β Claude | |
| is reading the slide itself, so sending Tesseract's version too would only waste | |
| input tokens. | |
| --- | |
| ## 2. Topic segmentation | |
| One call. Splits the transcript into sequential chapters. The user message is | |
| the transcript formatted with per-cue `[mm:ss]` timestamps. | |
| ### System prompt | |
| Base (`_TOPIC_SYSTEM_PROMPT`): | |
| > You segment lecture transcripts into topical sections. | |
| > | |
| > Given a transcript with per-cue timestamps, return 3β15 sequential, non-overlapping topics that together cover the entire duration. Each topic must capture a single coherent subject the lecturer discusses β prefer fewer, meaningful sections over many short ones. Topic boundaries should fall at natural conversational shifts (new subject, transitions like "now let's talk about", a clear pause in flow). | |
| > | |
| > For each topic, provide: | |
| > - start_seconds, end_seconds: float timestamps in seconds, matching the transcript timestamps. The first topic must start at the transcript's first cue and the last topic must end at the final cue's end. | |
| > - title: β€8 words, sentence case, no trailing punctuation. Describe the topic substantively ("How heat pumps work in cold climates"), not the slide chrome ("Slide 2"). | |
| > - summary: one-sentence (β€30 words) plain-prose recap of the topic. | |
| > | |
| > Return ONLY the structured topics. Do not invent content; if the transcript is too short or off-topic for clean segmentation, return a single topic covering the full range. | |
| **With slide thumbnails** β when "segment with images" is on, the base prompt is | |
| extended (`_TOPIC_SYSTEM_PROMPT_WITH_IMAGES`) and slide thumbnails + their start | |
| timestamps are attached to the user message: | |
| > You are also given thumbnails of each kept slide with its start timestamp. Use them to spot visual chapter markers (splash slides, title cards, big layout shifts) that the narration alone would miss, and to confirm boundaries when the spoken transition is subtle. A slide whose title differs sharply from its neighbours is a strong signal of a new topic; minor visual changes within the same topic (e.g. animation build-ups) are not. | |
| > | |
| > IMPORTANT β talking-head frames: some thumbnails show the instructor on camera rather than slide content (a face filling much of the frame, a webcam shot, a podium close-up, an empty stage). These carry no topical information β they are an artifact of cutaways during continuous narration. Treat them as transparent: do NOT start, end, or place a topic boundary on the basis of a talking-head frame, and do NOT let one displace a real slide as the anchor for a topic. The only time a talking-head frame should influence segmentation is if it is unambiguously the entire visual content of a stretch with no slide ever shown (e.g. a pure Q&A section); in that case it can be its own topic. Otherwise, set boundaries from the surrounding slide frames and the narration. | |
| **Genre hint** β an optional block *prepended* to the system prompt based on the | |
| chosen video type (`_TOPIC_VIDEO_TYPE_HINTS`). `general` adds nothing; the others: | |
| **lecture** | |
| > This is a classroom lecture. Lectures typically follow: | |
| > introduction / agenda β background or motivation β main concept(s) with sub-points β worked examples β summary or take-aways β Q&A or wrap-up. | |
| > Prefer topic boundaries that match these conventional sections; merge adjacent slides that elaborate on the same concept rather than splitting on every new example. | |
| **lab** | |
| > This is a lab walkthrough or experiment procedure. Lab videos typically follow: | |
| > introduction / purpose β background or theory β materials and equipment β step-by-step procedure β observations or measurements β analysis / discussion β conclusion. | |
| > Treat the procedure as one topic unless the lecturer explicitly marks a major phase change ('now we move to data collection'); individual procedural steps are not separate topics. | |
| **tutorial** | |
| > This is a how-to tutorial or screencast. Tutorials typically follow: | |
| > introduction / goal β prerequisites or setup β step-by-step instructions (often the bulk of the runtime) β verification of the expected result β troubleshooting or extensions β wrap-up. | |
| > Group consecutive steps that build a single feature into one topic; break at major phases (setup vs. main flow vs. testing). | |
| **interview** | |
| > This is an interview, panel, or Q&A. Topic boundaries usually coincide with a new question from the host or a clear pivot in subject matter; the answer that follows belongs to its question. | |
| **talk** | |
| > This is a talk or presentation (conference, keynote, TED-style). Talks typically follow: | |
| > hook / opening story β thesis or central question β 2β4 supporting points each with evidence β call to action or memorable closer. | |
| > Topic boundaries should fall on the major argument transitions, not on every supporting anecdote. | |
| **JSON shape directive** (`_TOPIC_JSON_DIRECTIVE`), appended: | |
| > Respond with ONLY a single JSON object and nothing else β no prose, no markdown code fences. Shape: | |
| > ```json | |
| > { | |
| > "topics": [ | |
| > {"start_seconds": <float>, "end_seconds": <float>, "title": "<string>", "summary": "<string>"} | |
| > ] | |
| > } | |
| > ``` | |
| --- | |
| ## 3. Slide-to-transcript alignment | |
| One call. Used in images-input mode when a transcript is present but the slide | |
| filenames carry **no `HH-MM-SS` timestamps**. Each slide is OCR'd locally | |
| (Tesseract) and this call decides where in the narration each slide begins β so | |
| it's **text-only, no images**, which means it works through any | |
| Anthropic-compatible endpoint including MIT Parley. On any failure the caller | |
| falls back to spreading the slides evenly across the transcript. | |
| ### System prompt | |
| > You align lecture slides to a transcript. You are given a transcript with per-cue [mm:ss] timestamps, and a list of N slides IN THE ORDER they were shown, each with the text that appears on it. For each slide, determine the time (in SECONDS from the start) at which the lecturer begins discussing that slide β match the slide's text to the narration that introduces it. The slides advance in order, so the start times MUST be non-decreasing. The first slide starts at the transcript's first cue. | |
| > | |
| > Respond with ONLY a JSON object and nothing else β no prose, no markdown fences: | |
| > `{"slide_starts": [<number>, ...]}` | |
| > with EXACTLY {N} numbers, one per slide, in slide order. | |
| ### User message | |
| > Transcript (mm:ss timestamps are cue starts): | |
| > | |
| > {transcript, one line per cue: `[mm:ss] text`} | |
| > | |
| > Slides, in the order shown: | |
| > | |
| > Slide 1: | |
| > {OCR text of slide 1, or `(no visible text)`} | |
| > | |
| > Slide 2: | |
| > {OCR text of slide 2} | |
| > | |
| > β¦ | |
| The model's returned start times are then **clamped to the transcript span, | |
| forced non-decreasing, and the first slide is pinned to the first cue** before | |
| they're used β so a stray or out-of-order value can't produce an inverted | |
| segment. | |