DocuMaker: video to step-by-step DOCX guide (Whisper + HF LLM + BLIP)

.env.example

@@ -0,0 +1,35 @@
+# Copy this file to ".env" and adjust as needed. All values are optional —
+# DocuMaker falls back to sensible defaults. HF_TOKEN is also read from your
+# environment (HF_TOKEN / HUGGINGFACEHUB_API_TOKEN) if not set here.
+
+# HuggingFace token — https://huggingface.co/settings/tokens
+# NOTE: the Gradio app takes your token from its UI field, NOT from here.
+# This is only used by the headless smoke test (scripts/smoke_test.py) / CLI.
+HF_TOKEN=
+
+# --- Text LLM (HuggingFace Inference API) ---
+# Any instruct model available via HF Inference Providers. Swap if a model is
+# unavailable on the free tier. Good options: Qwen/Qwen2.5-7B-Instruct,
+# meta-llama/Llama-3.1-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.3
+DOCUMAKER_LLM_MODEL=Qwen/Qwen2.5-7B-Instruct
+# DOCUMAKER_LLM_PROVIDER=          # e.g. hf-inference, together, novita (blank = auto)
+DOCUMAKER_LLM_MAX_TOKENS=2048
+DOCUMAKER_LLM_TEMPERATURE=0.3
+
+# --- Vision LLM (HuggingFace Inference API) ---
+DOCUMAKER_VLM_MODEL=Qwen/Qwen2-VL-7B-Instruct
+DOCUMAKER_ENABLE_VISION=1          # set 0 to skip captioning entirely
+# DOCUMAKER_VLM_PROVIDER=
+# DOCUMAKER_LOCAL_CAPTION_MODEL=Salesforce/blip-image-captioning-base
+
+# --- Whisper (local faster-whisper) ---
+DOCUMAKER_WHISPER_MODEL=small      # tiny | base | small | medium | large-v3
+DOCUMAKER_WHISPER_DEVICE=auto      # auto | cuda | cpu
+# DOCUMAKER_WHISPER_COMPUTE_TYPE=  # blank = auto; e.g. int8_float16, int8, float16
+
+# --- Frame extraction ---
+DOCUMAKER_SCENE_THRESHOLD=27.0     # lower = more scenes detected
+DOCUMAKER_DEDUP_HASH_DISTANCE=6    # higher = more aggressive dedup
+
+# --- Output ---
+DOCUMAKER_DOCX_IMAGE_WIDTH_INCHES=5.5

.gitignore

@@ -0,0 +1,20 @@
+# Runtime artifacts (uploads, frames, audio, generated docs)
+work/
+
+# Secrets
+.env
+
+# Virtual environments
+.venv/
+venv/
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+
+# Gradio cache
+.gradio/
+
+# HuggingFace / model caches living in-project (if any)
+.cache/

README.md

@@ -0,0 +1,201 @@
+---
+title: DocuMaker
+emoji: 🎬
+colorFrom: indigo
+colorTo: purple
+sdk: gradio
+sdk_version: 6.18.0
+app_file: app.py
+python_version: "3.11"
+pinned: false
+license: mit
+short_description: Turn a tutorial video into a step-by-step DOCX guide
+---
+
+# DocuMaker
+
+Turn a tutorial/screencast **video** into a polished **step-by-step `.docx` guide**
+with screenshots — using open-source tools and **free** HuggingFace models.
+
+> The block above is HuggingFace Spaces config. On GitHub it renders as a small
+> table; on Spaces it tells the platform how to run the app.
+
+Pipeline: `video → preview → frames (manual + automatic) → audio transcription
+(Whisper) → LLM cleanup & step-structuring → image/step alignment + captions →
+DOCX export`.
+
+- **UI:** a local Gradio app.
+- **Transcription:** runs **locally** with [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
+  (GPU with automatic CPU fallback).
+- **Guide writing:** the **HuggingFace Inference API** (uses your HF token).
+- **Image captions:** a vision model. It tries an API vision-chat model first, then
+  falls back to a **local BLIP** captioner — which is the path most free HF accounts
+  end up using, since few enabled providers serve a vision model on the free tier.
+- **Frames:** one-click manual snapshots **and** automatic scene-detection
+  (PySceneDetect) de-duplicated with perceptual hashing.
+
+### How a frame is chosen for each step
+
+Relevancy is decided by combining accurate signals (no single model "judges" it):
+
+1. **Timestamp alignment** — the frame on screen while that step was narrated
+   (Whisper timestamps ↔ step time). The strongest signal for tutorials.
+2. **Sharpness** — variance-of-Laplacian, to avoid blurry scene-transition frames.
+3. **BLIP caption match** — the BLIP caption is compared to the step's text; a frame
+   whose description overlaps the step gets a nudge. This *suggests*, it doesn't decide.
+4. **Manual preference** — frames you snapshot yourself win ties.
+
+See `_pick_frame` in [src/guide.py](src/guide.py).
+
+## Requirements
+
+- Python 3.11
+- **ffmpeg** on your `PATH` (`ffmpeg -version` should work)
+- A HuggingFace token — get one at <https://huggingface.co/settings/tokens>
+  (a free **Read** token works). You **paste it into the app's UI**; it is not read
+  from the environment. (The headless smoke test reads it from `HF_TOKEN` /
+  `HUGGINGFACEHUB_API_TOKEN` instead.)
+- A CUDA GPU is optional (Whisper falls back to CPU automatically)
+
+## Setup
+
+```bash
+python -m venv .venv
+# Windows (PowerShell):  .venv\Scripts\Activate.ps1
+# Git Bash:              source .venv/Scripts/activate
+pip install -r requirements.txt
+
+cp .env.example .env   # optional — tweak model ids / Whisper size
+```
+
+`requirements.txt` includes the local **BLIP** captioner (`torch` + `transformers`,
+CPU build). The BLIP model (~1 GB) downloads on first use and is cached. Captions
+are nice-to-have: without a captioner you still get images + step text, and frame
+selection uses timestamp + sharpness only.
+
+To run BLIP on a local **NVIDIA GPU**, reinstall a CUDA build of torch — it's used
+automatically when available:
+
+```bash
+pip install torch --index-url https://download.pytorch.org/whl/cu124
+```
+
+### HuggingFace token
+
+The **app takes your token from the UI** — paste it into the "🔑 HuggingFace token"
+box at the top of the page. When you enter it (local single-user mode), the app
+sets it as the process `HF_TOKEN` (in memory only, never written to disk) so every
+HuggingFace operation — the guide LLM and model downloads — authenticates with it,
+overriding any stale token already in your environment.
+
+**Shared / multi-user deployments:** if you launch with `DOCUMAKER_SHARE=1` or a
+non-localhost `DOCUMAKER_SERVER_NAME`, the app switches to per-session tokens and
+**does not** touch the global `HF_TOKEN` — so one user's token can never leak to
+another. The token is still threaded directly to that user's LLM/caption calls (the
+guide LLM clients are created per request, never cached across sessions).
+
+The **headless smoke test** (`scripts/smoke_test.py`) instead reads the token from
+`HF_TOKEN` / `HUGGINGFACEHUB_API_TOKEN` and validates which one actually
+authenticates (handy if one is stale).
+
+## Run
+
+```bash
+python app.py
+```
+
+Open the printed local URL, then:
+
+1. **Paste your HuggingFace token** into the 🔑 box at the top.
+2. **Upload & preview** a video.
+3. **Capture frames** — scrub the seek bar and click *📸 Capture current frame*,
+   and/or click *✨ Auto-extract frames* for scene-based snapshots.
+4. **Transcribe audio** (Whisper, local). Edit the transcript if you like.
+5. **Generate step-by-step guide** (HF LLM) and review the steps.
+6. **Build DOCX** — images are matched to steps and captioned, then download `guide.docx`.
+
+## Quick backend check (no UI)
+
+```bash
+python scripts/make_sample.py   # synthesizes work/sample/sample.mp4 (4 scenes + narration)
+python scripts/smoke_test.py    # runs the full pipeline and asserts a valid DOCX
+```
+
+The smoke test falls back to a naive draft if the HF API is unreachable, so DOCX
+assembly is always exercised.
+
+## Deploy to HuggingFace Spaces
+
+This repo is Spaces-ready: the YAML header in this README, `packages.txt` (installs
+`ffmpeg`), and a single `requirements.txt` are all the platform needs. The app
+auto-detects Spaces (via `SPACE_ID`), binds to `0.0.0.0`, and switches to
+**multi-user-safe tokens** — each visitor pastes their own HF token in the UI and it
+never touches the shared environment.
+
+1. Create a new **Gradio** Space at <https://huggingface.co/new-space>.
+2. Push this repo to it:
+   ```bash
+   git init && git add -A && git commit -m "DocuMaker"
+   git remote add space https://huggingface.co/spaces/<your-username>/<space-name>
+   git push space main
+   ```
+   (Or drag the files into the Space's *Files* tab.)
+3. The Space builds and starts automatically. Visitors paste their **own** HF token —
+   you don't expose yours or share its quota.
+
+Notes for the free CPU tier: transcription and BLIP run on CPU (slower but fine for a
+demo); set `DOCUMAKER_WHISPER_MODEL=base` in the Space *Settings → Variables* for
+snappier transcription. You can push to **both** GitHub and the Space (add both as
+git remotes).
+
+## Configuration
+
+All settings are environment variables (see [.env.example](.env.example)). Highlights:
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `DOCUMAKER_LLM_MODEL` | `Qwen/Qwen2.5-7B-Instruct` | Text LLM (any HF instruct model) |
+| `DOCUMAKER_VLM_MODEL` | `Qwen/Qwen2-VL-7B-Instruct` | API vision model tried before local BLIP |
+| `DOCUMAKER_LOCAL_CAPTION_MODEL` | `Salesforce/blip-image-captioning-base` | Local captioner |
+| `DOCUMAKER_ENABLE_VISION` | `1` | Set `0` to skip captioning |
+| `DOCUMAKER_WHISPER_MODEL` | `small` | `tiny`…`large-v3` |
+| `DOCUMAKER_WHISPER_DEVICE` | `auto` | `auto` / `cuda` / `cpu` |
+| `DOCUMAKER_SCENE_THRESHOLD` | `27.0` | Lower = more auto-frames |
+| `DOCUMAKER_SHARE` | `0` | `1` = Gradio public share link (enables multi-user-safe tokens) |
+| `DOCUMAKER_SERVER_NAME` | `127.0.0.1` | Bind address; non-localhost enables multi-user-safe tokens |
+
+## Troubleshooting
+
+- **Whisper CUDA errors / cuDNN not found:** faster-whisper (CTranslate2) needs the
+  NVIDIA CUDA libraries. Either install them —
+  `pip install nvidia-cublas-cu12 nvidia-cudnn-cu12` — or force CPU with
+  `DOCUMAKER_WHISPER_DEVICE=cpu` (slower but always works).
+- **LLM call failed / model not available:** free-tier model availability changes.
+  Set `DOCUMAKER_LLM_MODEL` to another available instruct model, or pin a provider
+  with `DOCUMAKER_LLM_PROVIDER`.
+- **No captions in the DOCX:** the API vision model usually isn't served on free HF
+  accounts ("not supported by any provider you have enabled"), so DocuMaker uses
+  local BLIP (installed via `requirements.txt`). Frame *selection* and images still
+  work without it.
+- **Video doesn't preview:** the app serves files from `work/` via Gradio's
+  `allowed_paths` (URL prefix `/gradio_api/file=`). Make sure the upload completed;
+  very large files take a moment.
+- **No images in the DOCX:** capture or auto-extract frames before *Build DOCX*.
+  Steps with a timestamp but no nearby frame pull a fresh frame from the video.
+
+## Project layout
+
+```
+app.py                 Gradio UI + event wiring
+src/config.py          env-driven settings
+src/video.py           ffmpeg audio extract / duration / frame@timestamp
+src/frames.py          scene detection, dedup, manual-capture decode
+src/transcribe.py      faster-whisper (CUDA→CPU fallback)
+src/llm.py             HF Inference: transcript → structured step JSON
+src/vision.py          VLM captioning (HF API) + local BLIP fallback
+src/guide.py           align frames↔steps, caption
+src/docx_export.py     python-docx assembly
+src/web/player.html    custom HTML5 player for seek + snapshot
+scripts/make_sample.py synthesize a test clip
+scripts/smoke_test.py  headless end-to-end check
+```

app.py

@@ -0,0 +1,360 @@
+"""DocuMaker — Gradio app: video -> frames + transcript -> LLM guide -> DOCX.
+
+Run with:  python app.py
+Then open the printed local URL in your browser.
+"""
+from __future__ import annotations
+
+import os
+import re
+import shutil
+import uuid
+from dataclasses import asdict
+from pathlib import Path
+
+import gradio as gr
+
+from src import config
+from src import docx_export
+from src import guide as guide_lib
+from src import llm
+from src import transcribe as transcribe_lib
+from src import video
+from src.frames import FrameRecord, extract_auto_frames, save_manual_frame
+from src.transcribe import Transcript, TranscriptSegment
+
+# --- Static assets -----------------------------------------------------------
+PLAYER_TEMPLATE = (Path(__file__).parent / "src" / "web" / "player.html").read_text(
+    encoding="utf-8"
+)
+
+# JS run in the browser when the user clicks "Capture current frame". It draws
+# the *currently displayed* video frame onto a canvas and returns the PNG data
+# URL + the playback time. The 4 returned values replace the 4 wired inputs
+# (session and frames pass through unchanged; the last two carry the capture).
+CAPTURE_JS = """
+(session, frames, _url, _time) => {
+  const v = document.getElementById('dm-video');
+  if (!v || !v.videoWidth) { return [session, frames, '', 0]; }
+  const c = document.createElement('canvas');
+  c.width = v.videoWidth;
+  c.height = v.videoHeight;
+  c.getContext('2d').drawImage(v, 0, 0, c.width, c.height);
+  let url = '';
+  try { url = c.toDataURL('image/png'); } catch (e) { url = ''; }
+  return [session, frames, url, v.currentTime];
+}
+"""
+
+
+# --- Small helpers -----------------------------------------------------------
+def _fmt_ts(seconds: float | int | None) -> str:
+    s = int(seconds or 0)
+    return f"{s // 60:02d}:{s % 60:02d}"
+
+
+def _player_html(file_path_posix: str) -> str:
+    # The template builds the Gradio file URL (with a legacy-prefix fallback).
+    return PLAYER_TEMPLATE.replace("__VIDEO_PATH__", file_path_posix)
+
+
+def _gallery_value(frames: list[dict]) -> list[tuple[str, str]]:
+    return [
+        (f["path"], f"{f.get('source', '')} @ {_fmt_ts(f.get('timestamp', 0))}")
+        for f in frames
+    ]
+
+
+def _parse_timestamped_text(text: str) -> Transcript:
+    """Re-parse the (possibly user-edited) '[mm:ss] text' transcript box."""
+    segments: list[TranscriptSegment] = []
+    for line in text.splitlines():
+        m = re.match(r"\s*\[(\d{1,2}):(\d{2})\]\s*(.*)", line)
+        if m:
+            mm, ss, body = m.groups()
+            start = float(int(mm) * 60 + int(ss))
+            segments.append(TranscriptSegment(start=start, end=start, text=body))
+        elif line.strip():
+            if segments:
+                segments[-1].text += " " + line.strip()
+            else:
+                segments.append(TranscriptSegment(0.0, 0.0, line.strip()))
+    return Transcript(segments=segments)
+
+
+def _draft_to_md(draft: llm.GuideDraft) -> str:
+    lines = [f"## {draft.title}", "", draft.intro or "", ""]
+    if draft.prerequisites:
+        lines.append("**Prerequisites**")
+        lines += [f"- {p}" for p in draft.prerequisites]
+        lines.append("")
+    for i, step in enumerate(draft.steps, start=1):
+        ts = ""
+        if step.approx_timestamp is not None:
+            ts = f"  _(~{_fmt_ts(step.approx_timestamp)})_"
+        lines.append(f"**Step {i}: {step.heading}**{ts}")
+        lines.append(step.text)
+        lines.append("")
+    return "\n".join(lines)
+
+
+# --- Event handlers ----------------------------------------------------------
+def init_session():
+    sid = uuid.uuid4().hex[:12]
+    config.session_dir(sid)
+    return sid, [], "", None, None, "Session ready — upload a video to begin."
+
+
+def on_upload(file_path: str | None, session: str):
+    if not file_path:
+        return gr.update(), "", "No file received."
+    sdir = config.session_dir(session)
+    dest = sdir / f"source{Path(file_path).suffix or '.mp4'}"
+    shutil.copyfile(file_path, dest)
+    duration = video.get_duration(dest)
+    html = _player_html(dest.as_posix())
+    return (
+        html,
+        str(dest),
+        f"Loaded video ({duration:.1f}s). Seek + capture frames, or auto-extract.",
+    )
+
+
+def on_capture(session: str, frames: list[dict], data_url: str, current_time: float):
+    rec = save_manual_frame(data_url, current_time or 0.0, config.session_dir(session))
+    if rec is None:
+        return _gallery_value(frames), frames, "Capture failed — let the video load, then retry."
+    frames = frames + [asdict(rec)]
+    return (
+        _gallery_value(frames),
+        frames,
+        f"Captured frame at {_fmt_ts(rec.timestamp)} ({len(frames)} total).",
+    )
+
+
+def on_auto(session: str, frames: list[dict], video_path: str, progress=gr.Progress()):
+    if not video_path:
+        return _gallery_value(frames), frames, "Upload a video first."
+    progress(0.1, "Detecting scenes…")
+    recs = extract_auto_frames(video_path, config.session_dir(session))
+    merged = frames + [asdict(r) for r in recs]
+    progress(1.0, "Done.")
+    return (
+        _gallery_value(merged),
+        merged,
+        f"Auto-extracted {len(recs)} frames ({len(merged)} total).",
+    )
+
+
+def on_clear():
+    return [], [], "Cleared all frames."
+
+
+def on_transcribe(session: str, video_path: str, progress=gr.Progress()):
+    if not video_path:
+        return "", None, "Upload a video first."
+    sdir = config.session_dir(session)
+    progress(0.05, "Extracting audio…")
+    wav = video.extract_audio(video_path, sdir / "audio.wav")
+    progress(0.1, "Loading Whisper…")
+    tr = transcribe_lib.transcribe(wav, progress=progress)
+    return (
+        tr.to_timestamped_text(),
+        tr,
+        f"Transcribed {len(tr.segments)} segments "
+        f"(lang={tr.language or '?'}, device={tr.device or 'cpu'}).",
+    )
+
+
+def on_token_set(hf_token: str):
+    """Mirror the UI token into the HF_TOKEN environment variable."""
+    token = config.apply_token(hf_token)
+    if token:
+        return "🔑 HuggingFace token set for this session."
+    return "Enter your HuggingFace token to generate the guide."
+
+
+def on_generate(transcript_text: str, transcript_obj, hf_token: str, progress=gr.Progress()):
+    token = config.apply_token(hf_token)
+    if not token:
+        return "", None, "⚠️ Enter your HuggingFace token above to generate the guide."
+    tr = _parse_timestamped_text(transcript_text) if transcript_text.strip() else transcript_obj
+    if tr is None or not tr.segments:
+        return "", None, "Transcribe the audio first (or paste a transcript)."
+    try:
+        draft = llm.build_guide_draft(tr, token=token, progress=progress)
+    except RuntimeError as exc:
+        return "", None, f"⚠️ {exc}"
+    if not draft.steps:
+        return "", None, "The LLM returned no steps — try a different DOCUMAKER_LLM_MODEL."
+    return _draft_to_md(draft), draft, f"Drafted {len(draft.steps)} steps. Review, then build the DOCX."
+
+
+def on_build(
+    session: str,
+    draft,
+    frames: list[dict],
+    video_path: str,
+    do_caption: bool,
+    hf_token: str,
+    progress=gr.Progress(),
+):
+    if draft is None or not getattr(draft, "steps", None):
+        return None, "Generate the step-by-step guide first."
+    token = config.apply_token(hf_token)
+    recs = [FrameRecord(**d) for d in frames]
+    progress(0.1, "Matching images to steps…")
+    g = guide_lib.assemble_guide(
+        draft,
+        recs,
+        video_path=video_path or None,
+        session_dir=config.session_dir(session),
+        do_caption=do_caption,
+        token=token,
+        progress=progress,
+    )
+    out = config.session_dir(session) / "guide.docx"
+    docx_export.export_docx(g, out)
+    n_imgs = sum(1 for s in g.steps if s.image_path)
+    progress(1.0, "Done.")
+    return str(out), f"Built {out.name}: {len(g.steps)} steps, {n_imgs} images."
+
+
+# --- UI ----------------------------------------------------------------------
+def build_ui() -> gr.Blocks:
+    with gr.Blocks(title="DocuMaker") as demo:
+        gr.Markdown(
+            "# 🎬➜📄 DocuMaker\n"
+            "Turn a tutorial video into a step-by-step **DOCX** guide with screenshots. "
+            "Transcription runs locally (Whisper); the guide text uses a free HuggingFace "
+            "model via your token; image captions use local BLIP."
+        )
+
+        with gr.Accordion("🔑 HuggingFace token (required to generate the guide)", open=True):
+            hf_token = gr.Textbox(
+                label="HuggingFace token",
+                placeholder="hf_…  (paste your token — used only for this session, never stored)",
+                type="password",
+                autofocus=True,
+            )
+            gr.Markdown(
+                "Create a token at "
+                "[huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) "
+                "(a free **Read** token works). It's kept in memory for this session only."
+            )
+
+        session_state = gr.State("")
+        frames_state = gr.State([])
+        video_state = gr.State("")
+        transcript_state = gr.State(None)
+        draft_state = gr.State(None)
+        # Hidden carriers that the capture JS fills in.
+        cap_url = gr.Textbox(visible=False)
+        cap_time = gr.Number(visible=False)
+
+        with gr.Row():
+            with gr.Column(scale=3):
+                gr.Markdown("### 1 · Upload & preview")
+                upload = gr.File(
+                    label="Upload a video",
+                    type="filepath",
+                    file_types=[".mp4", ".mov", ".mkv", ".webm", ".avi", ".m4v", ".mpg", ".mpeg"],
+                )
+                player = gr.HTML()
+                with gr.Row():
+                    capture_btn = gr.Button("📸 Capture current frame", variant="primary")
+                    auto_btn = gr.Button("✨ Auto-extract frames")
+            with gr.Column(scale=2):
+                gr.Markdown("### Captured frames")
+                gallery = gr.Gallery(
+                    label="Frames pool", columns=3, height=320, object_fit="contain", allow_preview=True
+                )
+                clear_btn = gr.Button("🗑️ Clear frames")
+
+        gr.Markdown("### 2 · Transcribe → 3 · Generate guide")
+        with gr.Row():
+            with gr.Column():
+                transcribe_btn = gr.Button("🎙️ Transcribe audio (Whisper)")
+                transcript_box = gr.Textbox(
+                    label="Transcript (editable — '[mm:ss] text' per line)", lines=12
+                )
+            with gr.Column():
+                generate_btn = gr.Button("📝 Generate step-by-step guide (LLM)")
+                guide_md = gr.Markdown()
+
+        gr.Markdown("### 4 · Build the document")
+        with gr.Row():
+            caption_chk = gr.Checkbox(value=config.ENABLE_VISION, label="Caption images with vision model")
+            build_btn = gr.Button("📄 Build DOCX", variant="primary")
+            download = gr.File(label="Download guide.docx")
+
+        status = gr.Markdown("")
+
+        # --- wiring ---
+        demo.load(
+            init_session,
+            outputs=[session_state, frames_state, transcript_box, transcript_state, draft_state, status],
+        )
+        # Mirror the token into HF_TOKEN as soon as it's entered (so even model
+        # downloads during transcription authenticate with it).
+        hf_token.blur(on_token_set, [hf_token], [status])
+        hf_token.submit(on_token_set, [hf_token], [status])
+        upload.change(on_upload, [upload, session_state], [player, video_state, status])
+
+        capture_btn.click(
+            on_capture,
+            inputs=[session_state, frames_state, cap_url, cap_time],
+            outputs=[gallery, frames_state, status],
+            js=CAPTURE_JS,
+        )
+        auto_btn.click(on_auto, [session_state, frames_state, video_state], [gallery, frames_state, status])
+        clear_btn.click(on_clear, None, [gallery, frames_state, status])
+
+        transcribe_btn.click(
+            on_transcribe, [session_state, video_state], [transcript_box, transcript_state, status]
+        )
+        generate_btn.click(
+            on_generate,
+            [transcript_box, transcript_state, hf_token],
+            [guide_md, draft_state, status],
+        )
+        build_btn.click(
+            on_build,
+            [session_state, draft_state, frames_state, video_state, caption_chk, hf_token],
+            [download, status],
+        )
+
+    return demo
+
+
+if __name__ == "__main__":
+    # HuggingFace Spaces sets SPACE_ID and serves the app publicly, so treat it as
+    # multi-user and bind to all interfaces (Spaces expects 0.0.0.0:7860).
+    on_spaces = bool(os.getenv("SPACE_ID"))
+    share = os.getenv("DOCUMAKER_SHARE", "0").lower() in ("1", "true", "yes")
+
+    if on_spaces:
+        server_name = "0.0.0.0"
+        multiuser = True
+    else:
+        server_name = os.getenv("DOCUMAKER_SERVER_NAME", "127.0.0.1")
+        multiuser = share or server_name not in ("127.0.0.1", "localhost", "::1")
+
+    # In shared/multi-user mode keep each user's token in their own session: do
+    # NOT mirror it into the process-global environment.
+    config.set_allow_env_token(not multiuser)
+    if multiuser:
+        print(
+            "DocuMaker: shared/multi-user mode — HF tokens are kept per session "
+            "(HF_TOKEN env is not set)."
+        )
+
+    app = build_ui().queue()
+    app.launch(
+        theme=gr.themes.Soft(),
+        allowed_paths=[str(config.WORK_DIR)],
+        share=share and not on_spaces,  # Spaces provides its own URL — no tunnel
+        server_name=server_name,
+        show_error=True,
+        inbrowser=not multiuser,
+    )

packages.txt

@@ -0,0 +1 @@
+ffmpeg

requirements.txt

@@ -0,0 +1,31 @@
+# DocuMaker — single requirements file (HuggingFace Spaces installs only this one).
+
+# ---- PyTorch CPU wheels: keeps the Spaces / CPU image small. ----
+# For a local NVIDIA GPU instead, reinstall torch from the CUDA index:
+#   pip install torch --index-url https://download.pytorch.org/whl/cu124
+--extra-index-url https://download.pytorch.org/whl/cpu
+
+# Core app / UI
+gradio>=6.0
+python-dotenv>=1.0
+
+# Transcription (local, CTranslate2 — no torch needed)
+faster-whisper>=1.0.3
+
+# Frame extraction / scene detection / dedup
+# headless OpenCV: no system GUI libs (libGL) required — important on Spaces.
+scenedetect>=0.6.4
+opencv-python-headless>=4.9
+Pillow>=10.2
+imagehash>=4.3
+numpy>=1.26
+
+# HuggingFace Inference API (text LLM) + token validation
+huggingface_hub>=0.27
+
+# Local BLIP image captioner (vision)
+transformers>=4.44
+torch>=2.2
+
+# DOCX output
+python-docx>=1.1

scripts/make_sample.py

@@ -0,0 +1,117 @@
+"""Synthesize a tiny tutorial-style test clip with no external assets.
+
+Produces ``work/sample/sample.mp4``: four solid-color scenes (so scene detection
+has clear cuts) plus spoken narration generated with the built-in Windows SAPI
+voice (so Whisper has real speech to transcribe).
+"""
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+try:  # Windows consoles default to cp1252 and choke on non-ASCII output.
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+except Exception:
+    pass
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from src import config  # noqa: E402
+
+
+def _powershell() -> str:
+    """Locate powershell.exe (it lives in a v1.0\\ subdir not always on PATH)."""
+    found = shutil.which("powershell") or shutil.which("pwsh")
+    if found:
+        return found
+    candidate = Path(os.environ.get("SystemRoot", r"C:\Windows")) / (
+        "System32/WindowsPowerShell/v1.0/powershell.exe"
+    )
+    if candidate.exists():
+        return str(candidate)
+    raise RuntimeError("Could not find powershell.exe to synthesize narration audio.")
+
+NARRATION = (
+    "Welcome to this quick tutorial. "
+    "First, open the application from your desktop. "
+    "Next, click the File menu in the top left corner. "
+    "Then choose the Export option from the list. "
+    "Finally, pick a folder and save your document."
+)
+# Textured, visually distinct patterns so scene detection finds clear cuts and
+# perceptual-hash dedup keeps them (solid colors collapse to one pHash).
+PATTERNS = [
+    "smptebars=size=1280x720",
+    "testsrc2=size=1280x720",
+    "rgbtestsrc=size=1280x720",
+    "mandelbrot=size=1280x720",
+]
+
+
+def _run(cmd: list[str]) -> None:
+    print("+", " ".join(cmd))
+    proc = subprocess.run(cmd, capture_output=True, text=True)
+    if proc.returncode != 0:
+        raise RuntimeError(f"Command failed: {' '.join(cmd)}\n{proc.stderr}")
+
+
+def make_narration(out_wav: Path) -> None:
+    ps = (
+        "Add-Type -AssemblyName System.Speech; "
+        "$s = New-Object System.Speech.Synthesis.SpeechSynthesizer; "
+        f"$s.SetOutputToWaveFile('{out_wav.as_posix()}'); "
+        f"$s.Speak('{NARRATION}'); $s.Dispose();"
+    )
+    _run([_powershell(), "-NoProfile", "-Command", ps])
+
+
+def make_slides(dirpath: Path) -> list[Path]:
+    paths = []
+    for i, pattern in enumerate(PATTERNS):
+        p = dirpath / f"slide_{i}.png"
+        _run(
+            [
+                config.FFMPEG_BIN, "-y",
+                "-f", "lavfi", "-i", pattern,
+                "-frames:v", "1", str(p),
+            ]
+        )
+        paths.append(p)
+    return paths
+
+
+def main() -> Path:
+    out_dir = config.WORK_DIR / "sample"
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    narration = out_dir / "narration.wav"
+    make_narration(narration)
+    slides = make_slides(out_dir)
+
+    listfile = out_dir / "slides.txt"
+    lines: list[str] = []
+    for p in slides:
+        lines.append(f"file '{p.as_posix()}'")
+        lines.append("duration 3")
+    lines.append(f"file '{slides[-1].as_posix()}'")  # concat needs the last file twice
+    listfile.write_text("\n".join(lines), encoding="utf-8")
+
+    out_mp4 = out_dir / "sample.mp4"
+    _run(
+        [
+            config.FFMPEG_BIN, "-y",
+            "-f", "concat", "-safe", "0", "-i", str(listfile),
+            "-i", str(narration),
+            "-c:v", "libx264", "-pix_fmt", "yuv420p",
+            "-c:a", "aac", "-shortest", str(out_mp4),
+        ]
+    )
+    print("Sample video:", out_mp4)
+    return out_mp4
+
+
+if __name__ == "__main__":
+    main()

scripts/smoke_test.py

@@ -0,0 +1,95 @@
+"""Headless end-to-end backend test (no Gradio UI).
+
+Runs: sample video -> audio -> Whisper -> scene frames -> LLM step draft ->
+assemble -> DOCX, asserting each stage produced output. The LLM step falls back
+to a naive sentence-per-step draft if the HuggingFace API is unreachable, so the
+DOCX assembly is always validated.
+"""
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+try:  # Windows consoles default to cp1252 and choke on emoji/non-ASCII output.
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+except Exception:
+    pass
+
+ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(ROOT))
+sys.path.insert(0, str(ROOT / "scripts"))
+
+import make_sample  # noqa: E402
+from src import config, docx_export, video  # noqa: E402
+from src import frames as frames_lib  # noqa: E402
+from src import guide as guide_lib  # noqa: E402
+from src import llm, transcribe  # noqa: E402
+from src.llm import GuideDraft, StepDraft  # noqa: E402
+
+
+def naive_draft(tr) -> GuideDraft:
+    steps = [
+        StepDraft(heading=f"Step {i}", text=seg.text.strip(), approx_timestamp=seg.start)
+        for i, seg in enumerate(tr.segments, start=1)
+        if seg.text.strip()
+    ]
+    return GuideDraft(title="Sample Guide", intro="Generated offline.", steps=steps)
+
+
+def main() -> None:
+    sample = config.WORK_DIR / "sample" / "sample.mp4"
+    if not sample.exists():
+        sample = make_sample.main()
+
+    sdir = config.session_dir("smoke")
+
+    # The app takes the token from the UI; this headless test reads it from the
+    # environment (validating which of HF_TOKEN / HUGGINGFACEHUB_API_TOKEN works).
+    token = config.resolve_hf_token()
+    print(f"HF token: {'present' if token else 'MISSING (LLM step will use naive fallback)'}")
+
+    print("\n[1/5] Extract audio + transcribe…")
+    wav = video.extract_audio(sample, sdir / "audio.wav")
+    tr = transcribe.transcribe(wav)
+    print(f"  device={tr.device} segments={len(tr.segments)} text={tr.text[:120]!r}")
+    assert tr.text.strip(), "Transcript is empty"
+
+    print("[2/5] Auto-extract frames…")
+    recs = frames_lib.extract_auto_frames(sample, sdir)
+    print(f"  frames={len(recs)}")
+    assert recs, "No frames were extracted"
+
+    print("[3/5] Build guide draft (LLM)…")
+    try:
+        draft = llm.build_guide_draft(tr, token=token)
+        if not draft.steps:
+            raise RuntimeError("LLM returned no steps")
+        print(f"  LLM ok: '{draft.title}' ({len(draft.steps)} steps)")
+    except Exception as exc:
+        print(f"  LLM unavailable ({exc}); using naive fallback draft.")
+        draft = naive_draft(tr)
+    assert draft.steps, "No steps in draft"
+
+    print("[4/5] Assemble (align + caption)…")
+    g = guide_lib.assemble_guide(
+        draft, recs, video_path=str(sample), session_dir=sdir, do_caption=True, token=token
+    )
+
+    print("[5/5] Export DOCX…")
+    out = sdir / "guide.docx"
+    docx_export.export_docx(g, out)
+    assert out.exists() and out.stat().st_size > 0, "DOCX not written"
+
+    for s in g.steps:
+        ts = f"{int((s.timestamp or 0))//60:02d}:{int((s.timestamp or 0))%60:02d}"
+        print(f"   - [{ts}] {s.heading!r}: img={'yes' if s.image_path else 'no'} cap={s.caption!r}")
+    n_imgs = sum(1 for s in g.steps if s.image_path)
+    n_caps = sum(1 for s in g.steps if s.caption)
+    print(
+        f"\nOK ✅  {out}  ({out.stat().st_size} bytes, "
+        f"{len(g.steps)} steps, {n_imgs} images, {n_caps} captions)"
+    )
+
+
+if __name__ == "__main__":
+    main()

src/__init__.py

@@ -0,0 +1,3 @@
+"""DocuMaker — turn a tutorial video into a step-by-step DOCX guide."""
+
+__version__ = "0.1.0"

src/config.py

@@ -0,0 +1,142 @@
+"""Central configuration for DocuMaker.
+
+Every tunable is read from environment variables (optionally a local ``.env``
+file), so model ids / devices can be swapped without touching code.
+"""
+from __future__ import annotations
+
+import functools
+import os
+from pathlib import Path
+
+from dotenv import load_dotenv
+
+# Project root = parent of the ``src`` package directory.
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+
+# Load .env from the project root if present (silently ignored if missing).
+load_dotenv(PROJECT_ROOT / ".env")
+
+
+def _flag(name: str, default: str = "0") -> bool:
+    return os.getenv(name, default).strip().lower() not in ("0", "false", "no", "")
+
+
+# --- Paths -------------------------------------------------------------------
+WORK_DIR = Path(os.getenv("DOCUMAKER_WORK_DIR", str(PROJECT_ROOT / "work"))).resolve()
+WORK_DIR.mkdir(parents=True, exist_ok=True)
+
+# --- HuggingFace credentials -------------------------------------------------
+def _token_candidates() -> list[str]:
+    seen: set[str] = set()
+    out: list[str] = []
+    for value in (
+        os.getenv("DOCUMAKER_HF_TOKEN"),
+        os.getenv("HF_TOKEN"),
+        os.getenv("HUGGINGFACEHUB_API_TOKEN"),
+    ):
+        value = (value or "").strip()
+        if value and value not in seen:
+            seen.add(value)
+            out.append(value)
+    return out
+
+
+# Whether a UI token may be mirrored into the process environment. True for the
+# default local single-user app; turned off automatically for shared/multi-user
+# launches so one user's token can't leak to another via the global environment.
+_ALLOW_ENV_TOKEN = True
+
+
+def set_allow_env_token(allowed: bool) -> None:
+    global _ALLOW_ENV_TOKEN
+    _ALLOW_ENV_TOKEN = bool(allowed)
+
+
+def apply_token(token: str | None) -> str | None:
+    """Return the cleaned UI token and, **in single-user mode only**, mirror it
+    into the process ``HF_TOKEN``/``HUGGINGFACEHUB_API_TOKEN`` so huggingface_hub
+    (InferenceClient auto-discovery, model downloads) also uses it.
+
+    In multi-user/shared mode the environment is left untouched — the token is
+    still threaded explicitly to the LLM and captioner, so it stays scoped to the
+    caller's session and nothing leaks across users. Returns None if empty.
+    """
+    token = (token or "").strip()
+    if token and _ALLOW_ENV_TOKEN:
+        os.environ["HF_TOKEN"] = token
+        os.environ["HUGGINGFACEHUB_API_TOKEN"] = token
+    return token or None
+
+
+@functools.lru_cache(maxsize=1)
+def resolve_hf_token() -> str | None:
+    """Return the first *valid* HF token among the configured candidates.
+
+    Environments often have a stale ``HF_TOKEN`` alongside a working
+    ``HUGGINGFACEHUB_API_TOKEN`` (or vice versa). We validate via ``whoami`` and
+    pick the one that authenticates, then point huggingface_hub's own
+    auto-discovery (model downloads, etc.) at the same working token.
+    """
+    candidates = _token_candidates()
+    if not candidates:
+        return None
+
+    chosen = candidates[0]
+    try:
+        from huggingface_hub import whoami
+
+        for token in candidates:
+            try:
+                whoami(token=token)
+                chosen = token
+                break
+            except Exception:
+                continue
+    except Exception:
+        pass  # offline / hub import issue — fall back to the first candidate
+
+    os.environ["HF_TOKEN"] = chosen
+    os.environ["HUGGINGFACEHUB_API_TOKEN"] = chosen
+    return chosen
+
+# --- Text LLM (HF Inference API) --------------------------------------------
+LLM_MODEL = os.getenv("DOCUMAKER_LLM_MODEL", "Qwen/Qwen2.5-7B-Instruct")
+LLM_PROVIDER = os.getenv("DOCUMAKER_LLM_PROVIDER", "").strip() or None
+LLM_MAX_TOKENS = int(os.getenv("DOCUMAKER_LLM_MAX_TOKENS", "2048"))
+LLM_TEMPERATURE = float(os.getenv("DOCUMAKER_LLM_TEMPERATURE", "0.3"))
+# Approx. characters of transcript per LLM chunk (keeps prompts within context).
+LLM_CHUNK_CHARS = int(os.getenv("DOCUMAKER_LLM_CHUNK_CHARS", "6000"))
+
+# --- Vision LLM (HF Inference API) + local fallback -------------------------
+ENABLE_VISION = _flag("DOCUMAKER_ENABLE_VISION", "1")
+VLM_MODEL = os.getenv("DOCUMAKER_VLM_MODEL", "Qwen/Qwen2-VL-7B-Instruct")
+VLM_PROVIDER = os.getenv("DOCUMAKER_VLM_PROVIDER", "").strip() or None
+LOCAL_CAPTION_MODEL = os.getenv(
+    "DOCUMAKER_LOCAL_CAPTION_MODEL", "Salesforce/blip-image-captioning-base"
+)
+
+# --- Whisper (local faster-whisper) -----------------------------------------
+WHISPER_MODEL = os.getenv("DOCUMAKER_WHISPER_MODEL", "small")
+WHISPER_DEVICE = os.getenv("DOCUMAKER_WHISPER_DEVICE", "auto").strip().lower()
+# Blank => choose automatically per device (int8_float16 on CUDA, int8 on CPU).
+WHISPER_COMPUTE_TYPE = os.getenv("DOCUMAKER_WHISPER_COMPUTE_TYPE", "").strip()
+
+# --- Frame extraction --------------------------------------------------------
+SCENE_THRESHOLD = float(os.getenv("DOCUMAKER_SCENE_THRESHOLD", "27.0"))
+SCENE_MIN_LEN_SEC = float(os.getenv("DOCUMAKER_SCENE_MIN_LEN_SEC", "1.0"))
+DEDUP_HASH_DISTANCE = int(os.getenv("DOCUMAKER_DEDUP_HASH_DISTANCE", "6"))
+
+# --- DOCX --------------------------------------------------------------------
+DOCX_IMAGE_WIDTH_INCHES = float(os.getenv("DOCUMAKER_DOCX_IMAGE_WIDTH_INCHES", "5.5"))
+
+# --- External binaries -------------------------------------------------------
+FFMPEG_BIN = os.getenv("DOCUMAKER_FFMPEG_BIN", "ffmpeg")
+FFPROBE_BIN = os.getenv("DOCUMAKER_FFPROBE_BIN", "ffprobe")
+
+
+def session_dir(session_id: str) -> Path:
+    """Return (creating if needed) the working directory for one session."""
+    d = WORK_DIR / session_id
+    (d / "frames").mkdir(parents=True, exist_ok=True)
+    return d

src/docx_export.py

@@ -0,0 +1,55 @@
+"""Render a :class:`~src.guide.Guide` into a Word .docx with images + captions."""
+from __future__ import annotations
+
+from pathlib import Path
+
+from . import config
+from .guide import Guide
+
+
+def export_docx(guide: Guide, out_path: str | Path) -> Path:
+    from docx import Document
+    from docx.enum.text import WD_ALIGN_PARAGRAPH
+    from docx.shared import Inches, Pt
+
+    out_path = Path(out_path)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+
+    doc = Document()
+    doc.add_heading(guide.title or "Step-by-Step Guide", level=0)
+
+    if guide.intro:
+        doc.add_paragraph(guide.intro)
+
+    if guide.prerequisites:
+        doc.add_heading("Prerequisites", level=1)
+        for item in guide.prerequisites:
+            doc.add_paragraph(item, style="List Bullet")
+
+    doc.add_heading("Steps", level=1)
+
+    figure_no = 0
+    for i, step in enumerate(guide.steps, start=1):
+        heading = step.heading.strip() if step.heading else ""
+        doc.add_heading(f"Step {i}: {heading}" if heading else f"Step {i}", level=2)
+
+        if step.text:
+            doc.add_paragraph(step.text)
+
+        if step.image_path and Path(step.image_path).exists():
+            try:
+                doc.add_picture(step.image_path, width=Inches(config.DOCX_IMAGE_WIDTH_INCHES))
+                doc.paragraphs[-1].alignment = WD_ALIGN_PARAGRAPH.CENTER
+            except Exception:
+                pass  # skip an unreadable image rather than fail the export
+            else:
+                if step.caption:
+                    figure_no += 1
+                    caption_par = doc.add_paragraph()
+                    caption_par.alignment = WD_ALIGN_PARAGRAPH.CENTER
+                    run = caption_par.add_run(f"Figure {figure_no}: {step.caption}")
+                    run.italic = True
+                    run.font.size = Pt(9)
+
+    doc.save(str(out_path))
+    return out_path

src/frames.py

@@ -0,0 +1,124 @@
+"""Frame extraction: automatic (scene detection) and manual (browser capture).
+
+A :class:`FrameRecord` is the common unit passed around the app and into the
+guide builder. Automatic frames come from PySceneDetect scene midpoints (with a
+uniform-sampling fallback for single-scene videos) and are de-duplicated with a
+perceptual hash. Manual frames arrive as base64 data URLs from the browser.
+"""
+from __future__ import annotations
+
+import base64
+import binascii
+from dataclasses import dataclass
+from pathlib import Path
+
+import imagehash
+from PIL import Image
+
+from . import config, video
+
+
+@dataclass
+class FrameRecord:
+    """One extracted frame plus where it came from in the video."""
+
+    path: str
+    timestamp: float
+    source: str = "auto"  # "auto" | "manual"
+    caption: str | None = None
+
+    @property
+    def label(self) -> str:
+        mm, ss = divmod(int(self.timestamp), 60)
+        return f"{self.source} @ {mm:02d}:{ss:02d}"
+
+
+def detect_scenes(video_path: str | Path) -> list[tuple[float, float]]:
+    """Return a list of (start_sec, end_sec) scene spans via PySceneDetect."""
+    try:
+        from scenedetect import ContentDetector, detect
+
+        scenes = detect(str(video_path), ContentDetector(threshold=config.SCENE_THRESHOLD))
+    except Exception:
+        # Detection is best-effort; callers fall back to uniform sampling.
+        return []
+    return [(start.get_seconds(), end.get_seconds()) for start, end in scenes]
+
+
+def _scene_timestamps(video_path: str | Path, max_frames: int) -> list[float]:
+    scenes = detect_scenes(video_path)
+    if scenes:
+        timestamps = [(start + end) / 2.0 for start, end in scenes]
+    else:
+        # Single static scene (or detection failed): sample uniformly.
+        duration = video.get_duration(video_path) or 60.0
+        count = min(max_frames, max(3, int(duration // 5)))
+        step = duration / (count + 1)
+        timestamps = [step * (i + 1) for i in range(count)]
+
+    # Keep an evenly spaced subset if we overshoot the cap.
+    if len(timestamps) > max_frames:
+        last = len(timestamps) - 1
+        picks = sorted({round(i * last / (max_frames - 1)) for i in range(max_frames)})
+        timestamps = [timestamps[i] for i in picks]
+    return timestamps
+
+
+def extract_auto_frames(
+    video_path: str | Path, session_dir: str | Path, max_frames: int = 40
+) -> list[FrameRecord]:
+    """Extract one representative frame per detected scene, then dedup."""
+    frames_dir = Path(session_dir) / "frames"
+    frames_dir.mkdir(parents=True, exist_ok=True)
+
+    records: list[FrameRecord] = []
+    for i, ts in enumerate(_scene_timestamps(video_path, max_frames)):
+        out = frames_dir / f"auto_{i:03d}_{int(ts * 1000):08d}.png"
+        try:
+            video.extract_frame(video_path, ts, out)
+        except Exception:
+            continue
+        records.append(FrameRecord(path=str(out), timestamp=ts, source="auto"))
+    return dedup_records(records)
+
+
+def dedup_records(
+    records: list[FrameRecord], distance: int | None = None
+) -> list[FrameRecord]:
+    """Drop near-identical frames using perceptual hashing (pHash)."""
+    distance = config.DEDUP_HASH_DISTANCE if distance is None else distance
+    kept: list[FrameRecord] = []
+    hashes: list[imagehash.ImageHash] = []
+    for rec in records:
+        try:
+            with Image.open(rec.path) as im:
+                phash = imagehash.phash(im)
+        except Exception:
+            continue
+        if any((phash - kept_hash) <= distance for kept_hash in hashes):
+            Path(rec.path).unlink(missing_ok=True)  # remove the redundant file
+            continue
+        hashes.append(phash)
+        kept.append(rec)
+    return kept
+
+
+def save_manual_frame(
+    data_url: str, timestamp: float, session_dir: str | Path
+) -> FrameRecord | None:
+    """Decode a base64 image data URL captured in the browser and save it."""
+    if not data_url:
+        return None
+    frames_dir = Path(session_dir) / "frames"
+    frames_dir.mkdir(parents=True, exist_ok=True)
+
+    b64 = data_url.split(",", 1)[1] if "," in data_url else data_url
+    try:
+        raw = base64.b64decode(b64)
+    except (binascii.Error, ValueError):
+        return None
+
+    ts = float(timestamp or 0.0)
+    out = frames_dir / f"manual_{int(ts * 1000):08d}.png"
+    out.write_bytes(raw)
+    return FrameRecord(path=str(out), timestamp=ts, source="manual")

src/guide.py

@@ -0,0 +1,205 @@
+"""Assemble a final guide: match a frame to each step, then caption it.
+
+Takes the LLM's :class:`~src.llm.GuideDraft` (pure text) plus the pool of
+extracted :class:`~src.frames.FrameRecord` s and produces a :class:`Guide` where
+each step carries the best-matching image and a caption. If a step has a
+timestamp but no nearby frame, a fresh frame is pulled from the video so every
+step can be illustrated.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Callable
+
+from . import config, video, vision
+from .frames import FrameRecord
+from .llm import GuideDraft
+
+
+@dataclass
+class GuideStep:
+    heading: str
+    text: str
+    timestamp: float | None = None
+    image_path: str | None = None
+    caption: str | None = None
+
+
+@dataclass
+class Guide:
+    title: str = "Step-by-Step Guide"
+    intro: str = ""
+    prerequisites: list[str] = field(default_factory=list)
+    steps: list[GuideStep] = field(default_factory=list)
+
+
+# Relevance weights. Timestamp proximity is the most reliable signal for
+# tutorials; the BLIP-caption match and sharpness refine the choice, break ties,
+# and carry the decision when a step has no timestamp.
+_W_PROX, _W_SEM, _W_SHARP, _MANUAL_BONUS = 0.55, 0.30, 0.15, 0.25
+
+_STOPWORDS = {
+    "the", "a", "an", "to", "of", "and", "or", "in", "on", "at", "for", "with",
+    "your", "you", "is", "are", "be", "this", "that", "it", "from", "by", "as",
+    "into", "then", "will", "can", "should", "have", "has", "its", "their", "our",
+    "up", "out", "off", "over", "we", "i",
+}
+
+_SHARPNESS_CACHE: dict[str, float] = {}
+
+
+def _sharpness(path: str) -> float:
+    if path not in _SHARPNESS_CACHE:
+        _SHARPNESS_CACHE[path] = vision.frame_score(path)
+    return _SHARPNESS_CACHE[path]
+
+
+def _keywords(text: str | None) -> set[str]:
+    return {
+        w
+        for w in re.findall(r"[a-z0-9]+", (text or "").lower())
+        if len(w) > 2 and w not in _STOPWORDS
+    }
+
+
+def _text_relevance(caption: str | None, step_text: str) -> float:
+    """Fraction of the step's keywords that BLIP's caption mentions (0..1).
+
+    This is the BLIP *suggestion* signal: it nudges selection toward a frame
+    whose description overlaps the step, without letting BLIP decide alone.
+    """
+    step_kw = _keywords(step_text)
+    cap_kw = _keywords(caption)
+    if not step_kw or not cap_kw:
+        return 0.0
+    return min(len(step_kw & cap_kw) / len(step_kw), 1.0)
+
+
+def _pick_frame(
+    candidates: list[FrameRecord],
+    timestamp: float | None,
+    step_text: str,
+    used: set[str],
+    window: float = 30.0,
+) -> FrameRecord | None:
+    """Pick the most relevant frame for a step by combining accurate signals:
+    timestamp proximity + sharpness + BLIP-caption semantic match + manual bonus.
+    """
+    pool = [f for f in candidates if f.path not in used] or candidates
+    if not pool:
+        return None
+
+    if timestamp is None:
+        within = pool
+    else:
+        within = [f for f in pool if abs(f.timestamp - timestamp) <= window] or pool
+
+    sharps = [_sharpness(f.path) for f in within]
+    smin, smax = min(sharps), max(sharps)
+
+    def norm_sharp(value: float) -> float:
+        return (value - smin) / (smax - smin) if smax > smin else 1.0
+
+    def score(frame: FrameRecord) -> float:
+        if timestamp is None:
+            prox = 0.0
+        else:
+            prox = 1.0 - min(abs(frame.timestamp - timestamp) / window, 1.0)
+        sem = _text_relevance(frame.caption, step_text)
+        sharp = norm_sharp(_sharpness(frame.path))
+        total = _W_PROX * prox + _W_SEM * sem + _W_SHARP * sharp
+        if frame.source == "manual":
+            total += _MANUAL_BONUS
+        return total
+
+    return max(within, key=score)
+
+
+def assemble_guide(
+    draft: GuideDraft,
+    frames: list[FrameRecord],
+    *,
+    video_path: str | Path | None = None,
+    session_dir: str | Path | None = None,
+    do_caption: bool = True,
+    token: str | None = None,
+    progress: Callable[[float, str], None] | None = None,
+) -> Guide:
+    """Combine a guide draft with frames into a fully illustrated :class:`Guide`.
+
+    When captioning is on, the whole (deduped) frame pool is captioned once so
+    BLIP can both *suggest* the most relevant frame per step and supply the
+    figure captions.
+    """
+    frames_sorted = sorted(frames, key=lambda f: f.timestamp)
+
+    # Caption the pool up front (once per frame, context-free to keep the
+    # relevance signal unbiased) so captions feed both selection and figures.
+    if do_caption and frames_sorted:
+        for i, rec in enumerate(frames_sorted):
+            if rec.caption is None:
+                if progress:
+                    progress(
+                        0.05 + 0.45 * (i / len(frames_sorted)),
+                        f"Captioning frame {i + 1}/{len(frames_sorted)}…",
+                    )
+                rec.caption = vision.caption_image(rec.path, token=token) or ""
+
+    used: set[str] = set()
+    steps: list[GuideStep] = []
+    total = max(len(draft.steps), 1)
+
+    for i, sd in enumerate(draft.steps):
+        if progress:
+            progress(0.5 + 0.5 * (i / total), f"Matching image to step {i + 1}/{total}…")
+
+        step_text = f"{sd.heading} {sd.text}".strip()
+        chosen = _pick_frame(frames_sorted, sd.approx_timestamp, step_text, used)
+
+        # No suitable frame nearby — extract one at the step timestamp.
+        if (
+            chosen is None
+            and video_path
+            and session_dir
+            and sd.approx_timestamp is not None
+        ):
+            out = Path(session_dir) / "frames" / f"step_{i:03d}_{int(sd.approx_timestamp * 1000):08d}.png"
+            try:
+                video.extract_frame(video_path, sd.approx_timestamp, out)
+                chosen = FrameRecord(path=str(out), timestamp=sd.approx_timestamp, source="auto")
+            except Exception:
+                chosen = None
+
+        image_path = None
+        caption = None
+        if chosen is not None:
+            used.add(chosen.path)
+            image_path = chosen.path
+            if chosen.caption is None and do_caption:  # freshly extracted frame
+                chosen.caption = (
+                    vision.caption_image(chosen.path, token=token, context=step_text) or ""
+                )
+            caption = chosen.caption or None
+
+        steps.append(
+            GuideStep(
+                heading=sd.heading,
+                text=sd.text,
+                timestamp=sd.approx_timestamp if sd.approx_timestamp is not None
+                else (chosen.timestamp if chosen else None),
+                image_path=image_path,
+                caption=caption,
+            )
+        )
+
+    if progress:
+        progress(1.0, "Guide assembled.")
+
+    return Guide(
+        title=draft.title,
+        intro=draft.intro,
+        prerequisites=draft.prerequisites,
+        steps=steps,
+    )

src/llm.py

@@ -0,0 +1,273 @@
+"""Transcript cleanup + step-structuring via the HuggingFace Inference API.
+
+The LLM turns a rough, timestamped transcript into a structured guide draft
+(title, intro, prerequisites, ordered steps). Long transcripts are processed
+map-reduce style so prompts stay within the model's context window. All model
+ids are config-driven and responses are parsed defensively, because free-tier
+model availability and exact output formatting both vary.
+"""
+from __future__ import annotations
+
+import json
+import re
+from dataclasses import dataclass, field
+from typing import Any, Callable
+
+from . import config
+from .transcribe import Transcript
+
+_SYSTEM = (
+    "You are a meticulous technical writer. You convert rough spoken transcripts "
+    "from how-to/tutorial videos into clear, accurate, step-by-step instructions. "
+    "You never invent actions that are not in the transcript."
+)
+
+
+@dataclass
+class StepDraft:
+    heading: str
+    text: str
+    approx_timestamp: float | None = None
+
+
+@dataclass
+class GuideDraft:
+    title: str = "Step-by-Step Guide"
+    intro: str = ""
+    prerequisites: list[str] = field(default_factory=list)
+    steps: list[StepDraft] = field(default_factory=list)
+
+
+# --- Inference client --------------------------------------------------------
+def get_client(token: str | None):
+    """Build an InferenceClient for the given user-supplied token.
+
+    Creating a client is cheap (no network until a call), so we don't cache —
+    this lets the token change at runtime (it comes from the UI field).
+    """
+    from huggingface_hub import InferenceClient
+
+    kwargs: dict[str, Any] = {"model": config.LLM_MODEL}
+    if token:
+        kwargs["token"] = token
+    if config.LLM_PROVIDER:
+        kwargs["provider"] = config.LLM_PROVIDER
+    return InferenceClient(**kwargs)
+
+
+def _chat(client, user_prompt: str, *, max_tokens: int | None = None) -> str:
+    try:
+        resp = client.chat_completion(
+            messages=[
+                {"role": "system", "content": _SYSTEM},
+                {"role": "user", "content": user_prompt},
+            ],
+            max_tokens=max_tokens or config.LLM_MAX_TOKENS,
+            temperature=config.LLM_TEMPERATURE,
+        )
+    except Exception as exc:
+        raise RuntimeError(
+            f"HuggingFace LLM call failed for model '{config.LLM_MODEL}'. "
+            f"Check the model is available on your plan or set DOCUMAKER_LLM_MODEL "
+            f"to another instruct model.\nDetails: {exc}"
+        ) from exc
+    return resp.choices[0].message.content or ""
+
+
+# --- JSON / time parsing -----------------------------------------------------
+def _extract_json(text: str) -> Any:
+    """Best-effort extraction of a JSON object/array from an LLM response."""
+    text = text.strip()
+    try:
+        return json.loads(text)
+    except Exception:
+        pass
+
+    fenced = re.search(r"```(?:json)?\s*(.*?)```", text, re.S)
+    if fenced:
+        try:
+            return json.loads(fenced.group(1))
+        except Exception:
+            pass
+
+    for open_ch, close_ch in (("{", "}"), ("[", "]")):
+        i, j = text.find(open_ch), text.rfind(close_ch)
+        if i != -1 and j > i:
+            try:
+                return json.loads(text[i : j + 1])
+            except Exception:
+                continue
+    return None
+
+
+def _parse_time(value: Any) -> float | None:
+    """Parse 'ss', 'mm:ss', or 'hh:mm:ss' (or a number) into seconds."""
+    if value is None:
+        return None
+    if isinstance(value, (int, float)):
+        return float(value)
+    s = str(value).strip()
+    if not s:
+        return None
+    try:
+        parts = [float(p) for p in s.split(":")]
+    except ValueError:
+        return None
+    seconds = 0.0
+    for part in parts:
+        seconds = seconds * 60 + part
+    return seconds
+
+
+# --- Prompt builders ---------------------------------------------------------
+_JSON_FULL = (
+    '{"title": "...", "intro": "...", "prerequisites": ["..."], '
+    '"steps": [{"heading": "short title", "text": "what to do", "approx_time": "mm:ss"}]}'
+)
+_JSON_STEPS = '{"steps": [{"heading": "short title", "text": "what to do", "approx_time": "mm:ss"}]}'
+
+
+def _full_prompt(timestamped_text: str) -> str:
+    return (
+        "Convert this timestamped tutorial transcript into a clean step-by-step guide.\n"
+        "- Fix obvious speech-to-text errors; remove filler words and repetition.\n"
+        "- Write each step as a clear, imperative instruction.\n"
+        "- Add a short descriptive title, a 1-2 sentence introduction, and any "
+        "prerequisites that are implied (empty list if none).\n"
+        "- For each step set \"approx_time\" as \"mm:ss\" from the nearest [mm:ss] marker.\n"
+        f"Respond with ONLY JSON in this exact shape:\n{_JSON_FULL}\n\n"
+        f"Transcript:\n{timestamped_text}"
+    )
+
+
+def _chunk_prompt(timestamped_text: str) -> str:
+    return (
+        "From this timestamped transcript excerpt of a tutorial video, extract the "
+        "concrete actions as an ordered list of steps.\n"
+        "- Fix speech-to-text errors; remove filler and repetition.\n"
+        "- Write each step as a clear, imperative instruction.\n"
+        "- Set \"approx_time\" as \"mm:ss\" from the nearest [mm:ss] marker.\n"
+        f"Respond with ONLY JSON in this exact shape:\n{_JSON_STEPS}\n\n"
+        f"Transcript:\n{timestamped_text}"
+    )
+
+
+def _reduce_prompt(steps_json: str) -> str:
+    return (
+        "You are assembling the final step-by-step guide from steps extracted across "
+        "several transcript chunks.\n"
+        "- Merge near-duplicates and keep a logical order.\n"
+        "- Keep every distinct action; do not invent new steps.\n"
+        "- Add a short descriptive title, a 1-2 sentence introduction, and prerequisites "
+        "if implied (empty list if none). Preserve each step's \"approx_time\".\n"
+        f"Respond with ONLY JSON in this exact shape:\n{_JSON_FULL}\n\n"
+        f"Extracted steps:\n{steps_json}"
+    )
+
+
+# --- Chunking ----------------------------------------------------------------
+def _timestamped_lines(transcript: Transcript) -> list[str]:
+    lines = []
+    for seg in transcript.segments:
+        mm, ss = divmod(int(seg.start), 60)
+        lines.append(f"[{mm:02d}:{ss:02d}] {seg.text.strip()}")
+    return lines
+
+
+def _chunk_transcript(transcript: Transcript, max_chars: int) -> list[str]:
+    lines = _timestamped_lines(transcript)
+    if not lines:
+        return [transcript.text] if transcript.text else []
+
+    chunks: list[str] = []
+    current: list[str] = []
+    length = 0
+    for line in lines:
+        if current and length + len(line) > max_chars:
+            chunks.append("\n".join(current))
+            current, length = [], 0
+        current.append(line)
+        length += len(line) + 1
+    if current:
+        chunks.append("\n".join(current))
+    return chunks
+
+
+# --- Result parsing ----------------------------------------------------------
+def _parse_steps(raw_steps: Any) -> list[StepDraft]:
+    steps: list[StepDraft] = []
+    if not isinstance(raw_steps, list):
+        return steps
+    for item in raw_steps:
+        if not isinstance(item, dict):
+            continue
+        heading = str(item.get("heading") or item.get("title") or "").strip()
+        text = str(item.get("text") or item.get("instruction") or "").strip()
+        if not (heading or text):
+            continue
+        ts = _parse_time(item.get("approx_time", item.get("approx_timestamp")))
+        steps.append(StepDraft(heading=heading or text[:60], text=text, approx_timestamp=ts))
+    return steps
+
+
+def _parse_guide(data: Any) -> GuideDraft:
+    if not isinstance(data, dict):
+        return GuideDraft()
+    prereqs = data.get("prerequisites") or []
+    if not isinstance(prereqs, list):
+        prereqs = [str(prereqs)]
+    return GuideDraft(
+        title=str(data.get("title") or "Step-by-Step Guide").strip(),
+        intro=str(data.get("intro") or "").strip(),
+        prerequisites=[str(p).strip() for p in prereqs if str(p).strip()],
+        steps=_parse_steps(data.get("steps")),
+    )
+
+
+# --- Public entry point ------------------------------------------------------
+def build_guide_draft(
+    transcript: Transcript,
+    *,
+    token: str | None = None,
+    progress: Callable[[float, str], None] | None = None,
+) -> GuideDraft:
+    """Turn a transcript into a structured :class:`GuideDraft` via the LLM.
+
+    ``token`` is the user's HuggingFace token (supplied in the UI).
+    """
+    chunks = _chunk_transcript(transcript, config.LLM_CHUNK_CHARS)
+    if not chunks:
+        return GuideDraft()
+
+    client = get_client(token)
+
+    if len(chunks) == 1:
+        if progress:
+            progress(0.1, "Writing the guide…")
+        draft = _parse_guide(_extract_json(_chat(client, _full_prompt(chunks[0]))))
+        if progress:
+            progress(1.0, "Guide drafted.")
+        return draft
+
+    # Map: extract steps per chunk.
+    all_steps: list[dict] = []
+    for i, chunk in enumerate(chunks):
+        if progress:
+            progress(i / (len(chunks) + 1), f"Structuring part {i + 1}/{len(chunks)}…")
+        data = _extract_json(_chat(client, _chunk_prompt(chunk)))
+        for step in _parse_steps((data or {}).get("steps") if isinstance(data, dict) else data):
+            mm, ss = divmod(int(step.approx_timestamp or 0), 60)
+            all_steps.append(
+                {"heading": step.heading, "text": step.text, "approx_time": f"{mm:02d}:{ss:02d}"}
+            )
+
+    # Reduce: merge into a titled guide.
+    if progress:
+        progress(len(chunks) / (len(chunks) + 1), "Assembling the final guide…")
+    reduced = _extract_json(_chat(client, _reduce_prompt(json.dumps({"steps": all_steps}))))
+    draft = _parse_guide(reduced)
+    if not draft.steps:  # reduce failed — fall back to the mapped steps
+        draft = _parse_guide({"steps": all_steps})
+    if progress:
+        progress(1.0, "Guide drafted.")
+    return draft

src/transcribe.py

@@ -0,0 +1,137 @@
+"""Local speech-to-text via faster-whisper (CTranslate2).
+
+The model is loaded lazily and cached. On a CUDA box we prefer
+``int8_float16``; if CUDA is unavailable or its libraries are missing we fall
+back to CPU ``int8`` automatically, so transcription always works.
+"""
+from __future__ import annotations
+
+from collections.abc import Iterator
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Callable
+
+from . import config
+
+# Cached (model, (device, compute_type)).
+_MODEL = None
+_MODEL_INFO: tuple[str, str] | None = None
+# Set once a CUDA runtime failure is seen, to skip the GPU on later calls.
+_FORCE_CPU = False
+
+
+@dataclass
+class TranscriptSegment:
+    start: float
+    end: float
+    text: str
+
+
+@dataclass
+class Transcript:
+    segments: list[TranscriptSegment] = field(default_factory=list)
+    language: str = ""
+    device: str = ""
+
+    @property
+    def text(self) -> str:
+        return " ".join(seg.text.strip() for seg in self.segments).strip()
+
+    def to_timestamped_text(self) -> str:
+        lines = []
+        for seg in self.segments:
+            mm, ss = divmod(int(seg.start), 60)
+            lines.append(f"[{mm:02d}:{ss:02d}] {seg.text.strip()}")
+        return "\n".join(lines)
+
+
+def _candidate_configs() -> Iterator[tuple[str, str]]:
+    """Yield (device, compute_type) attempts in priority order."""
+    override = config.WHISPER_COMPUTE_TYPE
+    device = config.WHISPER_DEVICE
+    if device == "cpu" or _FORCE_CPU:
+        yield ("cpu", override or "int8")
+        return
+    # "cuda" or "auto": try GPU first, then always fall back to CPU int8.
+    yield ("cuda", override or "int8_float16")
+    yield ("cpu", "int8")
+
+
+def _is_cuda_error(exc: Exception) -> bool:
+    msg = str(exc).lower()
+    return any(tok in msg for tok in ("cublas", "cudnn", "cuda", ".dll", "gpu"))
+
+
+def get_model():
+    """Load (once) and return the faster-whisper model with its resolved config."""
+    global _MODEL, _MODEL_INFO
+    if _MODEL is not None:
+        return _MODEL, _MODEL_INFO
+
+    from faster_whisper import WhisperModel
+
+    errors: list[str] = []
+    for device, compute_type in _candidate_configs():
+        try:
+            _MODEL = WhisperModel(
+                config.WHISPER_MODEL, device=device, compute_type=compute_type
+            )
+            _MODEL_INFO = (device, compute_type)
+            return _MODEL, _MODEL_INFO
+        except Exception as exc:  # CUDA libs missing, OOM, etc.
+            errors.append(f"  {device}/{compute_type}: {exc}")
+
+    raise RuntimeError(
+        "Could not load the Whisper model. Attempts:\n" + "\n".join(errors)
+    )
+
+
+def _run_transcribe(
+    media_path: str | Path, progress: Callable[[float, str], None] | None
+) -> Transcript:
+    model, info = get_model()
+    device = info[0] if info else ""
+
+    segment_iter, meta = model.transcribe(str(media_path), vad_filter=True, beam_size=5)
+    total = float(getattr(meta, "duration", 0.0)) or 0.0
+
+    # The CUDA libraries are loaded lazily on first encode, so a missing-cuBLAS
+    # error surfaces while consuming this generator — not at model construction.
+    segments: list[TranscriptSegment] = []
+    for seg in segment_iter:
+        segments.append(
+            TranscriptSegment(start=float(seg.start), end=float(seg.end), text=seg.text)
+        )
+        if progress and total:
+            frac = min(seg.end / total, 1.0)
+            progress(frac, f"Transcribing… {int(frac * 100)}%")
+
+    return Transcript(
+        segments=segments,
+        language=getattr(meta, "language", "") or "",
+        device=device,
+    )
+
+
+def transcribe(
+    media_path: str | Path,
+    *,
+    progress: Callable[[float, str], None] | None = None,
+) -> Transcript:
+    """Transcribe an audio/video file into timestamped segments.
+
+    Falls back from GPU to CPU automatically if a CUDA runtime error (e.g.
+    missing cuBLAS/cuDNN) occurs during inference.
+    """
+    global _MODEL, _MODEL_INFO, _FORCE_CPU
+    try:
+        return _run_transcribe(media_path, progress)
+    except RuntimeError as exc:
+        on_gpu = bool(_MODEL_INFO and _MODEL_INFO[0] == "cuda")
+        if not (on_gpu and _is_cuda_error(exc)):
+            raise
+        # Drop the GPU model and retry once on CPU.
+        _MODEL, _MODEL_INFO, _FORCE_CPU = None, None, True
+        if progress:
+            progress(0.0, "GPU unavailable — falling back to CPU…")
+        return _run_transcribe(media_path, progress)

src/video.py

@@ -0,0 +1,82 @@
+"""Video helpers: audio extraction, duration probing, single-frame grabbing.
+
+All heavy lifting is delegated to ffmpeg/ffprobe (already on PATH). ffmpeg's
+``-ss`` before ``-i`` is both fast and frame-accurate in modern builds, which we
+rely on for precise frame extraction at a given timestamp.
+"""
+from __future__ import annotations
+
+import subprocess
+from pathlib import Path
+
+from . import config
+
+
+class FFmpegError(RuntimeError):
+    """Raised when an ffmpeg/ffprobe subprocess fails."""
+
+
+def _run(cmd: list[str]) -> subprocess.CompletedProcess:
+    proc = subprocess.run(cmd, capture_output=True, text=True)
+    if proc.returncode != 0:
+        raise FFmpegError(
+            f"Command failed ({proc.returncode}): {' '.join(cmd)}\n{proc.stderr.strip()}"
+        )
+    return proc
+
+
+def get_duration(video_path: str | Path) -> float:
+    """Return the media duration in seconds (0.0 if it cannot be determined)."""
+    try:
+        proc = _run(
+            [
+                config.FFPROBE_BIN,
+                "-v", "error",
+                "-show_entries", "format=duration",
+                "-of", "default=noprint_wrappers=1:nokey=1",
+                str(video_path),
+            ]
+        )
+        return float(proc.stdout.strip())
+    except (FFmpegError, ValueError):
+        return 0.0
+
+
+def extract_audio(video_path: str | Path, out_wav: str | Path) -> Path:
+    """Extract a 16 kHz mono WAV (the format faster-whisper expects)."""
+    out_wav = Path(out_wav)
+    out_wav.parent.mkdir(parents=True, exist_ok=True)
+    _run(
+        [
+            config.FFMPEG_BIN,
+            "-y",
+            "-i", str(video_path),
+            "-vn",            # drop video
+            "-ac", "1",       # mono
+            "-ar", "16000",   # 16 kHz
+            "-f", "wav",
+            str(out_wav),
+        ]
+    )
+    return out_wav
+
+
+def extract_frame(video_path: str | Path, timestamp: float, out_png: str | Path) -> Path:
+    """Save a single frame at ``timestamp`` seconds as a PNG.
+
+    ``-ss`` is placed before ``-i`` for fast, frame-accurate seeking.
+    """
+    out_png = Path(out_png)
+    out_png.parent.mkdir(parents=True, exist_ok=True)
+    _run(
+        [
+            config.FFMPEG_BIN,
+            "-y",
+            "-ss", f"{max(timestamp, 0.0):.3f}",
+            "-i", str(video_path),
+            "-frames:v", "1",
+            "-q:v", "2",
+            str(out_png),
+        ]
+    )
+    return out_png

src/vision.py

@@ -0,0 +1,166 @@
+"""Multimodal pass: caption frames and score them for "informativeness".
+
+Captioning prefers a vision LLM on the HuggingFace Inference API and falls back
+to a local BLIP model (only if torch/transformers are installed). Frame scoring
+uses a cheap sharpness heuristic (variance of the Laplacian) so the guide builder
+can prefer crisp, content-rich frames over blurry scene-transition frames.
+"""
+from __future__ import annotations
+
+import base64
+import io
+from pathlib import Path
+
+from . import config
+
+_LOCAL_PROC = None
+_LOCAL_MODEL = None
+_LOCAL_DEVICE = "cpu"
+_LOCAL_FAILED = False
+# Many free HF accounts have no provider that serves a vision-chat model. Once
+# the API VLM fails, stop retrying it for the session and use local BLIP.
+_API_VLM_DISABLED = False
+
+_CAPTION_PROMPT = (
+    "In one concise sentence, describe what this screenshot from a tutorial shows, "
+    "focusing on the on-screen UI element or the action being performed. "
+    "Do not begin with phrases like 'The image shows'."
+)
+
+
+def _data_uri(image_path: str | Path, max_side: int = 1024) -> str:
+    """Downscale + JPEG-encode an image into a data URI (saves API bandwidth)."""
+    from PIL import Image
+
+    with Image.open(image_path) as im:
+        im = im.convert("RGB")
+        im.thumbnail((max_side, max_side))
+        buf = io.BytesIO()
+        im.save(buf, format="JPEG", quality=85)
+    return "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()
+
+
+def _get_vlm_client(token: str | None):
+    from huggingface_hub import InferenceClient
+
+    kwargs = {"model": config.VLM_MODEL}
+    if token:
+        kwargs["token"] = token
+    if config.VLM_PROVIDER:
+        kwargs["provider"] = config.VLM_PROVIDER
+    return InferenceClient(**kwargs)
+
+
+def _caption_via_api(image_path: str | Path, prompt: str, token: str | None) -> str:
+    client = _get_vlm_client(token)
+    resp = client.chat_completion(
+        messages=[
+            {
+                "role": "user",
+                "content": [
+                    {"type": "text", "text": prompt},
+                    {"type": "image_url", "image_url": {"url": _data_uri(image_path)}},
+                ],
+            }
+        ],
+        max_tokens=120,
+        temperature=0.2,
+    )
+    return (resp.choices[0].message.content or "").strip()
+
+
+def _load_local_captioner() -> None:
+    """Load the BLIP captioner directly (the image-to-text pipeline task was
+    removed in transformers 5). Uses the GPU if a CUDA build of torch is present.
+    """
+    global _LOCAL_PROC, _LOCAL_MODEL, _LOCAL_DEVICE
+    from transformers import AutoProcessor
+
+    try:
+        from transformers import AutoModelForImageTextToText as _AutoCaptionModel
+    except Exception:  # older transformers
+        from transformers import AutoModelForVision2Seq as _AutoCaptionModel
+
+    proc = AutoProcessor.from_pretrained(config.LOCAL_CAPTION_MODEL)
+    model = _AutoCaptionModel.from_pretrained(config.LOCAL_CAPTION_MODEL)
+
+    device = "cpu"
+    try:
+        import torch
+
+        if torch.cuda.is_available():
+            device = "cuda"
+            model = model.to(device)
+    except Exception:
+        pass
+
+    _LOCAL_PROC, _LOCAL_MODEL, _LOCAL_DEVICE = proc, model, device
+
+
+def _caption_via_local(image_path: str | Path) -> str:
+    """Local BLIP captioner. Returns '' if torch/transformers are unavailable."""
+    global _LOCAL_FAILED
+    if _LOCAL_FAILED:
+        return ""
+    if _LOCAL_MODEL is None:
+        try:
+            _load_local_captioner()
+        except Exception:
+            _LOCAL_FAILED = True
+            return ""
+    try:
+        import torch
+        from PIL import Image
+
+        with Image.open(image_path) as im:
+            img = im.convert("RGB")
+        inputs = _LOCAL_PROC(images=img, return_tensors="pt")
+        if _LOCAL_DEVICE != "cpu":
+            inputs = {k: v.to(_LOCAL_DEVICE) for k, v in inputs.items()}
+        with torch.no_grad():
+            out = _LOCAL_MODEL.generate(**inputs, max_new_tokens=40)
+        return _LOCAL_PROC.decode(out[0], skip_special_tokens=True).strip()
+    except Exception:
+        return ""
+
+
+def caption_image(
+    image_path: str | Path, *, token: str | None = None, context: str = ""
+) -> str | None:
+    """Return a one-line caption for a frame, or None if captioning is off/failed.
+
+    With a ``token`` it tries an API vision-chat model first (if any provider
+    serves one), then falls back to local BLIP. After the API VLM fails once it
+    is skipped for the rest of the session to avoid repeated dead calls. Local
+    BLIP needs no token.
+    """
+    global _API_VLM_DISABLED
+    if not config.ENABLE_VISION:
+        return None
+    prompt = _CAPTION_PROMPT
+    if context:
+        prompt += f" For context, this step is about: {context[:200]}"
+
+    if token and not _API_VLM_DISABLED:
+        try:
+            caption = _caption_via_api(image_path, prompt, token)
+            if caption:
+                return caption
+        except Exception:
+            _API_VLM_DISABLED = True  # no usable provider — switch to local BLIP
+
+    caption = _caption_via_local(image_path)
+    return caption or None
+
+
+def frame_score(image_path: str | Path) -> float:
+    """Sharpness score (variance of Laplacian). Higher = crisper/more detailed."""
+    try:
+        import cv2
+
+        img = cv2.imread(str(image_path), cv2.IMREAD_GRAYSCALE)
+        if img is None:
+            return 0.0
+        return float(cv2.Laplacian(img, cv2.CV_64F).var())
+    except Exception:
+        return 0.0

src/web/player.html

@@ -0,0 +1,10 @@
+<div class="dm-player" style="display:flex;flex-direction:column;gap:6px;">
+  <video id="dm-video" controls preload="metadata"
+         src="/gradio_api/file=__VIDEO_PATH__"
+         onerror="if(!this.dataset.fb){this.dataset.fb='1';this.src='/file=__VIDEO_PATH__';}"
+         style="width:100%;max-height:430px;background:#000;border-radius:8px;"></video>
+  <p style="margin:0;font-size:12px;opacity:0.75;">
+    Use the seek bar to find the exact moment, then click
+    <b>"📸 Capture current frame"</b> below to snapshot it.
+  </p>
+</div>