Spaces:
Sleeping
Sleeping
Commit History
-
Add AI slide-to-transcript alignment for untimestamped images
Claude +1: Claude Opus 4.8When the images-input mode gets a transcript but the slide filenames carry no HH-MM-SS timestamps, place each slide on the timeline by content instead of failing. Locally OCR each slide (Tesseract) and ask Claude when the narration first discusses it, yielding real per-slide start times. Falls back to an even split across the transcript span when there's no API key or alignment fails. - image_input.py: add all_have_timestamps, order_by_name, build_frames_from_starts, build_frames_even_split (clamped, ordered). - llm_extract.py: add align_slides_to_transcript + SlideAlignError (text-based, so it works through Anthropic-compatible proxies). - /jobs/from-images: three-way timeline branch (order-only / timestamp-anchored / AI-aligned) run off the event loop. - Update images.html hint, README to note timestamps are now optional. - Tests for the new helpers and alignment cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01…e646192 -
fix: images-mode ZIP dropped all images; retry malformed LLM JSON
ClaudeTwo issues from a real images-mode run: ZIP bundle had no images (broken guide). The /jobs/{id}/zip endpoint gated asset filenames on the video-mode scene_NNNN.jpg pattern, so images-mode uploads that keep their original names ("slide 1.jpg") were filtered out — never written to the temp dir, never bundled — leaving HTML that references static/ files that aren't there. Replace the rigid pattern with _is_safe_asset_name: a bare basename (no separators / no traversal, enforced by Path(name).name == name) with an allowed image/ audio extension. Keeps the path-traversal protection (that gate was the reason for the regex) while accepting any legitimate filename. Malformed-JSON responses dropped a slide's enrichment. The model occasionally emits invalid JSON (an unescaped quote → "Expecting ',' delimiter"), which failed the parse and left that segment blank. Now: json.loads uses strict=False (tolerates literal control chars in strings), and extract_one retries the call ONCE on a parse/vali…f09a075 -
fix: parse LLM output ourselves so it works through the Parley proxy
ClaudeThe per-segment and topic calls used client.messages.parse(output_format=…), Anthropic's structured-output beta. The MIT Parley proxy doesn't enforce it: the model returns free text — sometimes a ```json-fenced object, sometimes a conversational "I'm ready to analyze…" — and .parse then fails to validate all of it, so every enrichment came back empty. Switch both call sites to plain messages.create and extract the JSON ourselves: * _create_with_retry replaces _parse_with_retry (create, no output_format). * _response_text joins the response's text blocks; _extract_json_object strips a ```json fence and pulls the outermost {...} span, then json.loads. A reply with no JSON raises, and the caller returns empty for that segment (no crash). * The system prompts now spell out the exact JSON keys/shape and demand "only a JSON object, no fences" — nudges non-enforcing proxies onto the rails. * _build_extraction_model fields are now optional (defaults), so a response that omits one …e5c847b -
feat: route MIT Parley keys to the Parley proxy (alongside normal Anthropic keys)
ClaudeSupport both a normal Anthropic key (sk-ant-…) and an MIT Parley key (sk-parley-…) in the same API-key field. A Parley key must hit the Parley proxy rather than api.anthropic.com, so _build_client now resolves the base URL by key prefix: sk-parley → PARLEY_BASE_URL (default https://parley.api.mit.edu, env-overridable), everything else → the SDK default. An explicit ANTHROPIC_BASE_URL still routes non-Parley keys to a custom compatible proxy, but never overrides Parley routing. Detection is by prefix so no server config is needed — the user just pastes whichever key they have and it goes to the right endpoint. Tests cover Parley vs Anthropic routing and the env override. https://claude.ai/code/session_01EM8jiYxdVFy1w3LeJ9GBJg897f74e
-
fix: close the 3 previously-unverified low-severity audit findings
ClaudeThe first bug audit had 3 findings whose verifier agents died on an API session limit, leaving them unconfirmed. Re-verified against current code — all real, all low severity, two fixed here (the third is a recommendation, below): * extract_all built its shared anthropic client OUTSIDE any try, so a missing/broken SDK (the import is lazy precisely so the app runs without the package) would abort the whole job instead of degrading to empty results — contradicting the module's "always returns, pipeline can always proceed" contract that extract_one and extract_topics honor. Guard the shared-client construction: on failure, log and return empty results with progress marked complete. * /process (the legacy non-browser endpoint) lacked the "AI features requested but no API key" guard that /jobs and /jobs/from-images have, so a script POSTing llm_* toggles with no key got a 200 with every AI feature silently dropped. Add the same fail-fast 400. Not changed (recommendation…0658e45 -
fix: flatten RGBA/palette PNGs before JPEG-encoding for the LLM
ClaudeLive test of the images-input mode surfaced "cannot write mode RGBA as JPEG" on every frame: PNG slide exports commonly carry an alpha channel (RGBA) or a palette with transparency, and _encode_image saved straight to JPEG, which has no alpha — so PIL raised and every per-segment LLM call failed, leaving descriptions/alt-text/OCR empty. Flatten onto a white background before encoding (transparent slide regions read as white, matching how a sighted viewer sees them), and convert any other non-RGB/L mode to RGB. Only affects the bytes sent to Claude; the displayed/embedded image still uses the original PNG via the data-URI path, so on-screen quality is unchanged. Tests (runnable without cv2): RGBA and palette-with-transparency PNGs both encode to a valid RGB JPEG instead of raising. https://claude.ai/code/session_01EM8jiYxdVFy1w3LeJ9GBJg13e0c0c
-
editor: remove summary / key-terms / self-check (round 3)
ClaudeAccessibility team's round-three asks: * "Summary Field — redundant, remove" (now subsumed by the new detailed description paragraph from PR 2) * "Key Terms — REMOVE. This replicates the glossary." * "Self-check questions — REMOVE! This is an academic role and such questions are better placed in the platform." Scope of removal: * Editor (review.html): drops the summary <input>, the key-terms list editor block, the self-check-questions list editor block, the related CSS rules (.vgm-key-term-row, .vgm-self-check-row), the readState/applyState/syncBindings hooks, the makeKeyTermRow / makeSelfCheckRow row factories, the add-key-term / add-self-check click handlers, and the entire glossary/highlight injection inside buildFinalHTML. * Published guide (guide.html): drops the "In brief:" advance organizer, the self-check <details> block, the bottom glossary section, the highlight_terms filter usage on narration, and the build_glossary call at the top of <main>. * LLM (ll…b863381 -
editor: long-form image description field (round 2)
ClaudeAdds the "Detailed description" field the accessibility team asked for — an editable open paragraph rendered UNDER the slide image that doubles as the long-form image description. When non-empty AND included in the published guide, the <img alt> drops to "" and the image is marked role="presentation": the description carries the meaning in body text where sighted AND non-sighted readers both encounter it, so there's no need to duplicate it inside an alt attribute that only screen readers hear. The pieces: * SegmentMetadata + Segment + migration gain description / description_ reviewed fields. Round-trips through Save/Load JSON. * New LLMToggles.description toggle. _FIELD_PROMPTS adds a description prompt that follows the team's example ("A gallery of purple flower stem macro photographs is shown. These demonstrate..."), and the structured-output schema gains a matching description field that only fires when the toggle is on (no wasted output tokens otherwise). * Orchestra…cf53b59
-
llm_extract: cut per-segment API cost and add transient-error retry
ClaudeThree small efficiency wins on the Claude calls that dominate job cost: * Drop the Tesseract OCR block from the user message when the LLM is itself producing OCR (toggles.ocr_text). The block was being sent only to be ignored — ~50-300 wasted input tokens per segment, scaled with slide text density. * Build one Anthropic client per extract_all run and reuse it across all worker threads, instead of constructing a fresh client (and TCP+ TLS handshake) per segment. * Warm the prompt cache by firing the first segment synchronously before parallelising the rest. Previously the first wave of 4 worker threads raced for the cache write and only one won, so cache hits started at segment 5; now they start at segment 2. Also wraps both extract_one and extract_topics in a single-retry helper that catches RateLimitError / APIConnectionError / 5xx-overloaded statuses, so a transient blip no longer permanently drops a segment's LLM metadata. 4xx are not retried. https://claude.ai…3602e87