cvpfus Codex commited on
Commit
6710ea1
·
1 Parent(s): fb88a58

Add live image model integrations

Browse files

Co-authored-by: Codex <codex@openai.com>

.gitignore CHANGED
@@ -11,3 +11,5 @@ outputs/*.log
11
  server.pid
12
  server.out.log
13
  server.err.log
 
 
 
11
  server.pid
12
  server.out.log
13
  server.err.log
14
+ .*/skills/
15
+ .*/commands/
FIELD_NOTES.md CHANGED
@@ -20,9 +20,9 @@ Tiny Narrator is a small-model accessibility article reader. It is not a generic
20
  | Role | Model | Size | Why it belongs |
21
  | --- | --- | ---: | --- |
22
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | Local narration planning through llama.cpp. |
23
- | Vision | `openbmb/MiniCPM-V-2` | 3B | Describes generated article images and handles OCR-like image text. |
24
  | Speech | `hexgrad/Kokoro-82M` | 82M | Fast screen-reader voice with a tiny footprint. |
25
- | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Generates the article illustrations while staying Tiny Titan-safe. |
26
 
27
  The app exposes the same budget through `/api/model-budget`, including numeric `params_billion` values, per-model `within_limit` values, and a boolean `all_models_within_limit` check. The sidebar model stack renders each model id, runtime, parameter count, and Tiny Titan pass state so the claim is visible during the demo.
28
 
@@ -34,15 +34,15 @@ Submission readiness lives at `/api/submission-readiness`. It aggregates the dem
34
 
35
  The judge evidence bundle lives at `/api/evidence-bundle`. That endpoint returns the core receipts, including schema version, UTC generation time, `PUBLIC_BASE_URL`, runtime status, and readiness, as formatted JSON.
36
 
37
- Image provenance lives in `/api/image-descriptions`. Each article illustration carries the planned FLUX.2 klein model id, prompt, seed, asset URL, and fallback status so the generated-image claim is inspectable.
38
 
39
- The Generate route uses `/api/generate-article` to turn a user's topic into a short semantic article. It uses the reader-brain model path for article text when llama.cpp is online, keeps a deterministic fallback draft for demos, and attaches thumbnail provenance from `black-forest-labs/FLUX.2-klein-4B`.
40
 
41
  Accessibility evidence lives at `/api/accessibility-audit`. It records the reader-mode choices that matter most for this prototype: semantic reading order, keyboard navigation, reader cursor state, shortcut safety, live narration, image alt text, transcript review, user-controlled playback, and fallback resilience.
42
 
43
  ## Reader Mode Behavior
44
 
45
- The frontend creates a reading queue from semantic nodes: headings, paragraphs, quotes, figures, captions, and controls. It renders that same queue in the session panel with click-to-read controls, speaks one item at a time, keeps keyboard focus synchronized with the active node, marks that node with `aria-current`, and exposes the current narration through an `aria-live` region.
46
 
47
  When screen reader mode turns on, the reader cursor starts at the focused article node or the most visible article node. That makes the feature feel connected to what the user is already reading instead of always jumping back to the top of the page.
48
 
@@ -65,7 +65,7 @@ Keyboard map:
65
 
66
  ## Fallback Strategy
67
 
68
- Live accessibility demos need dependable behavior. If llama.cpp is not available, the app uses deterministic local narration prefixes. If Kokoro is not installed or cannot load a voice, the app returns a short silent WAV while keeping the transcript visible. If the VLM integration is unavailable, image ids map to cached alt text.
69
 
70
  The HTML starts with meaningful fallback `alt` text for each generated image. The app then preloads image descriptions through `/api/image-descriptions`, caches them in the frontend, and writes the descriptions back into each image's `alt` attribute. That keeps the visible article, transcript, and spoken image narration aligned even if the model path is unavailable.
71
 
@@ -87,4 +87,22 @@ The app exposes `/api/award-evidence`, so the live demo can point directly to th
87
 
88
  The app also exposes `/api/submission-readiness`, so the Field Notes story has a single rollup of which submission claims are currently backed by app evidence.
89
 
90
- `/api/runtime-status` keeps the demo honest: if llama.cpp or Kokoro is unavailable, the API labels the fallback state instead of silently pretending that every model path is online.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  | Role | Model | Size | Why it belongs |
21
  | --- | --- | ---: | --- |
22
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | Local narration planning through llama.cpp. |
23
+ | Vision | `openbmb/MiniCPM-V-4.6` | 1B | Describes generated article images and handles OCR-like image text through an OpenAI-compatible endpoint. |
24
  | Speech | `hexgrad/Kokoro-82M` | 82M | Fast screen-reader voice with a tiny footprint. |
25
+ | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Generates the article illustrations through a Modal-hosted worker, with bundled SVG fallback. |
26
 
27
  The app exposes the same budget through `/api/model-budget`, including numeric `params_billion` values, per-model `within_limit` values, and a boolean `all_models_within_limit` check. The sidebar model stack renders each model id, runtime, parameter count, and Tiny Titan pass state so the claim is visible during the demo.
28
 
 
34
 
35
  The judge evidence bundle lives at `/api/evidence-bundle`. That endpoint returns the core receipts, including schema version, UTC generation time, `PUBLIC_BASE_URL`, runtime status, and readiness, as formatted JSON.
36
 
37
+ Image provenance lives in `/api/image-descriptions`. Each article illustration carries the planned FLUX.2 klein generation model id, prompt, seed, asset URL, and fallback status. The same response also reports whether image descriptions came from MiniCPM-V-4.6 or cached fallback alt text.
38
 
39
+ The Generate route uses `/api/generate-article` to turn a user's topic into a short semantic article. It uses the reader-brain model path for article text when llama.cpp is online, keeps a deterministic fallback draft for demos, and attaches thumbnail provenance from `black-forest-labs/FLUX.2-klein-4B`. When `KLEIN_MODAL_ENDPOINT` is configured, the thumbnail uses live Modal Klein inference. Otherwise, bundled SVG assets are served with explicit fallback runtime metadata.
40
 
41
  Accessibility evidence lives at `/api/accessibility-audit`. It records the reader-mode choices that matter most for this prototype: semantic reading order, keyboard navigation, reader cursor state, shortcut safety, live narration, image alt text, transcript review, user-controlled playback, and fallback resilience.
42
 
43
  ## Reader Mode Behavior
44
 
45
+ The frontend creates an internal reading queue from semantic nodes: headings, paragraphs, quotes, figures, captions, and controls. It speaks one item at a time, keeps keyboard focus synchronized with the active node, marks that node with `aria-current`, and exposes the current narration through an `aria-live` region.
46
 
47
  When screen reader mode turns on, the reader cursor starts at the focused article node or the most visible article node. That makes the feature feel connected to what the user is already reading instead of always jumping back to the top of the page.
48
 
 
65
 
66
  ## Fallback Strategy
67
 
68
+ Live accessibility demos need dependable behavior. If llama.cpp is not available, the app uses deterministic local narration prefixes. If Kokoro is not installed or cannot load a voice, the app returns a short silent WAV while keeping the transcript visible. If the MiniCPM-V-4.6 endpoint is unavailable, image ids map to cached alt text.
69
 
70
  The HTML starts with meaningful fallback `alt` text for each generated image. The app then preloads image descriptions through `/api/image-descriptions`, caches them in the frontend, and writes the descriptions back into each image's `alt` attribute. That keeps the visible article, transcript, and spoken image narration aligned even if the model path is unavailable.
71
 
 
87
 
88
  The app also exposes `/api/submission-readiness`, so the Field Notes story has a single rollup of which submission claims are currently backed by app evidence.
89
 
90
+ `/api/runtime-status` keeps the demo honest: if llama.cpp, MiniCPM-V-4.6, Kokoro, or Modal Klein is unavailable, the API labels the fallback state instead of silently pretending that every model path is online. The same endpoint reports whether the Modal Klein image worker is online or fallback-ready, so the demo can distinguish live Klein inference from bundled assets.
91
+
92
+ ## MiniCPM-V-4.6 Image Descriptions
93
+
94
+ The image-description path uses `openbmb/MiniCPM-V-4.6` through an OpenAI-compatible `/v1/chat/completions` endpoint. Set `MINICPM_VISION_BASE_URL` and `MINICPM_VISION_API_KEY` to enable live alt-text generation. `MINICPM_VISION_MODEL` defaults to `openbmb/MiniCPM-V-4.6`, and `MINICPM_VISION_TIMEOUT_SECONDS` controls the request timeout.
95
+
96
+ The app sends a multimodal chat message with an accessibility-focused prompt and an `image_url`. It asks for one or two concise screen-reader sentences, visible content, meaningful layout or text, and no implementation details.
97
+
98
+ If the endpoint is missing, unreachable, or returns invalid content, `/api/describe-image` and `/api/image-descriptions` return deterministic cached alt text with fallback runtime metadata. API keys are never included in runtime setup, runtime status, or evidence bundle responses.
99
+
100
+ ## Modal Klein Image Generation
101
+
102
+ The image generation path runs `black-forest-labs/FLUX.2-klein-4B` through a Modal worker (`modal_workers/klein_image.py`). The worker exposes `POST /generate` for inference and `GET /media/{filename}` for serving generated PNGs from a persistent Modal volume.
103
+
104
+ Deploy the worker with `modal deploy modal_workers/klein_image.py`, then set `KLEIN_MODAL_ENDPOINT` to the deployed worker URL. The app's `generate_image_core()` calls the Modal endpoint when configured and falls back to bundled SVG assets on missing endpoint, network errors, timeouts, or invalid responses.
105
+
106
+ `KLEIN_MODAL_TIMEOUT_SECONDS` defaults to 120 seconds, which accommodates warm inference but avoids blocking local demos indefinitely during cold starts.
107
+
108
+ This design keeps verification stable without requiring Modal credentials or GPU access: the verifier checks fallback behavior, and live Modal deployment is a manual step documented in the README.
README.md CHANGED
@@ -28,9 +28,9 @@ For the hackathon form, see [SUBMISSION.md](SUBMISSION.md).
28
  | Role | Model | Params | Runtime |
29
  | --- | ---: | ---: | --- |
30
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | `llama.cpp` |
31
- | Image understanding | `openbmb/MiniCPM-V-2` | 3B | Python integration planned |
32
  | Text to speech | `hexgrad/Kokoro-82M` | 82M | Python |
33
- | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Python integration planned |
34
 
35
  ## Run Locally
36
 
@@ -54,6 +54,24 @@ python app.py
54
 
55
  Open the local URL printed by Gradio. The Reader route calls `/api/reader-brain`, `/api/image-descriptions`, `/api/describe-image`, `/api/speak`, and `/api/model-budget`. The Generate route calls `/api/generate-article`, which drafts an article with the reader-brain model path and attaches a Klein thumbnail receipt.
56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
  Useful environment variables:
58
 
59
  | Variable | Default | Purpose |
@@ -63,6 +81,12 @@ Useful environment variables:
63
  | `GRADIO_SERVER_NAME` | `0.0.0.0` | Bind address for local or Space runtime |
64
  | `GRADIO_SERVER_PORT` / `PORT` | `7860` | App port |
65
  | `PUBLIC_BASE_URL` | `http://localhost:7860` | Base URL used in generated judge API commands |
 
 
 
 
 
 
66
 
67
  ## Verification
68
 
@@ -115,7 +139,7 @@ Reader buttons expose `aria-keyshortcuts`, and the Repeat and Stop shortcuts are
115
 
116
  The session panel keeps a transcript of recent narration with reader position, runtime, latency, copy, and clear controls, making the spoken path inspectable during demos and useful for the Field Notes write-up. Copy actions report when browser clipboard access is unavailable, so the visible transcript and commands remain inspectable.
117
 
118
- Images start with meaningful fallback `alt` text in the HTML. Image descriptions are then preloaded into a local cache and written into the page's real `img alt` attributes. When the planned MiniCPM-V runtime is unavailable, deterministic alt-text fallbacks keep the screen-reader path usable.
119
 
120
  `/api/image-descriptions` exposes image receipts so judges can inspect the prompt, seed, planned <=4B image model, and fallback status behind each bundled article illustration without lengthening the reader sidebar.
121
 
@@ -143,4 +167,4 @@ The sidebar model stack reads `/api/model-budget` so judges can see each role's
143
 
144
  ## Generate Route
145
 
146
- The header exposes two routes: Reader and Generate. Generate lets a user enter a topic, calls `/api/generate-article`, and renders a short semantic article with a thumbnail receipt. The text path uses the configured llama.cpp reader-brain model when available and falls back to deterministic article structure. The thumbnail path uses the planned Klein image model metadata and bundled fallback assets until the full image adapter is online.
 
28
  | Role | Model | Params | Runtime |
29
  | --- | ---: | ---: | --- |
30
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | `llama.cpp` |
31
+ | Image understanding | `openbmb/MiniCPM-V-4.6` | 1B | OpenAI-compatible chat completions |
32
  | Text to speech | `hexgrad/Kokoro-82M` | 82M | Python |
33
+ | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Modal-hosted Klein |
34
 
35
  ## Run Locally
36
 
 
54
 
55
  Open the local URL printed by Gradio. The Reader route calls `/api/reader-brain`, `/api/image-descriptions`, `/api/describe-image`, `/api/speak`, and `/api/model-budget`. The Generate route calls `/api/generate-article`, which drafts an article with the reader-brain model path and attaches a Klein thumbnail receipt.
56
 
57
+ ## Modal Klein Image Generation
58
+
59
+ The image generation path uses a Modal-hosted `black-forest-labs/FLUX.2-klein-4B` worker. Deploy the worker:
60
+
61
+ ```powershell
62
+ modal deploy modal_workers/klein_image.py
63
+ ```
64
+
65
+ Set `KLEIN_MODAL_ENDPOINT` to the deployed worker URL. When configured, `/api/generate-image` and `/api/generate-article` call the live Modal endpoint for real Klein inference. When the endpoint is not set, unreachable, or returns invalid data, the app falls back to bundled SVG assets with explicit fallback metadata.
66
+
67
+ `/api/runtime-status` reports whether the Modal Klein worker is online or fallback-ready. `/api/runtime-setup` includes the deploy command and required environment variables.
68
+
69
+ ## MiniCPM-V-4.6 Image Descriptions
70
+
71
+ The image description path uses `openbmb/MiniCPM-V-4.6` through an OpenAI-compatible `/v1/chat/completions` endpoint. Provide `MINICPM_VISION_BASE_URL` and `MINICPM_VISION_API_KEY` to enable live screen-reader alt text for article images.
72
+
73
+ When the endpoint is not configured, unreachable, or returns invalid content, `/api/describe-image` and `/api/image-descriptions` fall back to deterministic cached alt text so reader mode remains usable.
74
+
75
  Useful environment variables:
76
 
77
  | Variable | Default | Purpose |
 
81
  | `GRADIO_SERVER_NAME` | `0.0.0.0` | Bind address for local or Space runtime |
82
  | `GRADIO_SERVER_PORT` / `PORT` | `7860` | App port |
83
  | `PUBLIC_BASE_URL` | `http://localhost:7860` | Base URL used in generated judge API commands |
84
+ | `KLEIN_MODAL_ENDPOINT` | *(empty)* | Base URL for the Modal Klein worker, without trailing slash. When set, `/api/generate-image` calls the live worker. |
85
+ | `KLEIN_MODAL_TIMEOUT_SECONDS` | `120` | Request timeout for Modal Klein image generation. |
86
+ | `MINICPM_VISION_BASE_URL` | *(empty)* | OpenAI-compatible MiniCPM-V-4.6 base URL; root or `/v1` both work. |
87
+ | `MINICPM_VISION_API_KEY` | *(empty)* | Bearer token for the MiniCPM vision endpoint. |
88
+ | `MINICPM_VISION_MODEL` | `openbmb/MiniCPM-V-4.6` | Model id sent to chat completions. |
89
+ | `MINICPM_VISION_TIMEOUT_SECONDS` | `45` | Request timeout for image descriptions. |
90
 
91
  ## Verification
92
 
 
139
 
140
  The session panel keeps a transcript of recent narration with reader position, runtime, latency, copy, and clear controls, making the spoken path inspectable during demos and useful for the Field Notes write-up. Copy actions report when browser clipboard access is unavailable, so the visible transcript and commands remain inspectable.
141
 
142
+ Images start with meaningful fallback `alt` text in the HTML. Image descriptions are then preloaded into a local cache and written into the page's real `img alt` attributes. When the MiniCPM-V-4.6 endpoint is unavailable, deterministic alt-text fallbacks keep the screen-reader path usable.
143
 
144
  `/api/image-descriptions` exposes image receipts so judges can inspect the prompt, seed, planned <=4B image model, and fallback status behind each bundled article illustration without lengthening the reader sidebar.
145
 
 
167
 
168
  ## Generate Route
169
 
170
+ The header exposes two routes: Reader and Generate. Generate lets a user enter a topic, calls `/api/generate-article`, and renders a short semantic article with a thumbnail receipt. The text path uses the configured llama.cpp reader-brain model when available and falls back to deterministic article structure. The thumbnail path calls the Modal Klein worker for live image generation when `KLEIN_MODAL_ENDPOINT` is configured, and falls back to bundled SVG assets with explicit runtime metadata when the worker is unavailable.
SUBMISSION.md CHANGED
@@ -22,16 +22,16 @@ The prototype is designed for a live hackathon demo: every model-facing path has
22
  | Role | Model | Size | Runtime |
23
  | --- | --- | ---: | --- |
24
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | `llama.cpp` |
25
- | Image understanding | `openbmb/MiniCPM-V-2` | 3B | Python integration planned |
26
  | Speech | `hexgrad/Kokoro-82M` | 82M | Python |
27
- | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Python integration planned |
28
 
29
  ## Demo flow
30
 
31
  1. Open the article and show that the app is a custom article interface, not a stock chatbot page.
32
  2. Turn on screen-reader mode and press `Space` or `Next` to narrate the first semantic node.
33
  3. Use `Heading`, `Image`, and `Summary` to navigate by article meaning instead of by raw page order.
34
- 4. Show the reader-first session panel: current item, live narration, reader queue, transcript, and latency.
35
  5. Point to the model stack panel for model id, runtime, parameter count, and Tiny Titan pass status.
36
  6. Open the Generate route, enter a topic, and show the generated article plus `black-forest-labs/FLUX.2-klein-4B` thumbnail receipt.
37
  7. Mention that `/api/demo-script` exposes the judge runbook and API evidence checks as structured data.
@@ -53,7 +53,9 @@ The checks in `/api/demo-script` include copyable curl and PowerShell-friendly `
53
 
54
  The accessibility audit also documents reader-mode details that matter during judging: the active item is exposed as a reader cursor with focus, visible outline, stable id, and `aria-current`; global shortcuts ignore form controls so voice, speed, and auto-advance settings remain usable while reader mode is active; reader controls expose `aria-keyshortcuts` plus visible Repeat and Stop commands.
55
 
56
- The image receipts keep the generated-asset claim inspectable: each article illustration names the planned `black-forest-labs/FLUX.2-klein-4B` path, prompt, seed, and bundled fallback asset.
 
 
57
 
58
  The submission-readiness panel and API give judges a compact checklist for the whole build, so the live demo can move from individual receipts to an overall readiness view.
59
 
@@ -61,4 +63,4 @@ The submission-readiness panel and API give judges a compact checklist for the w
61
 
62
  ## Reliability notes
63
 
64
- The demo remains navigable when local models are unavailable. The reader brain falls back to deterministic narration, generated images start with meaningful HTML alt text, image descriptions fall back to cached alt text, speech falls back to browser speech plus transcript, and generated images fall back to bundled article assets. Fallback states are labeled instead of hidden.
 
22
  | Role | Model | Size | Runtime |
23
  | --- | --- | ---: | --- |
24
  | Reader brain | `nvidia/NVIDIA-Nemotron-3-Nano-4B-GGUF` | 3.97B | `llama.cpp` |
25
+ | Image understanding | `openbmb/MiniCPM-V-4.6` | 1B | OpenAI-compatible chat completions |
26
  | Speech | `hexgrad/Kokoro-82M` | 82M | Python |
27
+ | Image generation | `black-forest-labs/FLUX.2-klein-4B` | 4B | Modal-hosted Klein (bundled fallback when not configured) |
28
 
29
  ## Demo flow
30
 
31
  1. Open the article and show that the app is a custom article interface, not a stock chatbot page.
32
  2. Turn on screen-reader mode and press `Space` or `Next` to narrate the first semantic node.
33
  3. Use `Heading`, `Image`, and `Summary` to navigate by article meaning instead of by raw page order.
34
+ 4. Show the reader-first session panel: current item, live narration, transcript, model stack, and latency.
35
  5. Point to the model stack panel for model id, runtime, parameter count, and Tiny Titan pass status.
36
  6. Open the Generate route, enter a topic, and show the generated article plus `black-forest-labs/FLUX.2-klein-4B` thumbnail receipt.
37
  7. Mention that `/api/demo-script` exposes the judge runbook and API evidence checks as structured data.
 
53
 
54
  The accessibility audit also documents reader-mode details that matter during judging: the active item is exposed as a reader cursor with focus, visible outline, stable id, and `aria-current`; global shortcuts ignore form controls so voice, speed, and auto-advance settings remain usable while reader mode is active; reader controls expose `aria-keyshortcuts` plus visible Repeat and Stop commands.
55
 
56
+ The image receipts distinguish live Modal Klein inference from bundled fallback assets. When `KLEIN_MODAL_ENDPOINT` is configured and reachable, generated thumbnails use real `black-forest-labs/FLUX.2-klein-4B` inference through Modal. When the worker is unavailable, bundled SVG assets are used with explicit fallback runtime metadata.
57
+
58
+ Image descriptions use `openbmb/MiniCPM-V-4.6` through an OpenAI-compatible `/v1/chat/completions` endpoint when `MINICPM_VISION_BASE_URL` and `MINICPM_VISION_API_KEY` are configured. Without that endpoint, cached deterministic alt text keeps screen-reader mode stable.
59
 
60
  The submission-readiness panel and API give judges a compact checklist for the whole build, so the live demo can move from individual receipts to an overall readiness view.
61
 
 
63
 
64
  ## Reliability notes
65
 
66
+ The demo remains navigable when local models are unavailable. The reader brain falls back to deterministic narration, generated images start with meaningful HTML alt text, image descriptions fall back to cached alt text, speech falls back to browser speech plus transcript, and image generation falls back to bundled article assets when the Modal Klein worker is not configured or unreachable. Fallback states are labeled instead of hidden.
app.py CHANGED
@@ -24,11 +24,25 @@ STATIC_DIR = ROOT / "static"
24
  OUTPUT_DIR = ROOT / "outputs"
25
  OUTPUT_DIR.mkdir(exist_ok=True)
26
 
 
 
 
 
 
 
 
 
27
  LLAMA_CPP_BASE_URL = os.getenv("LLAMA_CPP_BASE_URL", "http://localhost:8080/v1")
28
  LLAMA_CPP_MODEL = os.getenv("LLAMA_CPP_MODEL", "narrator-brain")
29
  GRADIO_SERVER_NAME = os.getenv("GRADIO_SERVER_NAME", "0.0.0.0")
30
  GRADIO_SERVER_PORT = int(os.getenv("PORT", os.getenv("GRADIO_SERVER_PORT", "7860")))
31
  PUBLIC_BASE_URL = os.getenv("PUBLIC_BASE_URL", f"http://localhost:{GRADIO_SERVER_PORT}").rstrip("/")
 
 
 
 
 
 
32
  TINY_TITAN_LIMIT_B = 4.0
33
 
34
  MODEL_MANIFEST: dict[str, dict[str, Any]] = {
@@ -40,10 +54,10 @@ MODEL_MANIFEST: dict[str, dict[str, Any]] = {
40
  "role": "Plans concise narration and reading-order phrasing.",
41
  },
42
  "vision": {
43
- "id": "openbmb/MiniCPM-V-2",
44
- "params": "3B",
45
- "params_billion": 3.0,
46
- "runtime": "Python integration planned",
47
  "role": "Describes images and OCR-like visual details.",
48
  },
49
  "speech": {
@@ -57,7 +71,7 @@ MODEL_MANIFEST: dict[str, dict[str, Any]] = {
57
  "id": "black-forest-labs/FLUX.2-klein-4B",
58
  "params": "4B",
59
  "params_billion": 4.0,
60
- "runtime": "Python integration planned",
61
  "role": "Creates article illustrations.",
62
  },
63
  }
@@ -188,18 +202,26 @@ def runtime_setup_core() -> dict[str, Any]:
188
  "role": "vision",
189
  "label": "Image descriptions",
190
  "model": MODEL_MANIFEST["vision"]["id"],
191
- "runtime": "Python integration planned",
192
- "command": "planned MiniCPM-V adapter",
193
- "env": {},
 
 
 
 
 
194
  "fallback": "cached deterministic alt text",
195
  },
196
  {
197
  "role": "image_generation",
198
  "label": "Article images",
199
  "model": MODEL_MANIFEST["image_generation"]["id"],
200
- "runtime": "Python integration planned",
201
- "command": "planned FLUX.2 klein adapter",
202
- "env": {},
 
 
 
203
  "fallback": "bundled generated article assets",
204
  },
205
  ],
@@ -229,7 +251,7 @@ def demo_script_core() -> dict[str, Any]:
229
  {"method": "GET", "path": "/api/runtime-setup", "expect": "llama.cpp, Kokoro, vision, and image paths"},
230
  {"method": "GET", "path": "/api/runtime-status", "expect": "online or fallback-ready runtime labels"},
231
  {"method": "GET", "path": "/api/accessibility-audit", "expect": "all reader-mode checks pass"},
232
- {"method": "GET", "path": "/api/image-descriptions", "expect": "three article image descriptions"},
233
  {"method": "GET", "path": "/api/submission-readiness", "expect": "all submission readiness checks pass"},
234
  {
235
  "method": "POST",
@@ -528,6 +550,7 @@ class ImageDescriptionRequest(BaseModel):
528
  image_id: str
529
  caption: str | None = None
530
  prompt: str | None = None
 
531
 
532
 
533
  class SpeechRequest(BaseModel):
@@ -553,6 +576,15 @@ def _elapsed_ms(start: float) -> int:
553
  return round((time.perf_counter() - start) * 1000)
554
 
555
 
 
 
 
 
 
 
 
 
 
556
  def _runtime_status_core() -> dict[str, Any]:
557
  start = time.perf_counter()
558
  llama_start = time.perf_counter()
@@ -584,15 +616,46 @@ def _runtime_status_core() -> dict[str, Any]:
584
  kokoro_available = importlib.util.find_spec("kokoro") is not None
585
  soundfile_available = importlib.util.find_spec("soundfile") is not None
586
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
587
  return {
588
  "ok": True,
589
  "reader_brain": llama_status,
590
- "vision": {
591
- "available": False,
592
- "status": "fallback-ready",
593
- "model": "OpenBMB MiniCPM-V-2",
594
- "fallback": "cached deterministic alt text",
595
- },
596
  "speech": {
597
  "available": kokoro_available and soundfile_available,
598
  "status": "online" if kokoro_available and soundfile_available else "fallback-ready",
@@ -601,12 +664,7 @@ def _runtime_status_core() -> dict[str, Any]:
601
  "soundfile_installed": soundfile_available,
602
  "fallback": "browser speech plus transcript",
603
  },
604
- "image_generation": {
605
- "available": False,
606
- "status": "placeholder-ready",
607
- "model": "black-forest-labs/FLUX.2-klein-4B",
608
- "fallback": "bundled generated article assets",
609
- },
610
  "elapsed_ms": _elapsed_ms(start),
611
  }
612
 
@@ -742,10 +800,36 @@ def reader_brain_core(node_type: str, text: str, position: str | None, mode: str
742
  }
743
 
744
 
745
- def describe_image_core(image_id: str, caption: str | None, prompt: str | None) -> dict[str, Any]:
746
- start = time.perf_counter()
747
- # The first committed slice keeps the API stable while the MiniCPM-V runtime lands next.
748
- # The frontend passes deterministic image ids so cached descriptions can replace this later.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
749
  descriptions = {
750
  "desk-reader": (
751
  "A person reads a long article on a laptop while an accessibility toolbar "
@@ -756,15 +840,136 @@ def describe_image_core(image_id: str, caption: str | None, prompt: str | None)
756
  "reader brain, speech, and image generation."
757
  ),
758
  }
759
- alt_text = descriptions.get(
760
  image_id,
761
  caption or prompt or "A generated article image awaiting model description.",
762
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
763
  return {
764
  "ok": True,
765
- "runtime": "MiniCPM-V placeholder",
766
- "model": "OpenBMB MiniCPM-V-2",
767
  "alt_text": alt_text,
 
768
  "elapsed_ms": _elapsed_ms(start),
769
  }
770
 
@@ -777,12 +982,14 @@ def describe_article_images_core() -> dict[str, Any]:
777
  image["id"],
778
  caption=image.get("caption"),
779
  prompt=image.get("prompt"),
 
780
  )
781
  descriptions.append({**image, **description})
 
782
  return {
783
  "ok": True,
784
- "runtime": "MiniCPM-V placeholder",
785
- "model": "OpenBMB MiniCPM-V-2",
786
  "descriptions": descriptions,
787
  "elapsed_ms": _elapsed_ms(start),
788
  }
@@ -845,19 +1052,64 @@ def speak_core(text: str, voice: str, speed: float) -> dict[str, Any]:
845
  }
846
 
847
 
848
- def generate_image_core(prompt: str, seed: int | None) -> dict[str, Any]:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
849
  start = time.perf_counter()
850
  return {
851
  "ok": True,
852
- "runtime": "placeholder",
853
- "model": "black-forest-labs/FLUX.2-klein-4B",
854
  "image_url": f"/static/generated/{'desk-reader.svg' if seed and seed % 2 else 'model-map.svg'}",
855
  "prompt": prompt,
856
  "seed": seed,
 
857
  "elapsed_ms": _elapsed_ms(start),
858
  }
859
 
860
 
 
 
 
 
 
 
 
 
 
861
  def generate_article_core(topic: str) -> dict[str, Any]:
862
  start = time.perf_counter()
863
  clean_topic = _compact_text(topic, 100).strip()
@@ -1008,7 +1260,7 @@ async def reader_brain_endpoint(payload: ReaderBrainRequest) -> JSONResponse:
1008
 
1009
  @app.post("/api/describe-image")
1010
  async def describe_image_endpoint(payload: ImageDescriptionRequest) -> JSONResponse:
1011
- return _json(describe_image_core(payload.image_id, payload.caption, payload.prompt))
1012
 
1013
 
1014
  @app.get("/api/image-descriptions")
@@ -1037,8 +1289,8 @@ def reader_brain_api(node_type: str, text: str, position: str = "", mode: str =
1037
 
1038
 
1039
  @app.api(name="describe_image")
1040
- def describe_image_api(image_id: str, caption: str = "", prompt: str = "") -> str:
1041
- return json.dumps(describe_image_core(image_id, caption, prompt))
1042
 
1043
 
1044
  @app.api(name="describe_article_images")
 
24
  OUTPUT_DIR = ROOT / "outputs"
25
  OUTPUT_DIR.mkdir(exist_ok=True)
26
 
27
+
28
+ def _int_env(name: str, default: int) -> int:
29
+ try:
30
+ return int(os.getenv(name, str(default)))
31
+ except (TypeError, ValueError):
32
+ return default
33
+
34
+
35
  LLAMA_CPP_BASE_URL = os.getenv("LLAMA_CPP_BASE_URL", "http://localhost:8080/v1")
36
  LLAMA_CPP_MODEL = os.getenv("LLAMA_CPP_MODEL", "narrator-brain")
37
  GRADIO_SERVER_NAME = os.getenv("GRADIO_SERVER_NAME", "0.0.0.0")
38
  GRADIO_SERVER_PORT = int(os.getenv("PORT", os.getenv("GRADIO_SERVER_PORT", "7860")))
39
  PUBLIC_BASE_URL = os.getenv("PUBLIC_BASE_URL", f"http://localhost:{GRADIO_SERVER_PORT}").rstrip("/")
40
+ KLEIN_MODAL_ENDPOINT = os.getenv("KLEIN_MODAL_ENDPOINT", "").rstrip("/")
41
+ KLEIN_MODAL_TIMEOUT_SECONDS = _int_env("KLEIN_MODAL_TIMEOUT_SECONDS", 120)
42
+ MINICPM_VISION_BASE_URL = os.getenv("MINICPM_VISION_BASE_URL", "").rstrip("/")
43
+ MINICPM_VISION_API_KEY = os.getenv("MINICPM_VISION_API_KEY", "")
44
+ MINICPM_VISION_MODEL = os.getenv("MINICPM_VISION_MODEL", "openbmb/MiniCPM-V-4.6")
45
+ MINICPM_VISION_TIMEOUT_SECONDS = _int_env("MINICPM_VISION_TIMEOUT_SECONDS", 45)
46
  TINY_TITAN_LIMIT_B = 4.0
47
 
48
  MODEL_MANIFEST: dict[str, dict[str, Any]] = {
 
54
  "role": "Plans concise narration and reading-order phrasing.",
55
  },
56
  "vision": {
57
+ "id": "openbmb/MiniCPM-V-4.6",
58
+ "params": "1B",
59
+ "params_billion": 1.0,
60
+ "runtime": "OpenAI-compatible chat completions",
61
  "role": "Describes images and OCR-like visual details.",
62
  },
63
  "speech": {
 
71
  "id": "black-forest-labs/FLUX.2-klein-4B",
72
  "params": "4B",
73
  "params_billion": 4.0,
74
+ "runtime": "Modal-hosted Klein",
75
  "role": "Creates article illustrations.",
76
  },
77
  }
 
202
  "role": "vision",
203
  "label": "Image descriptions",
204
  "model": MODEL_MANIFEST["vision"]["id"],
205
+ "runtime": "OpenAI-compatible chat completions",
206
+ "command": "vllm serve openbmb/MiniCPM-V-4.6 --host 0.0.0.0 --port 8000",
207
+ "env": {
208
+ "MINICPM_VISION_BASE_URL": MINICPM_VISION_BASE_URL or "(set to OpenAI-compatible base URL)",
209
+ "MINICPM_VISION_API_KEY": "(configured)" if MINICPM_VISION_API_KEY else "(not set)",
210
+ "MINICPM_VISION_MODEL": MINICPM_VISION_MODEL,
211
+ "MINICPM_VISION_TIMEOUT_SECONDS": str(MINICPM_VISION_TIMEOUT_SECONDS),
212
+ },
213
  "fallback": "cached deterministic alt text",
214
  },
215
  {
216
  "role": "image_generation",
217
  "label": "Article images",
218
  "model": MODEL_MANIFEST["image_generation"]["id"],
219
+ "runtime": "Modal-hosted Klein",
220
+ "command": "modal deploy modal_workers/klein_image.py",
221
+ "env": {
222
+ "KLEIN_MODAL_ENDPOINT": KLEIN_MODAL_ENDPOINT or "(set to Modal worker URL)",
223
+ "KLEIN_MODAL_TIMEOUT_SECONDS": str(KLEIN_MODAL_TIMEOUT_SECONDS),
224
+ },
225
  "fallback": "bundled generated article assets",
226
  },
227
  ],
 
251
  {"method": "GET", "path": "/api/runtime-setup", "expect": "llama.cpp, Kokoro, vision, and image paths"},
252
  {"method": "GET", "path": "/api/runtime-status", "expect": "online or fallback-ready runtime labels"},
253
  {"method": "GET", "path": "/api/accessibility-audit", "expect": "all reader-mode checks pass"},
254
+ {"method": "GET", "path": "/api/image-descriptions", "expect": "two article image descriptions with MiniCPM-V-4.6 or fallback alt text"},
255
  {"method": "GET", "path": "/api/submission-readiness", "expect": "all submission readiness checks pass"},
256
  {
257
  "method": "POST",
 
550
  image_id: str
551
  caption: str | None = None
552
  prompt: str | None = None
553
+ image_url: str | None = None
554
 
555
 
556
  class SpeechRequest(BaseModel):
 
576
  return round((time.perf_counter() - start) * 1000)
577
 
578
 
579
+ def _validate_modal_klein_health(payload: dict[str, Any]) -> None:
580
+ if payload.get("ok") is not True:
581
+ raise ValueError("health check did not return ok")
582
+ if payload.get("model") != MODEL_MANIFEST["image_generation"]["id"]:
583
+ raise ValueError("health check returned unexpected model")
584
+ if payload.get("runtime") != "modal-klein":
585
+ raise ValueError("health check returned unexpected runtime")
586
+
587
+
588
  def _runtime_status_core() -> dict[str, Any]:
589
  start = time.perf_counter()
590
  llama_start = time.perf_counter()
 
616
  kokoro_available = importlib.util.find_spec("kokoro") is not None
617
  soundfile_available = importlib.util.find_spec("soundfile") is not None
618
 
619
+ klein_status: dict[str, Any]
620
+ if KLEIN_MODAL_ENDPOINT:
621
+ try:
622
+ request = urllib.request.Request(
623
+ f"{KLEIN_MODAL_ENDPOINT}/health", method="GET",
624
+ )
625
+ with urllib.request.urlopen(request, timeout=5) as response:
626
+ payload = json.loads(response.read().decode("utf-8"))
627
+ _validate_modal_klein_health(payload)
628
+ klein_status = {
629
+ "available": True,
630
+ "status": "online",
631
+ "model": MODEL_MANIFEST["image_generation"]["id"],
632
+ "endpoint": KLEIN_MODAL_ENDPOINT,
633
+ "elapsed_ms": _elapsed_ms(start),
634
+ }
635
+ except (OSError, urllib.error.URLError, TimeoutError, ValueError, json.JSONDecodeError) as exc:
636
+ klein_status = {
637
+ "available": False,
638
+ "status": "fallback-ready",
639
+ "model": MODEL_MANIFEST["image_generation"]["id"],
640
+ "endpoint": KLEIN_MODAL_ENDPOINT,
641
+ "fallback": "bundled generated article assets",
642
+ "warning": exc.__class__.__name__,
643
+ "elapsed_ms": _elapsed_ms(start),
644
+ }
645
+ else:
646
+ klein_status = {
647
+ "available": False,
648
+ "status": "fallback-ready",
649
+ "model": MODEL_MANIFEST["image_generation"]["id"],
650
+ "fallback": "bundled generated article assets",
651
+ }
652
+
653
+ vision_status = _vision_runtime_status()
654
+
655
  return {
656
  "ok": True,
657
  "reader_brain": llama_status,
658
+ "vision": vision_status,
 
 
 
 
 
659
  "speech": {
660
  "available": kokoro_available and soundfile_available,
661
  "status": "online" if kokoro_available and soundfile_available else "fallback-ready",
 
664
  "soundfile_installed": soundfile_available,
665
  "fallback": "browser speech plus transcript",
666
  },
667
+ "image_generation": klein_status,
 
 
 
 
 
668
  "elapsed_ms": _elapsed_ms(start),
669
  }
670
 
 
800
  }
801
 
802
 
803
+ def _openai_compatible_url(base_url: str, path: str) -> str:
804
+ normalized = base_url.rstrip("/")
805
+ if normalized.endswith("/v1/chat/completions"):
806
+ root = normalized[: -len("/chat/completions")]
807
+ elif normalized.endswith("/v1"):
808
+ root = normalized
809
+ else:
810
+ root = f"{normalized}/v1"
811
+ return f"{root}/{path.lstrip('/')}"
812
+
813
+
814
+ def _article_image_by_id(image_id: str) -> dict[str, Any] | None:
815
+ return next((image for image in ARTICLE_IMAGES if image["id"] == image_id), None)
816
+
817
+
818
+ def _absolute_image_url(image_url: str | None, image_id: str | None = None) -> str | None:
819
+ candidate = image_url
820
+ if not candidate and image_id:
821
+ image = _article_image_by_id(image_id)
822
+ candidate = image.get("asset_url") if image else None
823
+ if not candidate:
824
+ return None
825
+ if candidate.startswith(("http://", "https://", "data:")):
826
+ return candidate
827
+ if candidate.startswith("/"):
828
+ return f"{PUBLIC_BASE_URL}{candidate}"
829
+ return f"{PUBLIC_BASE_URL}/{candidate}"
830
+
831
+
832
+ def _vision_fallback_text(image_id: str, caption: str | None, prompt: str | None) -> str:
833
  descriptions = {
834
  "desk-reader": (
835
  "A person reads a long article on a laptop while an accessibility toolbar "
 
840
  "reader brain, speech, and image generation."
841
  ),
842
  }
843
+ return descriptions.get(
844
  image_id,
845
  caption or prompt or "A generated article image awaiting model description.",
846
  )
847
+
848
+
849
+ def _clean_alt_text(text: str, limit: int = 260) -> str:
850
+ return _compact_text(" ".join(text.split()), limit)
851
+
852
+
853
+ def _minicpm_vision_prompt(caption: str | None, prompt: str | None) -> str:
854
+ context = " ".join(part for part in [caption, prompt] if part)
855
+ context_line = f"\nContext: {context}" if context else ""
856
+ return (
857
+ "Describe this image for a screen reader in one or two concise sentences. "
858
+ "Mention visible content, layout, important text, and purpose when relevant. "
859
+ "Do not guess hidden intent. Do not mention models, implementation details, or that you are an AI. "
860
+ "Return plain text only."
861
+ f"{context_line}"
862
+ )
863
+
864
+
865
+ def _call_minicpm_vision(image_url: str, caption: str | None, prompt: str | None) -> dict[str, Any]:
866
+ start = time.perf_counter()
867
+ body = json.dumps(
868
+ {
869
+ "model": MINICPM_VISION_MODEL,
870
+ "messages": [
871
+ {
872
+ "role": "user",
873
+ "content": [
874
+ {"type": "text", "text": _minicpm_vision_prompt(caption, prompt)},
875
+ {"type": "image_url", "image_url": {"url": image_url}},
876
+ ],
877
+ }
878
+ ],
879
+ "temperature": 0.1,
880
+ "max_tokens": 160,
881
+ }
882
+ ).encode("utf-8")
883
+ request = urllib.request.Request(
884
+ _openai_compatible_url(MINICPM_VISION_BASE_URL, "chat/completions"),
885
+ data=body,
886
+ headers={
887
+ "Authorization": f"Bearer {MINICPM_VISION_API_KEY}",
888
+ "Content-Type": "application/json",
889
+ },
890
+ method="POST",
891
+ )
892
+ with urllib.request.urlopen(request, timeout=MINICPM_VISION_TIMEOUT_SECONDS) as response:
893
+ payload = json.loads(response.read().decode("utf-8"))
894
+ content = payload["choices"][0]["message"]["content"]
895
+ if not isinstance(content, str) or not content.strip():
896
+ raise ValueError("MiniCPM vision response did not include text content")
897
+ return {
898
+ "runtime": "minicpm-v4.6",
899
+ "model": MINICPM_VISION_MODEL,
900
+ "alt_text": _clean_alt_text(content),
901
+ "elapsed_ms": _elapsed_ms(start),
902
+ }
903
+
904
+
905
+ def _vision_runtime_status() -> dict[str, Any]:
906
+ if not MINICPM_VISION_BASE_URL or not MINICPM_VISION_API_KEY:
907
+ return {
908
+ "available": False,
909
+ "status": "fallback-ready",
910
+ "model": MINICPM_VISION_MODEL,
911
+ "configured": False,
912
+ "fallback": "cached deterministic alt text",
913
+ }
914
+ start = time.perf_counter()
915
+ try:
916
+ request = urllib.request.Request(
917
+ _openai_compatible_url(MINICPM_VISION_BASE_URL, "models"),
918
+ headers={"Authorization": f"Bearer {MINICPM_VISION_API_KEY}"},
919
+ method="GET",
920
+ )
921
+ with urllib.request.urlopen(request, timeout=5) as response:
922
+ payload = json.loads(response.read().decode("utf-8"))
923
+ model_ids = [
924
+ item.get("id", "")
925
+ for item in payload.get("data", [])
926
+ if isinstance(item, dict)
927
+ ]
928
+ if model_ids and MINICPM_VISION_MODEL not in model_ids:
929
+ raise ValueError("MiniCPM vision model was not listed by /models")
930
+ return {
931
+ "available": True,
932
+ "status": "online",
933
+ "model": MINICPM_VISION_MODEL,
934
+ "configured": True,
935
+ "base_url": MINICPM_VISION_BASE_URL,
936
+ "models": model_ids,
937
+ "elapsed_ms": _elapsed_ms(start),
938
+ }
939
+ except (OSError, urllib.error.URLError, TimeoutError, ValueError, json.JSONDecodeError) as exc:
940
+ return {
941
+ "available": False,
942
+ "status": "fallback-ready",
943
+ "model": MINICPM_VISION_MODEL,
944
+ "configured": True,
945
+ "base_url": MINICPM_VISION_BASE_URL,
946
+ "fallback": "cached deterministic alt text",
947
+ "warning": exc.__class__.__name__,
948
+ "elapsed_ms": _elapsed_ms(start),
949
+ }
950
+
951
+
952
+ def describe_image_core(
953
+ image_id: str,
954
+ caption: str | None,
955
+ prompt: str | None,
956
+ image_url: str | None = None,
957
+ ) -> dict[str, Any]:
958
+ start = time.perf_counter()
959
+ resolved_image_url = _absolute_image_url(image_url, image_id)
960
+ alt_text = _vision_fallback_text(image_id, caption, prompt)
961
+ warning = None
962
+ if MINICPM_VISION_BASE_URL and MINICPM_VISION_API_KEY and resolved_image_url:
963
+ try:
964
+ return {"ok": True, **_call_minicpm_vision(resolved_image_url, caption, prompt)}
965
+ except (OSError, urllib.error.URLError, TimeoutError, KeyError, IndexError, TypeError, ValueError, json.JSONDecodeError) as exc:
966
+ warning = f"MiniCPM vision unavailable: {exc.__class__.__name__}"
967
  return {
968
  "ok": True,
969
+ "runtime": "fallback",
970
+ "model": MINICPM_VISION_MODEL,
971
  "alt_text": alt_text,
972
+ "warning": warning,
973
  "elapsed_ms": _elapsed_ms(start),
974
  }
975
 
 
982
  image["id"],
983
  caption=image.get("caption"),
984
  prompt=image.get("prompt"),
985
+ image_url=image.get("asset_url"),
986
  )
987
  descriptions.append({**image, **description})
988
+ runtimes = {item["runtime"] for item in descriptions}
989
  return {
990
  "ok": True,
991
+ "runtime": runtimes.pop() if len(runtimes) == 1 else "mixed",
992
+ "model": MINICPM_VISION_MODEL,
993
  "descriptions": descriptions,
994
  "elapsed_ms": _elapsed_ms(start),
995
  }
 
1052
  }
1053
 
1054
 
1055
+ def _call_modal_klein(prompt: str, seed: int | None) -> dict[str, Any]:
1056
+ """Call the Modal Klein worker for live image generation."""
1057
+ start = time.perf_counter()
1058
+ body = json.dumps({"prompt": prompt, "seed": seed}).encode("utf-8")
1059
+ request = urllib.request.Request(
1060
+ f"{KLEIN_MODAL_ENDPOINT}/generate",
1061
+ data=body,
1062
+ headers={"Content-Type": "application/json"},
1063
+ method="POST",
1064
+ )
1065
+ with urllib.request.urlopen(request, timeout=KLEIN_MODAL_TIMEOUT_SECONDS) as response:
1066
+ payload = json.loads(response.read().decode("utf-8"))
1067
+ if payload.get("ok") is not True:
1068
+ raise ValueError(str(payload.get("error") or payload.get("detail") or "Modal Klein returned ok=false"))
1069
+ if payload.get("model") != MODEL_MANIFEST["image_generation"]["id"]:
1070
+ raise ValueError("Modal Klein returned unexpected model")
1071
+ if payload.get("runtime") != "modal-klein":
1072
+ raise ValueError("Modal Klein returned unexpected runtime")
1073
+ image_url = payload.get("image_url", "")
1074
+ if not isinstance(image_url, str) or not image_url:
1075
+ raise ValueError("Modal Klein response did not include image_url")
1076
+ if image_url and image_url.startswith("/media/"):
1077
+ image_url = f"{KLEIN_MODAL_ENDPOINT}{image_url}"
1078
+ return {
1079
+ "ok": True,
1080
+ "runtime": payload.get("runtime", "modal-klein"),
1081
+ "model": payload.get("model", MODEL_MANIFEST["image_generation"]["id"]),
1082
+ "image_url": image_url,
1083
+ "prompt": payload.get("prompt", prompt),
1084
+ "seed": payload.get("seed", seed),
1085
+ "elapsed_ms": payload.get("elapsed_ms", _elapsed_ms(start)),
1086
+ }
1087
+
1088
+
1089
+ def _fallback_image(prompt: str, seed: int | None, warning: str | None = None) -> dict[str, Any]:
1090
+ """Return bundled fallback SVG assets when Modal is unavailable."""
1091
  start = time.perf_counter()
1092
  return {
1093
  "ok": True,
1094
+ "runtime": "fallback",
1095
+ "model": MODEL_MANIFEST["image_generation"]["id"],
1096
  "image_url": f"/static/generated/{'desk-reader.svg' if seed and seed % 2 else 'model-map.svg'}",
1097
  "prompt": prompt,
1098
  "seed": seed,
1099
+ "warning": warning,
1100
  "elapsed_ms": _elapsed_ms(start),
1101
  }
1102
 
1103
 
1104
+ def generate_image_core(prompt: str, seed: int | None) -> dict[str, Any]:
1105
+ if KLEIN_MODAL_ENDPOINT:
1106
+ try:
1107
+ return _call_modal_klein(prompt, seed)
1108
+ except (OSError, urllib.error.URLError, TimeoutError, ValueError, KeyError, json.JSONDecodeError) as exc:
1109
+ return _fallback_image(prompt, seed, warning=f"Modal Klein unavailable: {exc.__class__.__name__}")
1110
+ return _fallback_image(prompt, seed)
1111
+
1112
+
1113
  def generate_article_core(topic: str) -> dict[str, Any]:
1114
  start = time.perf_counter()
1115
  clean_topic = _compact_text(topic, 100).strip()
 
1260
 
1261
  @app.post("/api/describe-image")
1262
  async def describe_image_endpoint(payload: ImageDescriptionRequest) -> JSONResponse:
1263
+ return _json(describe_image_core(payload.image_id, payload.caption, payload.prompt, payload.image_url))
1264
 
1265
 
1266
  @app.get("/api/image-descriptions")
 
1289
 
1290
 
1291
  @app.api(name="describe_image")
1292
+ def describe_image_api(image_id: str, caption: str = "", prompt: str = "", image_url: str = "") -> str:
1293
+ return json.dumps(describe_image_core(image_id, caption, prompt, image_url))
1294
 
1295
 
1296
  @app.api(name="describe_article_images")
modal_workers/__init__.py ADDED
File without changes
modal_workers/klein_image.py ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Modal worker for black-forest-labs/FLUX.2-klein-4B image generation.
2
+
3
+ Deploy:
4
+ modal deploy modal_workers/klein_image.py
5
+
6
+ The deployed ASGI app exposes one base URL with:
7
+ POST /generate - accepts {"prompt": str, "seed": int | None}
8
+ GET /health - reports worker readiness
9
+ GET /media/{filename} - serves generated PNG files
10
+
11
+ Tiny Narrator's app.py calls these routes through KLEIN_MODAL_ENDPOINT.
12
+ """
13
+
14
+ from __future__ import annotations
15
+
16
+ import io
17
+ import time
18
+ from pathlib import Path
19
+ from uuid import uuid4
20
+
21
+ import modal
22
+
23
+ app = modal.App("tiny-narrator-klein")
24
+
25
+ output_volume = modal.Volume.from_name("tiny-narrator-klein-outputs", create_if_missing=True)
26
+ MEDIA_DIR = Path("/outputs")
27
+
28
+ IMAGE_MODEL_ID = "black-forest-labs/FLUX.2-klein-4B"
29
+ _PIPELINE = None
30
+
31
+ klein_image = (
32
+ modal.Image.debian_slim(python_version="3.11")
33
+ .pip_install(
34
+ "fastapi[standard]",
35
+ "huggingface-hub[hf-transfer]>=0.36.0",
36
+ "pillow",
37
+ "torch",
38
+ "transformers",
39
+ "git+https://github.com/huggingface/diffusers.git",
40
+ "safetensors>=0.8.0rc0",
41
+ "accelerate",
42
+ "sentencepiece",
43
+ )
44
+ .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
45
+ )
46
+
47
+
48
+ def _get_pipeline():
49
+ """Load the Klein pipeline once per warm ASGI container."""
50
+ global _PIPELINE
51
+ if _PIPELINE is None:
52
+ import torch
53
+ from diffusers import FluxPipeline
54
+
55
+ pipe = FluxPipeline.from_pretrained(
56
+ IMAGE_MODEL_ID,
57
+ torch_dtype=torch.bfloat16,
58
+ )
59
+ pipe.to("cuda")
60
+ _PIPELINE = pipe
61
+ return _PIPELINE
62
+
63
+
64
+ @app.function(
65
+ image=klein_image,
66
+ gpu="A10G",
67
+ timeout=900,
68
+ container_idle_timeout=300,
69
+ volumes={str(MEDIA_DIR): output_volume},
70
+ )
71
+ @modal.asgi_app()
72
+ def klein_api():
73
+ from fastapi import FastAPI, HTTPException
74
+ from fastapi.responses import Response
75
+ import torch
76
+
77
+ api = FastAPI(title="Tiny Narrator Klein Worker")
78
+
79
+ @api.get("/health")
80
+ async def health() -> dict:
81
+ return {
82
+ "ok": True,
83
+ "model": IMAGE_MODEL_ID,
84
+ "runtime": "modal-klein",
85
+ }
86
+
87
+ @api.post("/generate")
88
+ async def generate(request: dict) -> dict:
89
+ start = time.perf_counter()
90
+ prompt = str(request.get("prompt") or "").strip()
91
+ if not prompt:
92
+ raise HTTPException(status_code=400, detail="prompt is required")
93
+
94
+ seed = request.get("seed")
95
+ generator = None
96
+ if seed is not None:
97
+ try:
98
+ seed = int(seed)
99
+ except (TypeError, ValueError) as exc:
100
+ raise HTTPException(status_code=400, detail="seed must be an integer") from exc
101
+ generator = torch.Generator(device="cuda").manual_seed(seed)
102
+
103
+ result = _get_pipeline()(
104
+ prompt=prompt,
105
+ num_inference_steps=28,
106
+ guidance_scale=3.5,
107
+ generator=generator,
108
+ )
109
+ image = result.images[0]
110
+
111
+ filename = f"klein-{uuid4().hex}.png"
112
+ output_path = MEDIA_DIR / filename
113
+ output_path.parent.mkdir(parents=True, exist_ok=True)
114
+
115
+ buffer = io.BytesIO()
116
+ image.save(buffer, format="PNG")
117
+ output_path.write_bytes(buffer.getvalue())
118
+ output_volume.commit()
119
+
120
+ return {
121
+ "ok": True,
122
+ "runtime": "modal-klein",
123
+ "model": IMAGE_MODEL_ID,
124
+ "image_url": f"/media/{filename}",
125
+ "prompt": prompt,
126
+ "seed": seed,
127
+ "elapsed_ms": round((time.perf_counter() - start) * 1000),
128
+ }
129
+
130
+ @api.get("/media/{filename}")
131
+ async def media(filename: str):
132
+ output_volume.reload()
133
+ media_path = MEDIA_DIR / filename
134
+ if not media_path.exists() or not media_path.is_file():
135
+ raise HTTPException(status_code=404, detail=f"Media file not found: {filename}")
136
+
137
+ data = media_path.read_bytes()
138
+ content_type = "image/png"
139
+ if filename.endswith(".webp"):
140
+ content_type = "image/webp"
141
+ elif filename.endswith(".jpg") or filename.endswith(".jpeg"):
142
+ content_type = "image/jpeg"
143
+
144
+ return Response(content=data, media_type=content_type)
145
+
146
+ return api
openspec/changes/add-minicpm-v46-image-descriptor/.openspec.yaml ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ schema: spec-driven
2
+ created: 2026-06-15
openspec/changes/add-minicpm-v46-image-descriptor/design.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Design: MiniCPM-V-4.6 Image Descriptor
2
+
3
+ ## Overview
4
+
5
+ Tiny Narrator will keep the existing image-description APIs but upgrade the vision role from cached-only placeholder behavior to a live MiniCPM-V-4.6 path when configured.
6
+
7
+ The backend will call an OpenAI-compatible `/v1/chat/completions` endpoint. The user will provide the base URL and API key. Local development and verification will continue to pass without credentials by using deterministic fallbacks.
8
+
9
+ ## Configuration
10
+
11
+ Add environment variables:
12
+
13
+ | Variable | Purpose |
14
+ | --- | --- |
15
+ | `MINICPM_VISION_BASE_URL` | Base URL for the OpenAI-compatible MiniCPM-V-4.6 server. Accept either the server root or a URL ending in `/v1`; app code should normalize to `/v1/chat/completions`. |
16
+ | `MINICPM_VISION_API_KEY` | Bearer token for the MiniCPM vision endpoint. |
17
+ | `MINICPM_VISION_MODEL` | Optional model id, default `openbmb/MiniCPM-V-4.6`. |
18
+ | `MINICPM_VISION_TIMEOUT_SECONDS` | Request timeout, default short enough for UI responsiveness but long enough for warm inference. |
19
+
20
+ The API key must never be exposed in runtime setup/status payloads. Status may report whether a key is configured.
21
+
22
+ ## Request Shape
23
+
24
+ The app should send a multimodal chat-completions request:
25
+
26
+ ```json
27
+ {
28
+ "model": "openbmb/MiniCPM-V-4.6",
29
+ "messages": [
30
+ {
31
+ "role": "user",
32
+ "content": [
33
+ {
34
+ "type": "text",
35
+ "text": "Describe this image for a screen reader..."
36
+ },
37
+ {
38
+ "type": "image_url",
39
+ "image_url": {
40
+ "url": "https://public-base-url/static/generated/model-map.svg"
41
+ }
42
+ }
43
+ ]
44
+ }
45
+ ],
46
+ "temperature": 0.1,
47
+ "max_tokens": 160
48
+ }
49
+ ```
50
+
51
+ The prompt should ask for concise accessibility alt text:
52
+
53
+ - one or two sentences
54
+ - no implementation details
55
+ - describe visible content, layout, text, and purpose when relevant
56
+ - avoid guessing hidden intent
57
+ - return plain text only
58
+
59
+ ## Image URL Resolution
60
+
61
+ For known article images, use the `asset_url` from `ARTICLE_IMAGES`. If it is relative, prefix `PUBLIC_BASE_URL`.
62
+
63
+ For direct `POST /api/describe-image` calls:
64
+
65
+ - continue supporting `image_id`, `caption`, and `prompt`
66
+ - optionally add `image_url` to the request schema so future callers can describe external or generated images
67
+ - if no usable image URL is available, use fallback text instead of calling the vision endpoint
68
+
69
+ ## Response Handling
70
+
71
+ Parse the first chat completion message:
72
+
73
+ ```json
74
+ choices[0].message.content
75
+ ```
76
+
77
+ Validate that the content is a non-empty string. Normalize whitespace and limit length for safe UI/transcript use. Return the existing response shape with live metadata:
78
+
79
+ ```json
80
+ {
81
+ "ok": true,
82
+ "runtime": "minicpm-v4.6",
83
+ "model": "openbmb/MiniCPM-V-4.6",
84
+ "alt_text": "...",
85
+ "elapsed_ms": 1234
86
+ }
87
+ ```
88
+
89
+ If the endpoint is missing, credentials are missing, the image URL cannot be resolved, the request fails, or the response is invalid, return deterministic fallback alt text with:
90
+
91
+ ```json
92
+ {
93
+ "runtime": "fallback",
94
+ "model": "openbmb/MiniCPM-V-4.6",
95
+ "warning": "MiniCPM vision unavailable: ..."
96
+ }
97
+ ```
98
+
99
+ When no live call was attempted because configuration is absent, warning can be omitted to keep local demos quiet.
100
+
101
+ ## Runtime Setup And Status
102
+
103
+ `runtime_setup_core()` should document:
104
+
105
+ - the expected OpenAI-compatible serving path
106
+ - environment variables
107
+ - example commands for compatible runtimes, such as `vllm serve openbmb/MiniCPM-V-4.6` or `llama-server` with a MiniCPM-V-4.6 GGUF
108
+ - fallback behavior
109
+
110
+ `_runtime_status_core()` should:
111
+
112
+ - report `vision.status = "fallback-ready"` when not configured
113
+ - report `vision.status = "online"` only when both base URL and API key are configured and a lightweight readiness check succeeds
114
+ - avoid sending images in the status check; use `/models` if available, or a minimal chat-completions check only if necessary
115
+ - never expose `MINICPM_VISION_API_KEY`
116
+
117
+ ## Frontend Behavior
118
+
119
+ No major frontend changes are required. The existing reader route already preloads `/api/image-descriptions`, writes returned alt text into image attributes, and uses image descriptions for screen-reader narration.
120
+
121
+ If live MiniCPM inference is slow, the frontend should still remain usable because the backend returns fallback descriptions on timeout.
122
+
123
+ ## Testing Strategy
124
+
125
+ Verification should not need network or credentials. Add mocked tests for:
126
+
127
+ - no configuration returns fallback MiniCPM-V-4.6 metadata
128
+ - successful OpenAI-compatible response returns live alt text
129
+ - invalid response falls back with a warning
130
+ - failed request falls back with a warning
131
+ - runtime setup documents MiniCPM-V-4.6 configuration
132
+ - runtime status does not expose API keys
133
+ - existing `/api/image-descriptions` shape remains stable
134
+
135
+ Manual testing can cover a real configured endpoint after the user provides base URL and API key.
openspec/changes/add-minicpm-v46-image-descriptor/proposal.md ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Add MiniCPM-V-4.6 Image Descriptor
2
+
3
+ ## Why
4
+
5
+ Tiny Narrator currently lists `openbmb/MiniCPM-V-2` for the image descriptor role, but the implementation still returns deterministic cached alt text. The project needs real visual description inference for generated article images while keeping the existing screen-reader fallback path reliable.
6
+
7
+ `openbmb/MiniCPM-V-4.6` is a better fit for the role because it is newer, smaller, Tiny Titan-safe, and supports OpenAI-compatible chat-completions deployments through runtimes such as vLLM, SGLang, llama.cpp, or hosted services.
8
+
9
+ ## What Changes
10
+
11
+ - Switch the planned vision model from `openbmb/MiniCPM-V-2` to `openbmb/MiniCPM-V-4.6`.
12
+ - Add configuration for an OpenAI-compatible `/v1/chat/completions` vision endpoint.
13
+ - Call the configured MiniCPM-V-4.6 endpoint from the image-description path when credentials are available.
14
+ - Preserve deterministic cached alt-text fallbacks when the endpoint is not configured, unreachable, or returns invalid content.
15
+ - Update runtime setup/status, evidence payloads, docs, and verifier expectations to distinguish live MiniCPM-V-4.6 inference from fallback descriptions.
16
+
17
+ ## Capabilities
18
+
19
+ - `minicpm-v46-image-description`: generate screen-reader-ready image descriptions using MiniCPM-V-4.6 through an OpenAI-compatible chat-completions API.
20
+ - `image-description-fallbacks`: retain deterministic alt text for stable local and hackathon demos.
21
+ - `vision-runtime-evidence`: expose model id, runtime status, latency, fallback state, and setup instructions through the existing evidence APIs.
22
+
23
+ ## Non-Goals
24
+
25
+ - Do not change the reader-brain llama.cpp path.
26
+ - Do not change Kokoro TTS.
27
+ - Do not change the Modal Klein image-generation worker.
28
+ - Do not require the MiniCPM endpoint or API key for local verification to pass.
29
+ - Do not add a separate evidence UI page.
30
+
31
+ ## Impact
32
+
33
+ - `app.py` will gain MiniCPM-V-4.6 endpoint configuration and an OpenAI-compatible vision client.
34
+ - Existing `/api/describe-image` and `/api/image-descriptions` responses will keep their current shape but may return live MiniCPM runtime metadata.
35
+ - README, SUBMISSION, and FIELD_NOTES will describe MiniCPM-V-4.6 setup and fallback behavior.
36
+ - `scripts/verify.py` will cover fallback behavior and mocked live MiniCPM responses without making network calls.
openspec/changes/add-minicpm-v46-image-descriptor/tasks.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tasks: MiniCPM-V-4.6 Image Descriptor
2
+
3
+ ## 1. Model Metadata And Configuration
4
+
5
+ - [x] Update `MODEL_MANIFEST["vision"]` to `openbmb/MiniCPM-V-4.6` with Tiny Titan-safe parameter metadata.
6
+ - [x] Add `MINICPM_VISION_BASE_URL`, `MINICPM_VISION_API_KEY`, `MINICPM_VISION_MODEL`, and `MINICPM_VISION_TIMEOUT_SECONDS`.
7
+ - [x] Add safe env parsing so invalid timeout values do not break app import.
8
+ - [x] Ensure no API key value appears in runtime setup/status/evidence responses.
9
+
10
+ ## 2. OpenAI-Compatible Vision Client
11
+
12
+ - [x] Add a helper to normalize the configured base URL to `/v1/chat/completions`.
13
+ - [x] Add a helper to resolve known article image URLs from `ARTICLE_IMAGES` and `PUBLIC_BASE_URL`.
14
+ - [x] Add a MiniCPM-V-4.6 chat-completions client using `Authorization: Bearer <key>`.
15
+ - [x] Send multimodal messages with accessibility-focused alt-text instructions and `image_url` content.
16
+ - [x] Validate `choices[0].message.content` before using it as `alt_text`.
17
+ - [x] Normalize/compact returned alt text for UI and transcript safety.
18
+
19
+ ## 3. API Behavior
20
+
21
+ - [x] Update `ImageDescriptionRequest` to optionally accept `image_url`.
22
+ - [x] Update `describe_image_core()` to call MiniCPM-V-4.6 when configuration and image URL are available.
23
+ - [x] Preserve deterministic cached fallback descriptions when not configured.
24
+ - [x] Return fallback descriptions with visible warning metadata when live inference fails or returns invalid content.
25
+ - [x] Update `describe_article_images_core()` so article image receipts include live or fallback vision runtime metadata.
26
+ - [x] Keep `/api/describe-image` and `/api/image-descriptions` response shapes backward compatible.
27
+
28
+ ## 4. Runtime Evidence
29
+
30
+ - [x] Update `runtime_setup_core()` with OpenAI-compatible MiniCPM-V-4.6 setup guidance and required env vars.
31
+ - [x] Update `_runtime_status_core()` to report MiniCPM-V-4.6 online/fallback-ready status.
32
+ - [x] Use `/models` or another lightweight non-image readiness check where possible.
33
+ - [x] Update `submission_readiness_core()` and `evidence_bundle_core()` only if their expectations need to distinguish live vision from fallback.
34
+ - [x] Ensure `/api/model-budget` still passes Tiny Titan.
35
+
36
+ ## 5. Docs
37
+
38
+ - [x] Update `README.md` recommended model table and setup section.
39
+ - [x] Update `SUBMISSION.md` to name MiniCPM-V-4.6 for image understanding.
40
+ - [x] Update `FIELD_NOTES.md` with the OpenAI-compatible endpoint design and fallback behavior.
41
+ - [x] Mention that the user supplies `MINICPM_VISION_BASE_URL` and `MINICPM_VISION_API_KEY`.
42
+
43
+ ## 6. Verification
44
+
45
+ - [x] Add verifier tests for fallback image descriptions with no MiniCPM config.
46
+ - [x] Add mocked tests for successful MiniCPM-V-4.6 chat-completions response.
47
+ - [x] Add mocked tests for failed/invalid MiniCPM responses falling back with warnings.
48
+ - [x] Add tests that runtime status/setup mention MiniCPM-V-4.6 and do not leak API keys.
49
+ - [x] Run `python scripts\verify.py`.
50
+ - [x] Run `python -m py_compile app.py scripts\verify.py`.
51
+ - [x] Run `git diff --check`.
52
+
53
+ ## 7. Manual Endpoint Check
54
+
55
+ - [ ] Set `MINICPM_VISION_BASE_URL`.
56
+ - [ ] Set `MINICPM_VISION_API_KEY`.
57
+ - [ ] Call `/api/describe-image` for `model-map` and confirm live MiniCPM-V-4.6 runtime metadata.
58
+ - [ ] Open the Reader page and confirm image alt text is updated without breaking screen-reader narration.
openspec/changes/add-modal-klein-image-inference/.openspec.yaml ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ schema: spec-driven
2
+ created: 2026-06-14
openspec/changes/add-modal-klein-image-inference/design.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Design: Modal Klein Image Inference
2
+
3
+ ## Overview
4
+
5
+ Tiny Narrator will keep its current `POST /api/generate-image` and `/api/generate-article` contracts, but `generate_image_core()` will become a two-path adapter:
6
+
7
+ 1. If a Modal Klein endpoint is configured, call it for live `black-forest-labs/FLUX.2-klein-4B` inference.
8
+ 2. If the endpoint is missing or fails, return the current bundled fallback assets with explicit fallback metadata.
9
+
10
+ This preserves the app's hackathon reliability while making the image-generation claim real when the Modal worker is deployed.
11
+
12
+ ## Modal Worker
13
+
14
+ Add a new worker module, likely `modal_workers/klein_image.py`, with:
15
+
16
+ - a Modal `App`
17
+ - a persistent Modal `Volume` for generated images
18
+ - an image/container definition that installs FastAPI, Pillow, torch, transformers/diffusers dependencies, and the Klein runtime dependencies
19
+ - a model class or cached function that loads `black-forest-labs/FLUX.2-klein-4B` once per warm container
20
+ - a FastAPI web endpoint, for example `POST /generate`, accepting:
21
+ - `prompt: str`
22
+ - `seed: int | None`
23
+ - optional generation controls only if needed later
24
+ - a `GET /media/{filename}` endpoint for generated PNG/WebP files
25
+
26
+ The worker should return:
27
+
28
+ ```json
29
+ {
30
+ "ok": true,
31
+ "runtime": "modal-klein",
32
+ "model": "black-forest-labs/FLUX.2-klein-4B",
33
+ "image_url": "https://.../media/klein-....png",
34
+ "prompt": "...",
35
+ "seed": 123,
36
+ "elapsed_ms": 12345
37
+ }
38
+ ```
39
+
40
+ From the referenced TinyDeck thread, the important Modal implementation lessons are:
41
+
42
+ - Use an explicit media helper for `GET /media/{filename}`.
43
+ - Call `output_volume.reload()` before serving media from a web container.
44
+ - Return a clean 404 for missing media instead of letting `FileResponse` crash.
45
+ - Give the web endpoint a long enough timeout for cold starts.
46
+ - If using a GitHub Diffusers build, avoid incompatible `safetensors==0.7.0`; use a compatible lower bound such as `safetensors>=0.8.0rc0` if that dependency path is chosen.
47
+
48
+ ## App Configuration
49
+
50
+ Add environment variables:
51
+
52
+ | Variable | Purpose |
53
+ | --- | --- |
54
+ | `KLEIN_MODAL_ENDPOINT` | Base URL for the Modal Klein worker, without trailing slash. |
55
+ | `KLEIN_MODAL_TIMEOUT_SECONDS` | Request timeout for image generation, defaulting high enough for warm inference but not blocking local demos forever. |
56
+
57
+ No Modal dependency is needed in the main Gradio app unless the implementation chooses a direct Modal SDK client. Prefer plain HTTP from `app.py` so Hugging Face Spaces can call the worker with standard library or already-small dependencies.
58
+
59
+ ## App Integration
60
+
61
+ `generate_image_core(prompt, seed)` should:
62
+
63
+ - record a start time
64
+ - call `POST {KLEIN_MODAL_ENDPOINT}/generate` when configured
65
+ - validate the response shape before trusting it
66
+ - preserve the returned prompt, seed, model, runtime, image URL, and elapsed time
67
+ - add a warning and fall back to bundled SVG assets on network errors, bad responses, timeout, or missing endpoint
68
+
69
+ `generate_article_core()` can keep its current article text behavior and use the upgraded `generate_image_core()` for thumbnails.
70
+
71
+ `_runtime_status_core()` should report:
72
+
73
+ - `available: true`, `status: "online"` when the worker health check succeeds
74
+ - `available: false`, `status: "fallback-ready"` when not configured or not reachable
75
+ - endpoint and timeout metadata without exposing secrets
76
+
77
+ `runtime_setup_core()` should show a copyable `modal deploy modal_workers/klein_image.py` command and the required `KLEIN_MODAL_ENDPOINT` configuration.
78
+
79
+ ## Fallbacks
80
+
81
+ Fallbacks stay first-class and visible:
82
+
83
+ - Existing bundled SVGs remain available.
84
+ - Fallback payloads keep `ok: true` because the UI can still render.
85
+ - Payloads should include `runtime: "fallback"` or `generation_runtime: "bundled fallback asset"` and a `warning` when live inference failed.
86
+ - Verification should not require Modal credentials or network access.
87
+
88
+ ## Evidence And Docs
89
+
90
+ Update evidence APIs and docs so they no longer say "Python integration planned" for image generation once the worker exists. They should say Modal-hosted Klein is supported and report whether the current runtime is online or fallback-ready.
91
+
92
+ The Generate route can continue showing thumbnail provenance through the existing receipt text. If live Modal inference is active, the receipt should reflect `modal-klein` instead of placeholder/fallback runtime.
93
+
94
+ ## Testing Strategy
95
+
96
+ Use unit-style tests around `generate_image_core()`:
97
+
98
+ - no endpoint configured returns deterministic fallback
99
+ - configured endpoint returning valid JSON is surfaced as live Modal inference
100
+ - configured endpoint failure falls back with a warning
101
+ - `/api/runtime-status` and `/api/runtime-setup` mention the Modal Klein path
102
+ - `/api/generate-article` thumbnail provenance remains stable
103
+
104
+ The Modal worker should be syntax-checked locally. Live deployment testing can be documented as a manual command because CI/local verification should not require GPU infrastructure.
openspec/changes/add-modal-klein-image-inference/proposal.md ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Add Modal Klein Image Inference
2
+
3
+ ## Why
4
+
5
+ Tiny Narrator currently documents `black-forest-labs/FLUX.2-klein-4B` as the image-generation model, but `generate_image_core()` only returns bundled fallback SVG assets with placeholder runtime metadata. That is honest, but it leaves the Generate route short of real image-generation inference.
6
+
7
+ Adding a Modal-hosted Klein worker gives the project a live image-generation path while preserving the deterministic fallback behavior that keeps hackathon demos reliable.
8
+
9
+ ## What Changes
10
+
11
+ - Add a Modal worker for `black-forest-labs/FLUX.2-klein-4B` image generation.
12
+ - Expose an HTTP JSON endpoint that accepts a prompt and optional seed, writes generated images to Modal storage, and serves them through a media route.
13
+ - Update Tiny Narrator's image-generation backend path to call the Modal endpoint when configured.
14
+ - Keep bundled fallback assets when Modal is not configured, cold-starts fail, or generation times out.
15
+ - Update runtime setup/status, evidence payloads, docs, and verifier expectations so the app distinguishes live Klein inference from fallback assets.
16
+
17
+ ## Capabilities
18
+
19
+ - `modal-klein-image-generation`: generate article thumbnails through a Modal-hosted Klein inference endpoint.
20
+ - `image-generation-fallbacks`: retain deterministic local fallback assets and explicit fallback metadata.
21
+ - `image-generation-evidence`: surface runtime status, model id, prompt, seed, latency, and generated media URL in existing evidence APIs.
22
+
23
+ ## Non-Goals
24
+
25
+ - Do not implement MiniCPM-V image-description inference in this change.
26
+ - Do not replace the existing reader, TTS, or llama.cpp paths.
27
+ - Do not require live Modal inference for local verification to pass.
28
+ - Do not add a separate evidence UI page.
29
+
30
+ ## Impact
31
+
32
+ - `app.py` will gain Modal endpoint configuration, a small image-generation client, and updated runtime/status metadata.
33
+ - A new `modal_workers/` script will be added for deploying the Klein worker.
34
+ - `README.md`, `SUBMISSION.md`, and `FIELD_NOTES.md` will describe live Modal Klein setup and fallback behavior.
35
+ - `scripts/verify.py` will assert that the API contract and fallback path remain stable without requiring Modal credentials.
openspec/changes/add-modal-klein-image-inference/tasks.md ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tasks: Modal Klein Image Inference
2
+
3
+ ## 1. Modal Worker
4
+
5
+ - [x] Add `modal_workers/klein_image.py` with a Modal app, persistent output volume, and FastAPI web endpoints.
6
+ - [x] Implement `POST /generate` for `black-forest-labs/FLUX.2-klein-4B` prompt + optional seed inference.
7
+ - [x] Implement `GET /media/{filename}` with volume reload before serving and clean 404 handling for missing files.
8
+ - [x] Add worker-level timeout and dependency pins suitable for cold starts.
9
+ - [x] Document the deploy command and expected endpoint shape in comments or docs.
10
+
11
+ ## 2. App Configuration And Client
12
+
13
+ - [x] Add `KLEIN_MODAL_ENDPOINT` and `KLEIN_MODAL_TIMEOUT_SECONDS` configuration in `app.py`.
14
+ - [x] Add a small HTTP client/helper for calling the Modal Klein worker.
15
+ - [x] Update `generate_image_core()` to use Modal when configured.
16
+ - [x] Preserve deterministic bundled fallback behavior when Modal is missing, slow, or returns invalid data.
17
+ - [x] Include `warning`, `runtime`, `model`, `prompt`, `seed`, `image_url`, and `elapsed_ms` consistently in live and fallback responses.
18
+
19
+ ## 3. Runtime Evidence
20
+
21
+ - [x] Update `MODEL_MANIFEST["image_generation"]["runtime"]` from planned integration to Modal-hosted Klein.
22
+ - [x] Update `runtime_setup_core()` with Modal deploy/setup instructions and environment variables.
23
+ - [x] Update `_runtime_status_core()` to report Modal Klein online/fallback-ready status.
24
+ - [x] Update `demo_script_core()`, `submission_readiness_core()`, and `evidence_bundle_core()` expectations as needed.
25
+ - [x] Ensure `/api/model-budget` still proves Tiny Titan <= 4B.
26
+
27
+ ## 4. Generate Route Behavior
28
+
29
+ - [x] Ensure `/api/generate-image` returns live Modal image URLs when configured.
30
+ - [x] Ensure `/api/generate-article` thumbnail receipts reflect live Modal runtime when available.
31
+ - [x] Keep the Generate page UI stable and screen-reader-compatible with generated thumbnails.
32
+ - [x] Avoid adding a new evidence UI page.
33
+
34
+ ## 5. Docs
35
+
36
+ - [x] Update `README.md` with Modal Klein setup, environment variables, and fallback behavior.
37
+ - [x] Update `SUBMISSION.md` to distinguish live Modal Klein inference from bundled fallback assets.
38
+ - [x] Update `FIELD_NOTES.md` with architecture notes, runtime honesty, and manual deploy guidance.
39
+
40
+ ## 6. Verification
41
+
42
+ - [x] Add tests in `scripts/verify.py` for fallback image generation without Modal.
43
+ - [x] Add mocked/stubbed tests for successful Modal worker responses.
44
+ - [x] Add mocked/stubbed tests for Modal failure fallback with a visible warning.
45
+ - [x] Add static checks for the Modal worker file.
46
+ - [x] Run `python scripts\verify.py`.
47
+ - [x] Run `python -m py_compile app.py scripts\verify.py modal_workers\klein_image.py`.
48
+ - [x] Run `git diff --check`.
49
+
50
+ ## 7. Manual Modal Check
51
+
52
+ - [ ] Deploy with `modal deploy modal_workers/klein_image.py`.
53
+ - [ ] Set `KLEIN_MODAL_ENDPOINT` locally or in the Space runtime.
54
+ - [ ] Call `/api/generate-image` with a sample prompt and confirm it returns a Modal media URL.
55
+ - [ ] Generate an article from `/generate` and confirm the thumbnail receipt reports live Modal Klein runtime.
openspec/config.yaml ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ schema: spec-driven
2
+
3
+ # Project context (optional)
4
+ # This is shown to AI when creating artifacts.
5
+ # Add your tech stack, conventions, style guides, domain knowledge, etc.
6
+ # Example:
7
+ # context: |
8
+ # Tech stack: TypeScript, React, Node.js
9
+ # We use conventional commits
10
+ # Domain: e-commerce platform
11
+
12
+ # Per-artifact rules (optional)
13
+ # Add custom rules for specific artifacts.
14
+ # Example:
15
+ # rules:
16
+ # proposal:
17
+ # - Keep proposals under 500 words
18
+ # - Always include a "Non-goals" section
19
+ # tasks:
20
+ # - Break tasks into chunks of max 2 hours
scripts/verify.py CHANGED
@@ -1,9 +1,11 @@
1
  from __future__ import annotations
2
 
 
3
  import py_compile
4
  import sys
5
  import wave
6
  from pathlib import Path
 
7
 
8
 
9
  ROOT = Path(__file__).resolve().parents[1]
@@ -202,6 +204,8 @@ def verify_core_fallbacks() -> None:
202
  description = app.describe_image_core("model-map", caption=None, prompt=None)
203
  assert_true(description["ok"], "Image description did not return ok")
204
  assert_true("small AI models" in description["alt_text"], "Image description fallback changed unexpectedly")
 
 
205
  assert_true(isinstance(description["elapsed_ms"], int), "Image description should include elapsed_ms")
206
 
207
  article_descriptions = app.describe_article_images_core()
@@ -216,6 +220,8 @@ def verify_core_fallbacks() -> None:
216
  "Article image descriptions should include generation seeds",
217
  )
218
  assert_true(isinstance(article_descriptions["elapsed_ms"], int), "Article image descriptions should include elapsed_ms")
 
 
219
 
220
  speech = app.speak_core("Tiny Narrator verification.", voice="af_heart", speed=1.0)
221
  assert_true(speech["ok"], "Speech path did not return ok")
@@ -229,6 +235,16 @@ def verify_core_fallbacks() -> None:
229
  generated = app.generate_image_core("Tiny accessibility article image.", seed=3)
230
  assert_true(generated["ok"], "Image generation placeholder should return ok")
231
  assert_true(isinstance(generated["elapsed_ms"], int), "Image generation should include elapsed_ms")
 
 
 
 
 
 
 
 
 
 
232
 
233
  article = app.generate_article_core("tiny classroom robotics")
234
  assert_true(article["ok"], "Article generation should return ok")
@@ -240,6 +256,202 @@ def verify_core_fallbacks() -> None:
240
  )
241
 
242
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
243
  def verify_output_retention() -> None:
244
  keep_path = app.OUTPUT_DIR / "speech-retention-keep.wav"
245
  _ = app.speak_core("Tiny Narrator retention check.", voice="af_heart", speed=1.0)
@@ -540,8 +752,8 @@ def verify_routes() -> None:
540
  )
541
  assert_true(evidence_payload["runtime_status"]["ok"], "Evidence bundle should include ok runtime status")
542
  assert_true(
543
- {"reader_brain", "speech"} <= set(evidence_payload["runtime_status"]),
544
- "Evidence bundle runtime status should include reader brain and speech status",
545
  )
546
  assert_true(
547
  evidence_payload["runtime_status"]["reader_brain"]["status"] in {"online", "fallback-ready"},
@@ -551,6 +763,10 @@ def verify_routes() -> None:
551
  evidence_payload["runtime_status"]["speech"]["status"] in {"online", "fallback-ready"},
552
  "Evidence bundle speech status should be online or fallback-ready",
553
  )
 
 
 
 
554
 
555
  runtime = client.get("/api/runtime-status")
556
  assert_true(runtime.status_code == 200, "Runtime status route should return 200")
@@ -558,15 +774,25 @@ def verify_routes() -> None:
558
  assert_true(runtime_payload["ok"], "Runtime status payload should be ok")
559
  assert_true("reader_brain" in runtime_payload, "Runtime status should include reader brain status")
560
  assert_true("speech" in runtime_payload, "Runtime status should include speech status")
 
561
  assert_true(
562
  runtime_payload["reader_brain"]["status"] in {"online", "fallback-ready"},
563
  "Reader brain status should be online or fallback-ready",
564
  )
 
 
 
 
 
 
 
 
565
 
566
  image_descriptions = client.get("/api/image-descriptions")
567
  assert_true(image_descriptions.status_code == 200, "Image descriptions route should return 200")
568
  image_payload = image_descriptions.json()
569
- assert_true(image_payload["model"] == "OpenBMB MiniCPM-V-2", "Image route should document MiniCPM-V model")
 
570
  assert_true(
571
  {item["id"] for item in image_payload["descriptions"]} == {"desk-reader", "model-map"},
572
  "Image route should describe all visible article images",
@@ -580,12 +806,24 @@ def verify_routes() -> None:
580
  "Image route should label bundled image fallback status",
581
  )
582
 
 
 
 
 
 
 
 
 
 
583
 
584
  def main() -> None:
585
  py_compile.compile(str(ROOT / "app.py"), doraise=True)
 
586
  verify_static_assets()
587
  verify_space_metadata()
588
  verify_core_fallbacks()
 
 
589
  verify_output_retention()
590
  verify_routes()
591
  print("Tiny Narrator verification passed.")
 
1
  from __future__ import annotations
2
 
3
+ import json
4
  import py_compile
5
  import sys
6
  import wave
7
  from pathlib import Path
8
+ from unittest.mock import patch, MagicMock
9
 
10
 
11
  ROOT = Path(__file__).resolve().parents[1]
 
204
  description = app.describe_image_core("model-map", caption=None, prompt=None)
205
  assert_true(description["ok"], "Image description did not return ok")
206
  assert_true("small AI models" in description["alt_text"], "Image description fallback changed unexpectedly")
207
+ assert_true(description["runtime"] == "fallback", "Image description without MiniCPM config should use fallback runtime")
208
+ assert_true(description["model"] == app.MODEL_MANIFEST["vision"]["id"], "Image description should report the vision model id")
209
  assert_true(isinstance(description["elapsed_ms"], int), "Image description should include elapsed_ms")
210
 
211
  article_descriptions = app.describe_article_images_core()
 
220
  "Article image descriptions should include generation seeds",
221
  )
222
  assert_true(isinstance(article_descriptions["elapsed_ms"], int), "Article image descriptions should include elapsed_ms")
223
+ assert_true(article_descriptions["model"] == app.MODEL_MANIFEST["vision"]["id"], "Article image descriptions should report vision model id")
224
+ assert_true(article_descriptions["runtime"] == "fallback", "Article image descriptions should use fallback without MiniCPM config")
225
 
226
  speech = app.speak_core("Tiny Narrator verification.", voice="af_heart", speed=1.0)
227
  assert_true(speech["ok"], "Speech path did not return ok")
 
235
  generated = app.generate_image_core("Tiny accessibility article image.", seed=3)
236
  assert_true(generated["ok"], "Image generation placeholder should return ok")
237
  assert_true(isinstance(generated["elapsed_ms"], int), "Image generation should include elapsed_ms")
238
+ assert_true(generated["runtime"] == "fallback", "Image generation without Modal should report fallback runtime")
239
+ assert_true(generated["model"] == app.MODEL_MANIFEST["image_generation"]["id"], "Image generation should report Klein model id")
240
+ assert_true(generated["seed"] == 3, "Image generation should preserve the seed")
241
+ assert_true(generated["image_url"].startswith("/static/generated/"), "Fallback image should use bundled assets")
242
+ assert_true(generated.get("warning") is None, "Fallback without attempted call should have no warning")
243
+
244
+ generated_no_seed = app.generate_image_core("Another image prompt.", seed=None)
245
+ assert_true(generated_no_seed["ok"], "Image generation without seed should return ok")
246
+ assert_true(generated_no_seed["runtime"] == "fallback", "Image generation without Modal should report fallback")
247
+ assert_true(generated_no_seed["seed"] is None, "Image generation should pass through None seed")
248
 
249
  article = app.generate_article_core("tiny classroom robotics")
250
  assert_true(article["ok"], "Article generation should return ok")
 
256
  )
257
 
258
 
259
+ def verify_modal_klein_integration() -> None:
260
+ """Test Modal Klein integration with mocked HTTP responses."""
261
+ # Test: no endpoint configured returns deterministic fallback
262
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", ""):
263
+ result = app.generate_image_core("test prompt", seed=42)
264
+ assert_true(result["ok"], "No-endpoint fallback should return ok")
265
+ assert_true(result["runtime"] == "fallback", "No-endpoint should report fallback runtime")
266
+ assert_true(result["image_url"].startswith("/static/generated/"), "No-endpoint should use bundled assets")
267
+ assert_true(result["model"] == app.MODEL_MANIFEST["image_generation"]["id"], "Fallback should report Klein model")
268
+
269
+ # Test: configured endpoint returning valid JSON is surfaced as live Modal inference
270
+ mock_response_data = {
271
+ "ok": True,
272
+ "runtime": "modal-klein",
273
+ "model": "black-forest-labs/FLUX.2-klein-4B",
274
+ "image_url": "/media/klein-abc123.png",
275
+ "prompt": "test prompt",
276
+ "seed": 42,
277
+ "elapsed_ms": 5000,
278
+ }
279
+ mock_response = MagicMock()
280
+ mock_response.read.return_value = json.dumps(mock_response_data).encode("utf-8")
281
+ mock_response.__enter__ = lambda s: s
282
+ mock_response.__exit__ = MagicMock(return_value=False)
283
+
284
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", "https://example-modal.modal.run"):
285
+ with patch("urllib.request.urlopen", return_value=mock_response):
286
+ result = app.generate_image_core("test prompt", seed=42)
287
+ assert_true(result["ok"], "Modal success should return ok")
288
+ assert_true(result["runtime"] == "modal-klein", "Modal success should report modal-klein runtime")
289
+ assert_true(
290
+ result["image_url"] == "https://example-modal.modal.run/media/klein-abc123.png",
291
+ "Modal success should return full media URL",
292
+ )
293
+ assert_true(result["prompt"] == "test prompt", "Modal success should preserve prompt")
294
+ assert_true(result["seed"] == 42, "Modal success should preserve seed")
295
+
296
+ # Test: configured endpoint failure falls back with a visible warning
297
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", "https://example-modal.modal.run"):
298
+ with patch("urllib.request.urlopen", side_effect=TimeoutError("Connection timed out")):
299
+ result = app.generate_image_core("test prompt", seed=42)
300
+ assert_true(result["ok"], "Modal failure fallback should return ok")
301
+ assert_true(result["runtime"] == "fallback", "Modal failure should report fallback runtime")
302
+ assert_true(
303
+ result["image_url"].startswith("/static/generated/"),
304
+ "Modal failure should use bundled assets",
305
+ )
306
+ assert_true(
307
+ result.get("warning") is not None and "TimeoutError" in result["warning"],
308
+ "Modal failure should include a visible warning with the error class",
309
+ )
310
+
311
+ # Test: invalid Modal JSON response falls back instead of pretending success
312
+ invalid_response = MagicMock()
313
+ invalid_response.read.return_value = json.dumps({"ok": False, "error": "prompt is required"}).encode("utf-8")
314
+ invalid_response.__enter__ = lambda s: s
315
+ invalid_response.__exit__ = MagicMock(return_value=False)
316
+
317
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", "https://example-modal.modal.run"):
318
+ with patch("urllib.request.urlopen", return_value=invalid_response):
319
+ result = app.generate_image_core("test prompt", seed=42)
320
+ assert_true(result["runtime"] == "fallback", "Invalid Modal response should use fallback runtime")
321
+ assert_true(
322
+ result.get("warning") is not None and "ValueError" in result["warning"],
323
+ "Invalid Modal response should include a validation warning",
324
+ )
325
+
326
+ # Test: Modal worker static checks
327
+ worker_path = ROOT / "modal_workers" / "klein_image.py"
328
+ assert_true(worker_path.exists(), "Modal worker file should exist")
329
+ worker_source = worker_path.read_text(encoding="utf-8")
330
+ assert_true("modal.App" in worker_source, "Modal worker should create a Modal App")
331
+ assert_true("modal.Volume" in worker_source, "Modal worker should use a Modal Volume")
332
+ assert_true("FLUX.2-klein-4B" in worker_source, "Modal worker should reference the Klein model")
333
+ assert_true("modal.asgi_app" in worker_source, "Modal worker should expose one ASGI app")
334
+ assert_true('@api.post("/generate")' in worker_source, "Modal worker should expose POST /generate")
335
+ assert_true('@api.get("/health")' in worker_source, "Modal worker should expose GET /health")
336
+ assert_true('@api.get("/media/{filename}")' in worker_source, "Modal worker should expose GET /media/{filename}")
337
+ assert_true("404" in worker_source or "not found" in worker_source.lower(), "Modal worker should handle missing media with 404")
338
+ assert_true("reload" in worker_source, "Modal worker should reload volume before serving")
339
+
340
+ # Test: runtime status includes Modal Klein path
341
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", ""):
342
+ status = app._runtime_status_core()
343
+ assert_true("image_generation" in status, "Runtime status should include image_generation")
344
+ assert_true(
345
+ status["image_generation"]["status"] in {"online", "fallback-ready"},
346
+ "Image generation status should be online or fallback-ready",
347
+ )
348
+ assert_true(
349
+ status["image_generation"]["model"] == app.MODEL_MANIFEST["image_generation"]["id"],
350
+ "Image generation status should reference the Klein model",
351
+ )
352
+
353
+ # Test: invalid health payload is fallback-ready, not online
354
+ bad_health = MagicMock()
355
+ bad_health.read.return_value = json.dumps({"ok": True, "model": "wrong", "runtime": "modal-klein"}).encode("utf-8")
356
+ bad_health.__enter__ = lambda s: s
357
+ bad_health.__exit__ = MagicMock(return_value=False)
358
+ with patch.object(app, "KLEIN_MODAL_ENDPOINT", "https://example-modal.modal.run"):
359
+ with patch("urllib.request.urlopen", return_value=bad_health):
360
+ status = app._runtime_status_core()
361
+ assert_true(
362
+ status["image_generation"]["status"] == "fallback-ready",
363
+ "Invalid Modal health should not be marked online",
364
+ )
365
+ assert_true("ValueError" in status["image_generation"].get("warning", ""), "Invalid health should report validation warning")
366
+
367
+ # Test: runtime setup includes Modal deploy command
368
+ setup = app.runtime_setup_core()
369
+ image_step = next(step for step in setup["steps"] if step["role"] == "image_generation")
370
+ assert_true(
371
+ "modal deploy" in image_step["command"],
372
+ "Runtime setup should include modal deploy command for image generation",
373
+ )
374
+ assert_true(
375
+ "KLEIN_MODAL_ENDPOINT" in image_step["env"],
376
+ "Runtime setup should document KLEIN_MODAL_ENDPOINT",
377
+ )
378
+
379
+
380
+ def verify_minicpm_vision_integration() -> None:
381
+ with patch.object(app, "MINICPM_VISION_BASE_URL", ""), patch.object(app, "MINICPM_VISION_API_KEY", ""):
382
+ result = app.describe_image_core("model-map", caption=None, prompt=None)
383
+ assert_true(result["ok"], "No-config MiniCPM fallback should return ok")
384
+ assert_true(result["runtime"] == "fallback", "No-config MiniCPM should report fallback runtime")
385
+ assert_true(result["model"] == app.MODEL_MANIFEST["vision"]["id"], "No-config MiniCPM should report vision model")
386
+ assert_true(result.get("warning") is None, "No-config MiniCPM fallback should stay quiet")
387
+
388
+ mock_response_data = {
389
+ "choices": [
390
+ {
391
+ "message": {
392
+ "content": "A compact diagram shows four small AI models connected around an accessibility reader."
393
+ }
394
+ }
395
+ ]
396
+ }
397
+ mock_response = MagicMock()
398
+ mock_response.read.return_value = json.dumps(mock_response_data).encode("utf-8")
399
+ mock_response.__enter__ = lambda s: s
400
+ mock_response.__exit__ = MagicMock(return_value=False)
401
+
402
+ with patch.object(app, "MINICPM_VISION_BASE_URL", "https://vision.example/v1"):
403
+ with patch.object(app, "MINICPM_VISION_API_KEY", "secret-key"):
404
+ with patch("urllib.request.urlopen", return_value=mock_response) as urlopen_mock:
405
+ result = app.describe_image_core("model-map", caption="diagram", prompt="model map", image_url="/static/generated/model-map.svg")
406
+ assert_true(result["ok"], "Live MiniCPM response should return ok")
407
+ assert_true(result["runtime"] == "minicpm-v4.6", "Live MiniCPM response should report runtime")
408
+ assert_true(result["model"] == app.MINICPM_VISION_MODEL, "Live MiniCPM response should report configured model")
409
+ assert_true("accessibility reader" in result["alt_text"], "Live MiniCPM response should use returned alt text")
410
+ request = urlopen_mock.call_args.args[0]
411
+ assert_true(
412
+ request.full_url == "https://vision.example/v1/chat/completions",
413
+ "MiniCPM client should normalize base URL to chat completions",
414
+ )
415
+ assert_true(
416
+ request.headers.get("Authorization") == "Bearer secret-key",
417
+ "MiniCPM client should send bearer auth",
418
+ )
419
+
420
+ invalid_response = MagicMock()
421
+ invalid_response.read.return_value = json.dumps({"choices": [{"message": {"content": ""}}]}).encode("utf-8")
422
+ invalid_response.__enter__ = lambda s: s
423
+ invalid_response.__exit__ = MagicMock(return_value=False)
424
+
425
+ with patch.object(app, "MINICPM_VISION_BASE_URL", "https://vision.example"):
426
+ with patch.object(app, "MINICPM_VISION_API_KEY", "secret-key"):
427
+ with patch("urllib.request.urlopen", return_value=invalid_response):
428
+ result = app.describe_image_core("custom", caption="fallback caption", prompt=None, image_url="https://example.com/image.png")
429
+ assert_true(result["runtime"] == "fallback", "Invalid MiniCPM response should fall back")
430
+ assert_true(result["alt_text"] == "fallback caption", "Invalid MiniCPM response should preserve fallback text")
431
+ assert_true(
432
+ result.get("warning") is not None and "ValueError" in result["warning"],
433
+ "Invalid MiniCPM response should include validation warning",
434
+ )
435
+
436
+ models_response = MagicMock()
437
+ models_response.read.return_value = json.dumps({"data": [{"id": app.MINICPM_VISION_MODEL}]}).encode("utf-8")
438
+ models_response.__enter__ = lambda s: s
439
+ models_response.__exit__ = MagicMock(return_value=False)
440
+
441
+ with patch.object(app, "MINICPM_VISION_BASE_URL", "https://vision.example/v1"):
442
+ with patch.object(app, "MINICPM_VISION_API_KEY", "secret-key"):
443
+ with patch("urllib.request.urlopen", return_value=models_response):
444
+ status = app._vision_runtime_status()
445
+ assert_true(status["status"] == "online", "MiniCPM /models response should mark vision online")
446
+ assert_true("secret-key" not in json.dumps(status), "Vision runtime status should not expose API key")
447
+
448
+ setup = app.runtime_setup_core()
449
+ vision_step = next(step for step in setup["steps"] if step["role"] == "vision")
450
+ assert_true("MiniCPM-V-4.6" in vision_step["model"], "Runtime setup should document MiniCPM-V-4.6")
451
+ assert_true("MINICPM_VISION_BASE_URL" in vision_step["env"], "Runtime setup should document MiniCPM base URL")
452
+ assert_true("secret" not in json.dumps(vision_step).lower(), "Runtime setup should not expose MiniCPM API key")
453
+
454
+
455
  def verify_output_retention() -> None:
456
  keep_path = app.OUTPUT_DIR / "speech-retention-keep.wav"
457
  _ = app.speak_core("Tiny Narrator retention check.", voice="af_heart", speed=1.0)
 
752
  )
753
  assert_true(evidence_payload["runtime_status"]["ok"], "Evidence bundle should include ok runtime status")
754
  assert_true(
755
+ {"reader_brain", "speech", "image_generation"} <= set(evidence_payload["runtime_status"]),
756
+ "Evidence bundle runtime status should include reader brain, speech, and image generation status",
757
  )
758
  assert_true(
759
  evidence_payload["runtime_status"]["reader_brain"]["status"] in {"online", "fallback-ready"},
 
763
  evidence_payload["runtime_status"]["speech"]["status"] in {"online", "fallback-ready"},
764
  "Evidence bundle speech status should be online or fallback-ready",
765
  )
766
+ assert_true(
767
+ evidence_payload["runtime_status"]["image_generation"]["status"] in {"online", "fallback-ready"},
768
+ "Evidence bundle image generation status should be online or fallback-ready",
769
+ )
770
 
771
  runtime = client.get("/api/runtime-status")
772
  assert_true(runtime.status_code == 200, "Runtime status route should return 200")
 
774
  assert_true(runtime_payload["ok"], "Runtime status payload should be ok")
775
  assert_true("reader_brain" in runtime_payload, "Runtime status should include reader brain status")
776
  assert_true("speech" in runtime_payload, "Runtime status should include speech status")
777
+ assert_true("image_generation" in runtime_payload, "Runtime status should include image generation status")
778
  assert_true(
779
  runtime_payload["reader_brain"]["status"] in {"online", "fallback-ready"},
780
  "Reader brain status should be online or fallback-ready",
781
  )
782
+ assert_true(
783
+ runtime_payload["image_generation"]["status"] in {"online", "fallback-ready"},
784
+ "Image generation status should be online or fallback-ready",
785
+ )
786
+ assert_true(
787
+ runtime_payload["image_generation"]["model"] == app.MODEL_MANIFEST["image_generation"]["id"],
788
+ "Image generation status should reference the Klein model",
789
+ )
790
 
791
  image_descriptions = client.get("/api/image-descriptions")
792
  assert_true(image_descriptions.status_code == 200, "Image descriptions route should return 200")
793
  image_payload = image_descriptions.json()
794
+ assert_true(image_payload["model"] == app.MODEL_MANIFEST["vision"]["id"], "Image route should document MiniCPM-V model")
795
+ assert_true(image_payload["runtime"] == "fallback", "Image route should use fallback runtime without MiniCPM config")
796
  assert_true(
797
  {item["id"] for item in image_payload["descriptions"]} == {"desk-reader", "model-map"},
798
  "Image route should describe all visible article images",
 
806
  "Image route should label bundled image fallback status",
807
  )
808
 
809
+ # Verify generate-image endpoint returns fallback when Modal is not configured
810
+ gen_image = client.post("/api/generate-image", json={"prompt": "test thumbnail", "seed": 7})
811
+ assert_true(gen_image.status_code == 200, "Generate image route should return 200")
812
+ gen_image_payload = gen_image.json()
813
+ assert_true(gen_image_payload["ok"], "Generate image payload should be ok")
814
+ assert_true(gen_image_payload["model"] == app.MODEL_MANIFEST["image_generation"]["id"], "Generate image should report Klein model")
815
+ assert_true(gen_image_payload["runtime"] == "fallback", "Generate image without Modal should report fallback")
816
+ assert_true(isinstance(gen_image_payload["elapsed_ms"], int), "Generate image should include elapsed_ms")
817
+
818
 
819
  def main() -> None:
820
  py_compile.compile(str(ROOT / "app.py"), doraise=True)
821
+ py_compile.compile(str(ROOT / "modal_workers" / "klein_image.py"), doraise=True)
822
  verify_static_assets()
823
  verify_space_metadata()
824
  verify_core_fallbacks()
825
+ verify_modal_klein_integration()
826
+ verify_minicpm_vision_integration()
827
  verify_output_retention()
828
  verify_routes()
829
  print("Tiny Narrator verification passed.")