rupkotha_1st_commit

.gitignore

@@ -0,0 +1,46 @@
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+*.egg-info/
+*.png
+*.wav
+
+# Caches / logs
+.hf_cache/
+finetune/.hf_cache/
+*.log
+.claude
+.gradio/
+data/
+demo/
+docs/
+CLAUDE.md
+
+
+# Audio artifacts — regenerable demos and large recordings don't belong in git
+*.wav
+*.mp3
+*.flac
+
+# Large, reproducible training data (regenerate via finetune/collect.py + preprocess.py).
+# The distilled dataset finetune/data/train.json IS kept (the valuable artifact).
+finetune/data/images/
+finetune/data/processed/
+finetune/data/labelset/
+finetune/data/sample/
+finetune/data/train_review.md
+finetune/data/train_sample_review.md
+finetune/data/train_sample.json
+finetune/data/train_sample.json.rejected
+finetune/data/train.json.rejected
+
+# Eval / comparison outputs
+finetune/eval_results/
+finetune/quality_results/
+tests/
+tests/quality_results/
+reference_audio/
+
+# OS
+.DS_Store

README.md

@@ -1,14 +1,152 @@
 ---
 title: Rupkotha
-emoji: 🐨
-colorFrom: purple
-colorTo: indigo
+emoji: 🌙
+colorFrom: indigo
+colorTo: purple
 sdk: gradio
-sdk_version: 6.18.0
-python_version: '3.13'
+sdk_version: 6.17.3
 app_file: app.py
 pinned: false
-short_description: Submission for Build Small Hackathon
+short_description: Bedtime stories from kids' drawings, in English & Bengali
+tags:
+  - backyard-ai
+  - openbmb
+  - modal
+  - children
+  - storytelling
+  - bengali
+  - tts
+  - vision-language-model
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# রূপকথা · Rupkotha
+
+A bedtime-story app for kids. A child shows their drawings or toys, asks for a
+story in their own voice (English or Bengali), and hears it read back in a warm
+grandmother's voice. Built for the Build Small Hackathon (Track 1: Backyard AI).
+
+Inference runs on **Modal** (cloud GPUs) — the Gradio Space is a thin client with
+zero model weights.
+
+## 🏆 Build Small Hackathon — submission
+
+- **Track:** Practical · **Backyard AI** (a custom story generator — the track's own example use case).
+- **Targeting:** 🥇 **OpenBMB Best MiniCPM Build** · ⚡ **Modal Best Use**.
+- **🎬 Demo video:** https://youtu.be/mUUmy5JwBYo
+- **📣 Social post:** `TODO_ADD_SOCIAL_URL`
+
+**The idea.** Rupkotha (রূপকথা, "fairy tale") is a bedtime-story app for 4–9 year-olds
+who can't yet type. A child shows a crayon drawing or a toy, asks for a story **by
+voice** in **English or Bengali**, and hears it read aloud in a warm grandmother's
+voice — gentle, short, always ending in sleep.
+
+**The technical approach.**
+- **Vision → story:** `MiniCPM-V 4.5` (8B) reads 1–4 images + the request and writes
+  a 120–150-word bedtime fable.
+- **Native Bengali (our differentiator):** the stock model's Bengali was garbled, so
+  we **fine-tuned MiniCPM-V itself** — distilled ~389 native Bengali stories from a
+  Gemma teacher, LoRA-trained via ms-SWIFT, merged, and served on **vLLM**. A held-out
+  eval (Bengali speaker) confirmed it beats the base decisively.
+- **Voice:** `faster-whisper` for speech input; **VoxCPM2** (English) and **AI4Bharat
+  Indic-TTS** (Bengali) for the grandmother voice output.
+- **Infra:** every model runs on **Modal** (Ollama + vLLM + TTS containers, scale-to-zero
+  A10G/A100); the HF Space holds **zero weights** and just calls Modal functions. All
+  models are well under the 32B limit. Switching/serving is driven by one config object
+  (`core/model_config.py`).
+
+> **Running this Space:** it calls Modal at runtime — set `MODAL_TOKEN_ID` and
+> `MODAL_TOKEN_SECRET` as Space secrets, and deploy `core/modal_infra.py` +
+> `finetune/serve_vllm.py` first. First call after idle is a cold start (~1–5 min);
+> see the demo video for a smooth walkthrough.
+
+## Status
+
+Story (EN + BN), STT, and TTS run on Modal via `core/modal_infra.py`. The Bengali
+path is served by a **fine-tuned MiniCPM-V 4.5** (see below); English uses the
+stock model. Each `core/` function degrades gracefully to a safe fallback if a
+model is unavailable, so the app stays runnable. See `CLAUDE.md` for the full
+build order and remaining UI/polish items.
+
+## The stack
+
+The submission runs a single stack — **Stack A**, the OpenBMB prize path — defined in
+`core/model_config.py`. (The `StackConfig` machinery remains so a stack could be
+swapped in, but only Stack A is shipped.)
+
+| Layer | Model | ~Params |
+|---|---|---|
+| Vision + story | MiniCPM-V 4.5 (Bengali fine-tuned), via Ollama / vLLM | 8B |
+| STT | faster-whisper large-v3 | 1.55B |
+| English TTS | VoxCPM2 (Voice Design) | 2B |
+| Bengali TTS | AI4Bharat Indic-TTS (FastPitch + HiFi-GAN) | ~0.13B |
+
+All OpenBMB-family core models (MiniCPM-V + VoxCPM2) → eligible for the OpenBMB Best
+MiniCPM Build prize, strengthened by fine-tuning MiniCPM-V itself for Bengali.
+
+## Infrastructure
+
+All model inference runs on **Modal** — the Gradio Space holds zero model weights
+and makes zero local inference calls. Two Modal apps back the app:
+
+- **`rupkotha`** (`core/modal_infra.py`) — the base runtime: vision/story
+  (MiniCPM-V 4.5 via Ollama), STT (faster-whisper), TTS (VoxCPM2 for English,
+  AI4Bharat Indic-TTS for Bengali), and the IndicTrans2 translation path.
+- **`rupkotha-ft-serve`** (`finetune/serve_vllm.py`) — the **Bengali fine-tuned**
+  MiniCPM-V, served via **vLLM** on an A100-40GB (the full bf16 8B + vision
+  encoder needs more than a 24 GB card).
+
+The `core/` wrappers call these functions remotely; switching
+`COMPUTE_LOCATION` in `model_config.py` is the only change needed to run base
+inference locally (requires a GPU with 8+ GB VRAM).
+
+```bash
+uv run modal deploy core/modal_infra.py        # base runtime (EN vision, STT, TTS)
+uv run modal deploy finetune/serve_vllm.py     # Bengali fine-tuned model (vLLM)
+```
+
+## Bengali fine-tune — improving the OpenBMB model itself
+
+Stack A's one weak spot was native Bengali: stock MiniCPM-V 4.5 produced garbled,
+repetitive Bengali. Rather than swap in a different model, we **fine-tuned the
+OpenBMB model itself** to fix it — the strongest possible OpenBMB-prize story
+(*"we improved MiniCPM-V for Bengali"*). The whole pipeline lives in `finetune/`
+and runs on Modal:
+
+1. **Distill** native Bengali bedtime stories from a Gemma 3 teacher over ~450
+   children's drawings, filtered by a purity gate → **389 high-quality examples**.
+2. **LoRA fine-tune** MiniCPM-V 4.5 with **ms-SWIFT** on an A100-80GB — vision
+   encoder frozen, LoRA (r=16) on the LLM self-attention only (the weak-Bengali part).
+3. **Merge** the adapter into standalone weights and **serve via vLLM**.
+4. **Evaluate** FT vs the stock model on held-out drawings (`finetune/eval_ft.py`):
+   the fine-tune wins decisively — coherent native রূপকথা vs garbled, looping output
+   — confirmed by a Bengali speaker.
+
+Bengali story requests now route to the fine-tuned model
+(`FINETUNED_VISION_MODEL` in `model_config.py`, one-line revert to `None`); English
+and audio paths are unchanged. See `finetune/README.md` for the full pipeline.
+
+## Prizes targeted
+
+| Prize | Why eligible |
+|---|---|
+| Backyard AI track | Storybook generators are the track's example use case |
+| OpenBMB Best MiniCPM Build | Stack A is all-OpenBMB, **and we fine-tuned MiniCPM-V 4.5 itself** for native Bengali |
+| Modal Best Use | All inference on Modal; two apps — base runtime + a vLLM-served fine-tuned model, plus LoRA training on A100 |
+
+## Environment & setup
+
+This project uses **[uv](https://docs.astral.sh/uv/)** for all package and
+environment management (Python 3.10–3.12; VoxCPM2 requires < 3.13).
+
+```bash
+uv venv --python 3.12      # create the environment
+uv sync                    # install locked dependencies
+uv run python app.py       # launch the Gradio app
+```
+
+Add dependencies with `uv add <pkg>` (never `pip`). `requirements.txt` is
+generated from the lockfile (`uv export`), not hand-edited.
+
+## Layout
+
+See `CLAUDE.md` §3 for the full repository layout and §6 for the build order.

app.py

@@ -0,0 +1,232 @@
+# app.py — Gradio Blocks entry point. UI + wiring only. ZERO model references.
+"""Rupkotha (রূপকথা) — a bedtime-story app for kids.
+
+This file orchestrates the UI and chains core functions:
+    transcribe() → generate_story() → speak()
+It must contain no model names, paths, or model logic — those live only in core/.
+
+Layout: a two-panel "studio" — a Create panel (language/style, pictures, ask) and a
+Story panel (text + audio + save) — over a night-sky theme. Session memory uses
+gr.State, never browser storage (CLAUDE.md §11).
+"""
+
+from pathlib import Path
+
+import gradio as gr
+
+from core.vision_story import generate_story
+from core.stt import transcribe
+from core.tts import speak
+from core.prompts import STYLES
+
+# Language radio: display label → internal code passed to core functions.
+_LANGUAGES = [("English", "en"), ("বাংলা", "bn")]
+_STYLE_CHOICES = {lang: list(styles.keys()) for lang, styles in STYLES.items()}
+
+_CSS_PATH = Path(__file__).parent / "assets" / "styles.css"
+
+HISTORY_SIZE = 3  # how many recent stories to keep (CLAUDE.md §11: last 3)
+
+
+def _styles_for(language: str):
+    """Return a style-dropdown update for the chosen language."""
+    choices = _STYLE_CHOICES.get(language, _STYLE_CHOICES["en"])
+    return gr.update(choices=choices, value=choices[0])
+
+
+def _preview(files):
+    """Show uploaded images in the preview gallery; hide it when empty."""
+    files = files or []
+    return gr.update(value=files, visible=bool(files))
+
+
+def _voice_to_text(audio_path, language):
+    """Transcribe a mic recording into the instruction box. On empty/failed
+    transcription, leave whatever the child already typed untouched."""
+    print(f"[mic] stop_recording fired · audio={audio_path!r} · lang={language}", flush=True)
+    text = transcribe(audio_path, language)
+    print(f"[mic] transcribe -> {text!r}", flush=True)
+    return text if text else gr.update()
+
+
+def _tell_a_story(images, instruction, language, style, child_name):
+    """Chain: images + instruction → story text → grandmother-voice audio.
+
+    Each core call degrades gracefully (never raises), so the UI always shows
+    a story even if Modal is unreachable or audio synthesis fails. Also returns
+    a `current` dict so the Save button can capture the exact result shown.
+    """
+    image_paths = [img for img in (images or [])]
+    story, model_label = generate_story(
+        image_paths=image_paths,
+        instruction=instruction or "",
+        language=language,
+        style=style,
+        child_name=child_name or "",
+    )
+    wav_path, tts_label = speak(story, language)
+    badge = f"📖 {model_label}　·　🔊 {tts_label}"
+    current = {"story": story, "audio": wav_path, "badge": badge}
+    return story, wav_path, badge, current
+
+
+def _history_updates(history):
+    """Flatten `history` into per-slot updates: (group, markdown, audio) × N."""
+    updates = []
+    for i in range(HISTORY_SIZE):
+        if i < len(history):
+            entry = history[i]
+            body = f"{entry['story']}\n\n<span class='saved-badge'>{entry['badge']}</span>"
+            updates += [
+                gr.update(visible=True),
+                gr.update(value=body),
+                gr.update(value=entry.get("audio")),
+            ]
+        else:
+            updates += [
+                gr.update(visible=False),
+                gr.update(value=""),
+                gr.update(value=None),
+            ]
+    return updates
+
+
+def _save_story(current, history):
+    """Prepend the current story to the session history (newest first, max N)."""
+    history = list(history or [])
+    if current and current.get("story"):
+        history = ([current] + history)[:HISTORY_SIZE]
+    return [history, *_history_updates(history)]
+
+
+def build_ui() -> gr.Blocks:
+    theme = gr.themes.Soft(
+        primary_hue="amber",
+        secondary_hue="orange",
+        neutral_hue="slate",
+        radius_size="lg",
+        font=[gr.themes.GoogleFont("Nunito"), "ui-sans-serif", "sans-serif"],
+    )
+    css_kw = {"css_paths": [str(_CSS_PATH)]} if _CSS_PATH.exists() else {}
+    with gr.Blocks(title="রূপকথা · Rupkotha", theme=theme, fill_width=True, **css_kw) as demo:
+        # ── Hero ─────────────────────────────────────────────────────────
+        gr.HTML(
+            """
+            <div id="hero">
+              <div class="hero-moon">🌙</div>
+              <h1>রূপকথা · Rupkotha</h1>
+              <p>Show a picture, ask for a story — and hear it told in a warm
+                 grandmother's voice.</p>
+            </div>
+            """
+        )
+
+        with gr.Row(elem_id="studio", equal_height=False):
+            # ── Create panel ─────────────────────────────────────────────
+            with gr.Column(scale=5, elem_classes="panel"):
+                gr.HTML('<div class="panel-head"><span class="step">1</span>Choose</div>')
+                with gr.Row():
+                    language = gr.Radio(
+                        choices=_LANGUAGES, value="en",
+                        label="Language · ভাষা", elem_classes="seg",
+                    )
+                    style = gr.Dropdown(
+                        choices=_STYLE_CHOICES["en"], value=_STYLE_CHOICES["en"][0],
+                        label="Story style",
+                    )
+
+                gr.HTML('<div class="panel-head"><span class="step">2</span>Show your pictures</div>')
+                images = gr.File(
+                    file_count="multiple",
+                    type="filepath",
+                    file_types=["image"],
+                    label="Drawings or toys — 1 to 4 pictures",
+                    elem_classes="upload-box",
+                )
+                preview = gr.Gallery(
+                    label="Your pictures",
+                    columns=4,
+                    height="auto",
+                    object_fit="contain",   # show the whole image, don't crop/trim
+                    show_label=True,
+                    visible=False,
+                    elem_classes="preview",
+                )
+
+                gr.HTML('<div class="panel-head"><span class="step">3</span>Ask for a story</div>')
+                mic = gr.Audio(
+                    sources=["microphone"],
+                    type="filepath",
+                    label="🎤 Speak your request (optional) — it fills the box below",
+                )
+                instruction = gr.Textbox(
+                    label="What story do you want?",
+                    placeholder="tell me a story about my dragon…",
+                    lines=2,
+                )
+                child_name = gr.Textbox(
+                    label="Your name (optional)",
+                    placeholder="e.g. Rupa — woven into the story",
+                    lines=1,
+                )
+                generate_btn = gr.Button(
+                    "✨ Tell me a story", variant="primary", size="lg",
+                    elem_id="generate-btn",
+                )
+
+            # ── Story panel ──────────────────────────────────────────────
+            with gr.Column(scale=6, elem_classes="panel story-panel"):
+                gr.HTML('<div class="panel-head">📖 Your story</div>')
+                story_out = gr.Textbox(
+                    show_label=False,
+                    lines=8,
+                    max_lines=40,   # grow to fit the whole story (no inner scrollbar)
+                    autoscroll=False,
+                    placeholder="Your bedtime story will appear here…  ✨",
+                    elem_classes="story-text",
+                    container=False,
+                )
+                audio_out = gr.Audio(label="🔊 Listen (press play to replay)", type="filepath")
+                badge_out = gr.Markdown(elem_classes="model-badge")
+                save_btn = gr.Button("💾 Save this story", elem_id="save-btn")
+
+        # ── Saved stories: last 3, each replayable (gr.State session memory) ─
+        current = gr.State(None)
+        history = gr.State([])
+        gr.HTML('<div class="section-title">🌟 Your saved stories</div>')
+        slots = []
+        with gr.Row(elem_id="history-row", equal_height=False):
+            for _ in range(HISTORY_SIZE):
+                with gr.Column(scale=1, min_width=240):
+                    with gr.Group(visible=False, elem_classes="saved-card") as slot_group:
+                        slot_md = gr.Markdown(elem_classes="saved-text")
+                        slot_audio = gr.Audio(type="filepath", label="Replay")
+                slots.append((slot_group, slot_md, slot_audio))
+
+        # ── Wiring ───────────────────────────────────────────────────────
+        language.change(_styles_for, inputs=language, outputs=style)
+
+        # Show thumbnails of the uploaded pictures.
+        images.change(_preview, inputs=images, outputs=preview)
+
+        # Voice is a bonus: it fills the typed box, which stays primary (§2, §14).
+        mic.stop_recording(_voice_to_text, inputs=[mic, language], outputs=instruction)
+
+        generate_btn.click(
+            _tell_a_story,
+            inputs=[images, instruction, language, style, child_name],
+            outputs=[story_out, audio_out, badge_out, current],
+        )
+
+        # Flatten slots for the Save outputs: history + (group, md, audio) × N.
+        slot_outputs = [comp for slot in slots for comp in slot]
+        save_btn.click(
+            _save_story,
+            inputs=[current, history],
+            outputs=[history, *slot_outputs],
+        )
+    return demo
+
+
+if __name__ == "__main__":
+    build_ui().queue().launch()

assets/styles.css

@@ -0,0 +1,211 @@
+/* Rupkotha — calm "storybook" theme. Professional, low-contrast, presentable.
+   Soft parchment/lavender background, deep slate-indigo ink, muted terracotta +
+   dusty-gold accents. Warm and gentle — not high-contrast. */
+
+@import url('https://fonts.googleapis.com/css2?family=Baloo+2:wght@500;600;700&family=Noto+Sans+Bengali:wght@500;600;700&family=Nunito:wght@400;500;600;700;800&display=swap');
+
+:root {
+  --bg-1: #eef0f7;          /* soft lavender mist */
+  --bg-2: #f7f2ea;          /* warm cream */
+  --ink: #333a5c;           /* deep slate-indigo (headings) */
+  --ink-body: #4a5170;      /* body text */
+  --ink-soft: #7b81a0;      /* secondary/placeholder */
+  --panel: #ffffff;
+  --panel-2: #fbfaf6;
+  --panel-brd: rgba(51, 58, 92, 0.10);
+  --accent: #c4865a;        /* muted terracotta */
+  --accent-2: #d6a463;      /* dusty gold */
+  --accent-ink: #5a3a1e;
+  --story-bg: #fbf6ec;      /* parchment */
+  --story-ink: #43392b;
+  --radius: 20px;
+  --radius-sm: 13px;
+  --shadow: 0 12px 34px rgba(46, 52, 88, 0.10);
+  --shadow-sm: 0 4px 14px rgba(46, 52, 88, 0.08);
+}
+
+/* ── Background: soft, calm gradient (no harsh dark) ──────────────────── */
+.gradio-container,
+.gradio-container .fillable,
+body {
+  background:
+    radial-gradient(900px 480px at 85% -12%, rgba(214, 164, 99, 0.10), transparent 60%),
+    radial-gradient(820px 460px at 5% -6%, rgba(150, 160, 210, 0.12), transparent 58%),
+    linear-gradient(165deg, var(--bg-1) 0%, var(--bg-2) 70%) !important;
+  color: var(--ink-body) !important;
+  font-family: 'Nunito', ui-sans-serif, system-ui, sans-serif;
+}
+
+.gradio-container { max-width: 100% !important; margin: 0 !important; padding: 0 28px !important; }
+
+/* ── Hero ────────────────────────────────────────────────────────────── */
+#hero { text-align: center; padding: 1.7rem 1rem 0.4rem; }
+#hero .hero-moon {
+  font-size: 2.6rem; line-height: 1;
+  filter: drop-shadow(0 6px 14px rgba(196, 134, 90, 0.30));
+  animation: float 6s ease-in-out infinite;
+}
+@keyframes float { 0%,100%{ transform: translateY(0) } 50%{ transform: translateY(-6px) } }
+#hero h1 {
+  font-family: 'Baloo 2', 'Noto Sans Bengali', sans-serif;
+  font-size: 2.5rem; font-weight: 700; margin: .3rem 0 .25rem;
+  color: var(--ink);
+  letter-spacing: .2px;
+}
+#hero p { color: var(--ink-soft); font-size: 1.05rem; max-width: 560px; margin: 0 auto; }
+
+/* ── Panels (soft cards) ─────────────────────────────────────────────── */
+#studio { gap: 22px !important; margin-top: .8rem; align-items: stretch; }
+.panel {
+  background: var(--panel) !important;
+  border: 1px solid var(--panel-brd) !important;
+  border-radius: var(--radius) !important;
+  box-shadow: var(--shadow);
+  padding: 18px 20px !important;
+}
+.panel .block, .panel .form, .panel .gr-group, .panel fieldset {
+  background: transparent !important; border: none !important; box-shadow: none !important;
+}
+
+/* Section headers inside panels. */
+.panel-head {
+  display: flex; align-items: center; gap: .55rem;
+  font-family: 'Baloo 2', sans-serif; font-weight: 600;
+  font-size: 1.1rem; color: var(--ink);
+  margin: .7rem 0 .4rem;
+}
+.panel-head .step {
+  display: inline-grid; place-items: center;
+  width: 1.5rem; height: 1.5rem; border-radius: 50%;
+  background: linear-gradient(135deg, var(--accent-2), var(--accent));
+  color: #fff; font-size: .82rem; font-weight: 700;
+  box-shadow: var(--shadow-sm);
+}
+
+/* ── Labels & text ───────────────────────────────────────────────────── */
+.gradio-container label,
+.gradio-container .gr-text { color: var(--ink-body) !important; }
+.gradio-container label > span,
+.gradio-container .gr-form > span { font-weight: 600 !important; color: var(--ink) !important; }
+
+/* ── Inputs ──────────────────────────────────────────────────────────── */
+.gradio-container input[type="text"],
+.gradio-container textarea,
+.gradio-container .gr-dropdown,
+.gradio-container select {
+  background: var(--panel-2) !important;
+  border: 1px solid rgba(51, 58, 92, 0.14) !important;
+  border-radius: var(--radius-sm) !important;
+  color: var(--ink) !important;
+  font-size: 1.04rem !important;
+}
+.gradio-container input::placeholder,
+.gradio-container textarea::placeholder { color: var(--ink-soft) !important; }
+.gradio-container input:focus,
+.gradio-container textarea:focus {
+  border-color: var(--accent) !important;
+  box-shadow: 0 0 0 3px rgba(196, 134, 90, 0.16) !important;
+}
+
+/* Upload drop zone. */
+.upload-box, .upload-box .center, .upload-box .wrap {
+  border-radius: var(--radius-sm) !important;
+  border: 1.5px dashed rgba(196, 134, 90, 0.45) !important;
+  background: rgba(214, 164, 99, 0.06) !important;
+  color: var(--ink-body) !important;
+}
+
+/* ── Image preview gallery ───────────────────────────────────────────── */
+.preview {
+  border-radius: var(--radius-sm) !important;
+  border: 1px solid var(--panel-brd) !important;
+  background: var(--panel-2) !important;
+  padding: 6px !important; margin-top: .4rem;
+}
+.preview .thumbnail-item, .preview img, .preview button.thumbnail-item {
+  border-radius: 10px !important;
+  box-shadow: var(--shadow-sm);
+}
+
+/* ── Buttons ─────────────────────────────────────────────────────────── */
+#generate-btn {
+  font-family: 'Baloo 2', sans-serif !important;
+  font-size: 1.3rem !important; font-weight: 700 !important;
+  padding: .85rem 1.5rem !important; margin-top: .9rem !important; width: 100% !important;
+  background: linear-gradient(135deg, var(--accent-2), var(--accent)) !important;
+  color: #fff !important; border: none !important; border-radius: 999px !important;
+  box-shadow: 0 8px 22px rgba(196, 134, 90, 0.32) !important;
+  transition: transform .15s ease, box-shadow .15s ease !important;
+}
+#generate-btn:hover { transform: translateY(-2px); box-shadow: 0 12px 28px rgba(196,134,90,.40) !important; }
+#generate-btn:active { transform: translateY(0); }
+#save-btn {
+  font-weight: 700 !important; border-radius: 999px !important;
+  border: 1.5px solid var(--accent) !important; color: var(--accent) !important;
+  background: transparent !important; margin-top: .5rem !important;
+}
+#save-btn:hover { background: rgba(196, 134, 90, 0.10) !important; }
+
+/* ── Story panel ─────────────────────────────────────────────────────── */
+.story-panel { display: flex; flex-direction: column; }
+.story-text textarea {
+  font-family: 'Baloo 2', 'Noto Sans Bengali', serif !important;
+  font-size: 1.32rem !important; line-height: 1.85 !important;
+  background: var(--story-bg) !important;
+  color: var(--story-ink) !important;
+  border: 1px solid rgba(196, 134, 90, 0.18) !important;
+  border-radius: var(--radius-sm) !important;
+  padding: 1.1rem 1.3rem !important;
+  min-height: 180px;        /* small floor for the empty state; grows to fit content */
+  height: auto !important;
+  overflow-y: auto;
+}
+
+/* ── Model badge ─────────────────────────────────────────────────────── */
+.model-badge p {
+  display: inline-block; margin: .3rem 0 0 !important;
+  background: rgba(214, 164, 99, 0.12);
+  border: 1px solid rgba(196, 134, 90, 0.30);
+  border-radius: 999px; padding: .35rem 1rem;
+  font-size: .82rem; color: var(--accent-ink) !important;
+}
+
+/* ── Saved stories ───────────────────────────────────────────────────── */
+.section-title {
+  font-family: 'Baloo 2', sans-serif; font-weight: 600; font-size: 1.28rem;
+  color: var(--ink); text-align: center; margin: 1.8rem 0 .7rem;
+}
+#history-row { gap: 16px !important; }
+.saved-card {
+  background: var(--panel) !important;
+  border: 1px solid var(--panel-brd) !important;
+  border-radius: var(--radius) !important;
+  box-shadow: var(--shadow-sm); padding: 14px 16px !important;
+}
+.saved-text, .saved-text p {
+  font-family: 'Baloo 2', 'Noto Sans Bengali', sans-serif;
+  font-size: 1rem !important; line-height: 1.62; color: var(--story-ink) !important;
+}
+.saved-badge { display: inline-block; margin-top: .5rem; font-size: .72rem; color: var(--accent); }
+
+#stack-note {
+  text-align: center; color: var(--ink-soft);
+  font-size: .9rem; margin: 1.8rem 0 1rem;
+}
+#stack-note b { color: var(--accent); }
+
+/* ── Accessibility ───────────────────────────────────────────────────── */
+.gradio-container *:focus-visible {
+  outline: 3px solid var(--accent) !important; outline-offset: 2px;
+}
+@media (prefers-reduced-motion: reduce) {
+  #hero .hero-moon { animation: none !important; }
+  .gradio-container * { animation: none !important; transition: none !important; }
+}
+
+/* ── Mobile ──────────────────────────────────────────────────────────── */
+@media (max-width: 820px) {
+  #studio { flex-direction: column !important; }
+  #hero h1 { font-size: 2rem; }
+  .story-text textarea { font-size: 1.22rem !important; min-height: 240px; }
+}

core/modal_infra.py

@@ -0,0 +1,785 @@
+"""
+core/modal_infra.py — ALL Modal GPU functions live here.
+
+app.py and the other core/ wrappers call these remote functions.
+Nothing outside this file should import torch, transformers, parler_tts,
+faster_whisper, or any other ML library directly.
+
+GPU tier and container settings are read from model_config.get_compute()
+so that changing MODAL_GPU or MODAL_MIN_CONTAINERS in model_config.py
+propagates here automatically.
+"""
+
+import base64
+
+import modal
+from core.model_config import get_compute, get_config
+from core.prompts import SCENE_SENTINEL
+
+_compute = get_compute()
+
+app = modal.App("rupkotha")
+
+# Vision image: Ollama serves every stack's vision model behind one uniform API
+# (MiniCPM-V for Stack A, Gemma 3 for Stacks B/C). Switching stacks only changes
+# the model tag passed in from model_config.py — never this code.
+_ollama_image = (
+    modal.Image.debian_slim(python_version="3.12")
+    # zstd is required by the Ollama install script to extract its release archive.
+    .apt_install("curl", "zstd")
+    .run_commands("curl -fsSL https://ollama.com/install.sh | sh")
+    .pip_install("ollama>=0.3")
+    # This module imports core.model_config at load; current Modal no longer
+    # auto-mounts the project, so make `core` importable in the container.
+    .add_local_python_source("core")
+)
+
+# Pulled Ollama models persist here across containers, so a model is downloaded
+# at most once per stack (not on every cold start).
+_ollama_volume = modal.Volume.from_name("rupkotha-ollama", create_if_missing=True)
+
+# HuggingFace weights (TTS/STT) persist here so they download at most once.
+_hf_volume = modal.Volume.from_name("rupkotha-hf", create_if_missing=True)
+_HF_CACHE = "/root/.cache/huggingface"
+
+# HF auth for gated repos (e.g. ai4bharat/indic-parler-tts). Reuses the existing
+# 'algaeguard-secrets' Modal secret, which provides HF_TOKEN — huggingface_hub /
+# transformers read HF_TOKEN from the environment automatically.
+_hf_secrets = [modal.Secret.from_name("algaeguard-secrets")]
+
+# Warm-container model caches for the TTS/STT singletons. Populated lazily inside
+# their respective functions; reused across calls in the same container.
+_indic_parler: dict = {}
+_voxcpm: dict = {}
+_whisper: dict = {}
+_indictrans: dict = {}
+_indictts: dict = {}
+
+# ML image: used by the (still-stubbed) TTS/STT functions — transformers,
+# faster-whisper, parler-tts. Kept separate from the vision image so each
+# function pulls only what it needs and cold-starts faster.
+_ml_image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .apt_install("git")  # needed for the git-based parler-tts install below
+    .pip_install(
+        "torch>=2.2",
+        "transformers>=4.40",
+        "faster-whisper>=1.0",
+        "voxcpm",  # English TTS — VoxCPM2 (needs Python < 3.13; 3.12 here is fine)
+        "indictranstoolkit",  # IndicTrans2 pre/post-processing for the Bengali pivot
+        "soundfile",
+        "numpy",
+    )
+    .run_commands(
+        "pip install git+https://github.com/huggingface/parler-tts.git"
+    )
+    .add_local_python_source("core")  # see note on _ollama_image above
+)
+
+# AI4Bharat Indic-TTS (FastPitch + HiFi-GAN) image. AI4Bharat's repo isn't a Python
+# package — inference just uses a Coqui `Synthesizer` over their checkpoints. We use
+# the maintained `coqui-tts` fork (Synthesizer API unchanged) and skip their heavy
+# Indic text-normalization/denoiser layer (fragile pinned deps; unneeded for our
+# clean Bengali-script input). Python 3.10 for best coqui-tts/checkpoint compat.
+_indictts_image = (
+    modal.Image.debian_slim(python_version="3.10")
+    .apt_install("wget", "unzip", "libsndfile1", "espeak-ng")
+    # coqui-tts 0.27 imports transformers.pytorch_utils.isin_mps_friendly, removed in
+    # transformers 5.x — pin to 4.x so `import TTS` (pulls XTTS/tortoise) works.
+    .pip_install("coqui-tts[codec]", "transformers<5", "torch", "torchaudio", "soundfile", "numpy")
+    .add_local_python_source("core")
+)
+# Persists the ~1.5 GB Bengali checkpoint zip (downloaded + unzipped once).
+_indictts_volume = modal.Volume.from_name("rupkotha-indictts", create_if_missing=True)
+
+_gpu = _compute["gpu"]
+_min_containers = _compute["min_containers"]
+
+
+def _ensure_ollama_server() -> None:
+    """Start `ollama serve` in the background if it isn't already responding.
+
+    Idempotent: safe to call on every invocation. Runs only inside the Modal
+    container (never in the local Gradio process)."""
+    import subprocess
+    import time
+    import urllib.request
+
+    def _ready() -> bool:
+        try:
+            urllib.request.urlopen("http://127.0.0.1:11434/api/version", timeout=1)
+            return True
+        except Exception:
+            return False
+
+    if _ready():
+        return
+    subprocess.Popen(
+        ["ollama", "serve"],
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+    )
+    for _ in range(120):  # up to ~60s for the server to come up
+        if _ready():
+            return
+        time.sleep(0.5)
+    raise RuntimeError("ollama server did not become ready in time")
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Vision + story generation
+# ─────────────────────────────────────────────────────────────────────────────
+
+@app.function(
+    gpu=_gpu,
+    image=_ollama_image,
+    volumes={"/root/.ollama": _ollama_volume},
+    timeout=900,  # generous: covers a first-time model pull on a cold container
+    min_containers=_min_containers,
+)
+def run_vision_story(
+    image_bytes_list: list[bytes],
+    prompt: str,
+    model_id: str,
+    options: dict | None = None,
+    describe_prompt: str | None = None,
+) -> str:
+    """Generate a bedtime story from images + prompt via Ollama.
+
+    Model-agnostic: the call serves whatever Ollama tag `model_id` names — Stack A
+    uses MiniCPM-V 4.5. (Bengali is served separately by the fine-tuned model via
+    finetune/serve_vllm.py; this Ollama path handles English.)
+
+    Args:
+        image_bytes_list: Raw bytes of each uploaded image (1–4 images).
+        prompt: Full story-generation prompt (built by core/prompts.py).
+        model_id: Ollama model tag from get_config().vision_model
+            (e.g. 'openbmb/minicpm-v4.5').
+        options: Ollama decoding params (temperature, top_p, repeat_penalty, …).
+            Per-language profiles come from model_config.get_vision_options().
+        describe_prompt: Lever C (two-pass). When set, the model first DESCRIBES
+            the image(s) in English using this prompt, then narrates the story
+            text-only — `prompt` must contain SCENE_SENTINEL, which is replaced
+            with that description. Keeps perception and Bengali prose separate.
+
+    Returns:
+        Generated story text, or '' on failure.
+    """
+    import time
+
+    import ollama
+
+    _ensure_ollama_server()
+
+    # Pull on first use; the Volume keeps the weights warm for later calls.
+    # NOTE: the Stack A tag 'openbmb/minicpm-v4.5' is confirmed valid in the
+    # Ollama registry.
+    listed = ollama.list()
+    models = getattr(listed, "models", None) or listed.get("models", []) or []
+    local_tags = {(getattr(m, "model", None) or m.get("model", "")) for m in models}
+    if model_id not in local_tags and f"{model_id}:latest" not in local_tags:
+        print(f"[run_vision_story] pulling {model_id} (first use, may take minutes) ...")
+        ollama.pull(model_id)
+        _ollama_volume.commit()
+        print(f"[run_vision_story] pull complete for {model_id}")
+
+    chat_options = options or {"temperature": 0.8}
+
+    def _chat_once(prompt_text: str, images: list[bytes]) -> tuple[str, object]:
+        # The ollama client base64-encodes raw image bytes itself.
+        kwargs = dict(
+            model=model_id,
+            messages=[
+                {"role": "user", "content": prompt_text, "images": list(images or [])}
+            ],
+            options=chat_options,
+        )
+        # think=False disables MiniCPM-V's reasoning mode, which can otherwise
+        # consume the whole token budget (done_reason='length') and leave the
+        # answer 'content' empty. Fall back for older ollama clients.
+        try:
+            resp = ollama.chat(**kwargs, think=False)
+        except TypeError:
+            resp = ollama.chat(**kwargs)
+        # ollama returns a typed ChatResponse (subscriptable) or a plain dict.
+        try:
+            c = resp["message"]["content"]
+        except Exception:
+            c = getattr(getattr(resp, "message", None), "content", "") or ""
+        return (c or "").strip(), resp
+
+    def _chat_retry(prompt_text: str, images: list[bytes]) -> str:
+        # A freshly started server sometimes returns empty on the very first call
+        # while the model finishes loading into VRAM — retry a couple of times.
+        text, resp = _chat_once(prompt_text, images)
+        for attempt in range(2):
+            if text:
+                break
+            print(f"[run_vision_story] empty content (attempt {attempt + 1}); resp={repr(resp)[:200]}")
+            time.sleep(2)
+            text, resp = _chat_once(prompt_text, images)
+        return text
+
+    if describe_prompt:
+        # Lever C, pass 1: describe the image(s) in English (the model's strength).
+        description = _chat_retry(describe_prompt, image_bytes_list)
+        print(f"[run_vision_story] scene description: {description[:200]}")
+        # Pass 2: narrate from the description, text-only (no image attached).
+        story_prompt = prompt.replace(SCENE_SENTINEL, description)
+        return _chat_retry(story_prompt, [])
+
+    # Single pass: image(s) + prompt together.
+    return _chat_retry(prompt, image_bytes_list)
+
+
+def generate_story_remote(
+    images_b64: list[str],
+    prompt: str,
+    options: dict | None = None,
+    describe_prompt: str | None = None,
+) -> str:
+    """Plain-callable entry point used by core/vision_story.py.
+
+    Decodes the base64 images, reads the active vision model tag from
+    get_config() internally (model names never leave model_config.py), and
+    dispatches to the deployed Modal `run_vision_story` function. `options`
+    carries the per-language decoding profile from get_vision_options();
+    `describe_prompt` enables two-pass (Lever C) for Bengali.
+
+    Requires the Modal app to be deployed (`modal deploy core/modal_infra.py`)
+    and Modal credentials available to the Gradio process. May raise — callers
+    in core/vision_story.py wrap this in try/except and fall back to a friendly
+    bedtime message, so the app stays runnable even if Modal is unreachable.
+    """
+    image_bytes_list = [base64.b64decode(b) for b in (images_b64 or [])]
+    model_id = get_config().vision_model
+    fn = modal.Function.from_name("rupkotha", "run_vision_story")
+    return fn.remote(image_bytes_list, prompt, model_id, options, describe_prompt)
+
+
+def generate_story_ft_remote(images_b64: list[str], prompt: str) -> str:
+    """Plain-callable entry point for the Bengali-fine-tuned model, deployed
+    separately as the `rupkotha-ft-serve` app (finetune/serve_vllm.py — merged
+    LoRA served via vLLM). Used by core/vision_story.py only when
+    model_config.FINETUNED_VISION_MODEL is set. Mirrors generate_story_remote's
+    contract; may raise — callers wrap in try/except. Kept here (not imported from
+    finetune/) so core/ stays independent of the training package."""
+    image_bytes_list = [base64.b64decode(b) for b in (images_b64 or [])]
+    fn = modal.Function.from_name("rupkotha-ft-serve", "run_vision_story_ft")
+    return fn.remote(image_bytes_list, prompt)
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Translation — IndicTrans2 (English → Bengali "pivot" path)
+# ─────────────────────────────────────────────────────────────────────────────
+
+@app.function(
+    gpu=_gpu,
+    image=_ml_image,
+    volumes={_HF_CACHE: _hf_volume},
+    secrets=_hf_secrets,
+    timeout=600,  # generous: covers a first-time weight download on a cold start
+    min_containers=_min_containers,
+)
+def run_translate(text: str, src_lang: str, tgt_lang: str, model_repo: str) -> str:
+    """Translate text with IndicTrans2 (e.g. English → Bengali).
+
+    Args:
+        text: Source text (may be multiple sentences).
+        src_lang / tgt_lang: IndicTrans2 FLORES codes ('eng_Latn', 'ben_Beng').
+        model_repo: HF repo from get_config-side TRANSLATION_MODEL.
+
+    Returns:
+        Translated text, or '' on failure.
+    """
+    import re
+
+    import torch
+    from IndicTransToolkit import IndicProcessor
+    from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+    if not (text or "").strip():
+        return ""
+
+    # Lazy singleton — load once per warm container.
+    if "model" not in _indictrans:
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        tokenizer = AutoTokenizer.from_pretrained(model_repo, trust_remote_code=True)
+        model = AutoModelForSeq2SeqLM.from_pretrained(
+            model_repo, trust_remote_code=True
+        ).to(device)
+        _indictrans.update(
+            model=model, tokenizer=tokenizer, ip=IndicProcessor(inference=True), device=device
+        )
+        _hf_volume.commit()
+
+    model = _indictrans["model"]
+    tokenizer = _indictrans["tokenizer"]
+    ip = _indictrans["ip"]
+    device = _indictrans["device"]
+
+    # IndicTrans2 translates sentence-by-sentence; split the story first.
+    sentences = [s.strip() for s in re.split(r"(?<=[.!?।])\s+", text.strip()) if s.strip()]
+    if not sentences:
+        return ""
+
+    batch = ip.preprocess_batch(sentences, src_lang=src_lang, tgt_lang=tgt_lang)
+    enc = tokenizer(
+        batch, padding="longest", truncation=True, max_length=256, return_tensors="pt"
+    ).to(device)
+    with torch.inference_mode():
+        out = model.generate(**enc, num_beams=5, num_return_sequences=1, max_length=256)
+    decoded = tokenizer.batch_decode(
+        out, skip_special_tokens=True, clean_up_tokenization_spaces=True
+    )
+    translated = ip.postprocess_batch(decoded, lang=tgt_lang)
+    return " ".join(t.strip() for t in translated if t and t.strip())
+
+
+def translate_remote(text: str, src_language: str, tgt_language: str) -> str:
+    """Plain-callable entry point used by core/vision_story.py for the Bengali
+    translation pivot. Resolves the model + FLORES codes from model_config and
+    dispatches to the deployed Modal `run_translate`. May raise — caller falls back.
+    """
+    from core.model_config import TRANSLATION_MODEL, get_indictrans_code
+
+    src = get_indictrans_code(src_language)
+    tgt = get_indictrans_code(tgt_language)
+    fn = modal.Function.from_name("rupkotha", "run_translate")
+    return fn.remote(text, src, tgt, TRANSLATION_MODEL)
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Bengali TTS — Indic Parler-TTS
+# ─────────────────────────────────────────────────────────────────────────────
+
+@app.function(
+    gpu=_gpu,
+    image=_ml_image,
+    volumes={_HF_CACHE: _hf_volume},
+    secrets=_hf_secrets,
+    timeout=600,  # generous: covers a cold-start model download + load
+    min_containers=_min_containers,
+)
+def run_tts_bengali(text: str, caption: str, model_repo: str) -> bytes:
+    """Synthesise Bengali speech using Indic Parler-TTS.
+
+    Args:
+        text: Story text in Bengali.
+        caption: Voice-description prompt that controls the speaker persona.
+        model_repo: HuggingFace repo ID from get_tts_repo() (e.g.
+            'ai4bharat/indic-parler-tts').
+
+    Returns:
+        WAV audio as bytes, or b'' on failure.
+    """
+    import io
+    import re
+
+    import numpy as np
+    import soundfile as sf
+    import torch
+    from parler_tts import ParlerTTSForConditionalGeneration
+    from transformers import AutoTokenizer, set_seed
+
+    # Lazy singleton — load once per warm container, reuse across calls.
+    if "model" not in _indic_parler:
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        model = ParlerTTSForConditionalGeneration.from_pretrained(model_repo).to(device)
+        tokenizer = AutoTokenizer.from_pretrained(model_repo)
+        desc_tokenizer = AutoTokenizer.from_pretrained(
+            model.config.text_encoder._name_or_path
+        )
+        _indic_parler.update(
+            model=model,
+            tokenizer=tokenizer,
+            desc_tokenizer=desc_tokenizer,
+            device=device,
+            sampling_rate=model.config.sampling_rate,
+        )
+        _hf_volume.commit()  # persist downloaded weights for the next cold start
+
+    model = _indic_parler["model"]
+    tokenizer = _indic_parler["tokenizer"]
+    desc_tokenizer = _indic_parler["desc_tokenizer"]
+    device = _indic_parler["device"]
+
+    sr = _indic_parler["sampling_rate"]
+
+    # Indic Parler-TTS caps each generation at ~30s of audio, so synthesising a
+    # whole 150-word story in one pass truncates it (audio stops mid-script) and
+    # rushes the prosody. Instead render it sentence-by-sentence and stitch the
+    # segments with a short silence — no cut-offs, and a natural bedtime pause at
+    # each punctuation mark.
+    def _chunk(t: str, max_chars: int = 220) -> list[str]:
+        # Split on Bengali daari (।) and ? ! … . — keep the delimiter attached,
+        # then pack consecutive sentences up to max_chars so chunks stay well
+        # under the ~30s generation limit.
+        parts = [p for p in re.split(r"(?<=[।!?.…])\s+", t.strip()) if p]
+        chunks, cur = [], ""
+        for p in parts:
+            if cur and len(cur) + len(p) + 1 > max_chars:
+                chunks.append(cur)
+                cur = p
+            else:
+                cur = f"{cur} {p}".strip()
+        if cur:
+            chunks.append(cur)
+        return chunks or [t.strip()]
+
+    desc = desc_tokenizer(caption, return_tensors="pt").to(device)
+    gap = np.zeros(int(sr * 0.35), dtype=np.float32)  # ~0.35s pause between sentences
+    # Sample instead of greedy decoding: greedy makes Parler sound flat/monotone.
+    # do_sample + temperature gives more lively, natural prosody (the closest lever
+    # Bengali has — it has no emotion-prompt support, unlike VoxCPM2 for English).
+    # Fixed seed keeps a given story's audio reproducible across runs.
+    set_seed(0)
+    segments: list = []
+    for chunk in _chunk(text):
+        prompt = tokenizer(chunk, return_tensors="pt").to(device)
+        with torch.no_grad():
+            generation = model.generate(
+                input_ids=desc.input_ids,
+                attention_mask=desc.attention_mask,
+                prompt_input_ids=prompt.input_ids,
+                prompt_attention_mask=prompt.attention_mask,
+                do_sample=True,
+                temperature=1.0,
+            )
+        seg = generation.cpu().numpy().squeeze().astype(np.float32)
+        if seg.size == 0:
+            continue
+        segments.append(seg)
+        segments.append(gap)
+
+    if not segments:
+        return b""
+    audio = np.concatenate(segments[:-1])  # drop the trailing gap
+    buf = io.BytesIO()
+    sf.write(buf, audio, sr, format="WAV")
+    return buf.getvalue()
+
+
+@app.function(
+    gpu=_gpu,
+    image=_indictts_image,
+    volumes={"/models": _indictts_volume},
+    timeout=1800,  # first call downloads + unzips the ~1.5 GB checkpoint
+    min_containers=_min_containers,
+)
+def run_tts_indic_ai4bharat(text: str, checkpoint_url: str) -> bytes:
+    """Synthesise Bengali speech with AI4Bharat Indic-TTS (FastPitch + HiFi-GAN).
+
+    No reference clip — a dedicated, MOS-tuned Bengali acoustic model with a fixed
+    voice. The language checkpoint zip is downloaded + unzipped once into a volume.
+
+    Returns WAV bytes (model sample rate), or b'' on failure.
+    """
+    import glob
+    import io
+    import os
+    import re
+    import subprocess
+
+    import numpy as np
+    import soundfile as sf
+    import torch
+
+    ckpt_dir = "/models/indic_tts_bn"
+    # One-time download + unzip into the persistent volume.
+    if not glob.glob(f"{ckpt_dir}/**/fastpitch/best_model.pth", recursive=True):
+        os.makedirs(ckpt_dir, exist_ok=True)
+        zip_path = "/tmp/indictts_bn.zip"
+        subprocess.run(["wget", "-q", "-O", zip_path, checkpoint_url], check=True)
+        subprocess.run(["unzip", "-o", "-q", zip_path, "-d", ckpt_dir], check=True)
+        os.remove(zip_path)
+        _indictts_volume.commit()
+
+    def _cfg_near(ckpt_path: str) -> str | None:
+        d = os.path.dirname(ckpt_path)
+        for cand in (os.path.join(d, "config.json"),
+                     os.path.join(os.path.dirname(d), "config.json")):
+            if os.path.exists(cand):
+                return cand
+        hits = glob.glob(os.path.join(os.path.dirname(d), "**", "config.json"), recursive=True)
+        return hits[0] if hits else None
+
+    # Lazy singleton — build the Coqui Synthesizer once per warm container.
+    if "syn" not in _indictts:
+        from TTS.utils.synthesizer import Synthesizer
+
+        fp = sorted(glob.glob(f"{ckpt_dir}/**/fastpitch/best_model.pth", recursive=True))
+        voc = sorted(glob.glob(f"{ckpt_dir}/**/hifigan/best_model.pth", recursive=True))
+        if not fp or not voc:
+            print("[indic_tts] checkpoints not found:", glob.glob(f"{ckpt_dir}/**", recursive=True)[:20])
+            return b""
+        import json
+
+        spk_file = os.path.join(os.path.dirname(fp[0]), "speakers.pth")
+        fp_cfg_path = _cfg_near(fp[0])
+        # The config bakes a RELATIVE speakers_file path from AI4Bharat's training
+        # tree (resolved against CWD → FileNotFoundError). Rewrite it to our
+        # absolute path so coqui loads the right speaker map.
+        if fp_cfg_path and os.path.exists(spk_file):
+            with open(fp_cfg_path) as f:
+                cfg_json = json.load(f)
+            cfg_json["speakers_file"] = spk_file
+            if isinstance(cfg_json.get("model_args"), dict):
+                cfg_json["model_args"]["speakers_file"] = spk_file
+            fp_cfg_path = "/tmp/fastpitch_config_patched.json"
+            with open(fp_cfg_path, "w") as f:
+                json.dump(cfg_json, f)
+        syn = Synthesizer(
+            tts_checkpoint=fp[0],
+            tts_config_path=fp_cfg_path,
+            tts_speakers_file=spk_file if os.path.exists(spk_file) else None,
+            vocoder_checkpoint=voc[0],
+            vocoder_config=_cfg_near(voc[0]),
+            use_cuda=torch.cuda.is_available(),
+        )
+        _indictts["syn"] = syn
+        # Multi-speaker model (trained on male+female): prefer the female voice for
+        # the grandmother persona. AI4Bharat names them literally "male"/"female".
+        speaker = None
+        try:
+            names = list(syn.tts_model.speaker_manager.name_to_id.keys())
+            speaker = next((s for s in names if "fem" in s.lower()), names[0]) if names else None
+        except Exception:  # noqa: BLE001 — single-speaker model, no manager
+            speaker = None
+        _indictts["speaker"] = speaker
+        print("[indic_tts] loaded; speakers available:",
+              locals().get("names", "?"), "| using:", speaker)
+
+    syn = _indictts["syn"]
+    speaker = _indictts.get("speaker")
+    sr = syn.output_sample_rate
+
+    def _chunk(t: str, max_chars: int = 220, min_chars: int = 8) -> list[str]:
+        parts = [p for p in re.split(r"(?<=[।!?.…])\s+", t.strip()) if p.strip()]
+        chunks, cur = [], ""
+        for p in parts:
+            if cur and len(cur) + len(p) + 1 > max_chars:
+                chunks.append(cur)
+                cur = p
+            else:
+                cur = f"{cur} {p}".strip()
+        if cur:
+            chunks.append(cur)
+        # Merge too-short fragments (a lone quote/word) into a neighbour — FastPitch's
+        # conv kernel errors when a chunk is shorter than the kernel size.
+        merged: list = []
+        for c in chunks:
+            if merged and len(c.strip()) < min_chars:
+                merged[-1] = (merged[-1] + " " + c).strip()
+            else:
+                merged.append(c)
+        if len(merged) > 1 and len(merged[0].strip()) < min_chars:
+            merged[1] = (merged[0] + " " + merged[1]).strip()
+            merged = merged[1:]
+        return [c for c in merged if c.strip()] or [t.strip()]
+
+    gap = np.zeros(int(sr * 0.35), dtype=np.float32)
+    segments: list = []
+    for chunk in _chunk(text):
+        if len(chunk.strip()) < 2:  # never feed FastPitch a 1-char chunk
+            continue
+        kwargs = {"speaker_name": speaker} if speaker else {}
+        try:
+            wav = syn.tts(chunk, **kwargs)
+        except Exception as e:  # noqa: BLE001 — skip a bad chunk, don't fail the story
+            print(f"[indic_tts] chunk skipped ({e}): {chunk[:40]!r}", flush=True)
+            continue
+        seg = np.asarray(wav, dtype=np.float32).squeeze()
+        if seg.size == 0:
+            continue
+        segments.append(seg)
+        segments.append(gap)
+
+    if not segments:
+        return b""
+    audio = np.concatenate(segments[:-1])
+    buf = io.BytesIO()
+    sf.write(buf, audio, sr, format="WAV")
+    return buf.getvalue()
+
+
+def synthesize_bengali_remote(text: str, caption: str) -> bytes:
+    """Plain-callable entry point used by core/tts.py for the Bengali path.
+
+    Reads the active Bengali TTS backend from get_config() and dispatches to the
+    right deployed Modal function. May raise — core/tts.py wraps this and falls
+    back to text-only so audio failure never breaks the app.
+
+    - 'indic_tts': AI4Bharat Indic-TTS (FastPitch + HiFi-GAN), no reference clip.
+    - 'indic_parler': description-controlled Indic Parler-TTS.
+    """
+    from core.model_config import get_tts_repo
+
+    cfg = get_config()
+    backend = cfg.tts_bn_backend
+
+    if backend == "indic_tts":
+        # AI4Bharat Indic-TTS (FastPitch + HiFi-GAN) — no caption/reference; the
+        # 'repo' here is the checkpoint-zip URL.
+        fn = modal.Function.from_name("rupkotha", "run_tts_indic_ai4bharat")
+        return fn.remote(text, get_tts_repo("indic_tts"))
+
+    fn = modal.Function.from_name("rupkotha", "run_tts_bengali")
+    return fn.remote(text, caption, get_tts_repo("indic_parler"))
+
+
+# ─────────────────────────────────────────────────────────────────────────────
+# English TTS — VoxCPM2
+# ─────────────────────────────────────────────────────────────────────────────
+
+@app.function(
+    gpu=_gpu,
+    image=_ml_image,
+    volumes={_HF_CACHE: _hf_volume},
+    secrets=_hf_secrets,
+    timeout=900,  # generous: covers a cold-start model download + load
+    min_containers=_min_containers,
+)
+def run_tts_english(text: str, voice_prompt: str, model_repo: str) -> bytes:
+    """Synthesise English speech using VoxCPM2 Voice Design.
+
+    The persona is supplied via Voice Design: VoxCPM2 reads a parenthetical
+    description at the start of the text and generates a matching novel voice
+    (no reference audio needed).
+
+    Args:
+        text: Story text in English.
+        voice_prompt: Voice Design persona description (without parentheses).
+        model_repo: HuggingFace repo ID from get_tts_repo() (e.g. 'openbmb/VoxCPM2').
+
+    Returns:
+        WAV audio as bytes, or b'' on failure.
+    """
+    import io
+
+    import soundfile as sf
+    from voxcpm import VoxCPM
+
+    # Lazy singleton — load once per warm container, reuse across calls.
+    if "model" not in _voxcpm:
+        model = VoxCPM.from_pretrained(model_repo, load_denoiser=False)
+        _voxcpm.update(model=model, sampling_rate=model.tts_model.sample_rate)
+        _hf_volume.commit()  # persist downloaded weights for the next cold start
+
+    model = _voxcpm["model"]
+    # Voice Design: the persona goes in parentheses at the start of the text.
+    design_text = f"({voice_prompt}){text}"
+    audio = model.generate(
+        text=design_text,
+        cfg_value=2.0,
+        inference_timesteps=10,
+    )
+
+    buf = io.BytesIO()
+    sf.write(buf, audio, _voxcpm["sampling_rate"], format="WAV")
+    return buf.getvalue()
+
+
+def synthesize_english_remote(text: str, voice_prompt: str) -> bytes:
+    """Plain-callable entry point used by core/tts.py for the English path.
+
+    Reads the active English TTS backend from get_config(), resolves it to a
+    repo ID via get_tts_repo(), and dispatches to the deployed Modal
+    `run_tts_english` function. May raise — core/tts.py wraps this and falls
+    back to text-only so audio failure never breaks the app.
+    """
+    from core.model_config import get_tts_repo
+
+    repo = get_tts_repo(get_config().tts_en_backend)
+    fn = modal.Function.from_name("rupkotha", "run_tts_english")
+    return fn.remote(text, voice_prompt, repo)
+
+
+# ────────────────────────────────────────────────────���────────────────────────
+# Speech-to-text — faster-whisper
+# ─────────────────────────────────────────────────────────────────────────────
+
+def _stt_faster_whisper(audio_bytes: bytes, model_size: str, language: str) -> str:
+    """Transcribe with faster-whisper (used for size tags like 'large-v3')."""
+    import io
+
+    import torch
+    from faster_whisper import WhisperModel
+
+    key = ("fw", model_size)
+    if key not in _whisper:
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        compute_type = "float16" if device == "cuda" else "int8"
+        _whisper[key] = WhisperModel(
+            model_size, device=device, compute_type=compute_type, download_root=_HF_CACHE
+        )
+        _hf_volume.commit()  # persist the downloaded model for the next cold start
+
+    model = _whisper[key]
+    segments, _ = model.transcribe(
+        io.BytesIO(audio_bytes), language=language, vad_filter=True
+    )
+    return "".join(seg.text for seg in segments).strip()
+
+
+def _stt_transformers(audio_bytes: bytes, model_repo: str) -> str:
+    """Transcribe with a HF transformers ASR checkpoint (e.g. a Bengali-specific
+    fine-tuned Whisper). The pipeline ffmpeg-decodes raw bytes and resamples."""
+    import torch
+    from transformers import pipeline
+
+    key = ("hf", model_repo)
+    if key not in _whisper:
+        device = 0 if torch.cuda.is_available() else -1
+        _whisper[key] = pipeline(
+            "automatic-speech-recognition", model=model_repo, device=device
+        )
+        _hf_volume.commit()
+
+    result = _whisper[key](audio_bytes)
+    return (result.get("text") or "").strip()
+
+
+@app.function(
+    gpu=_gpu,
+    image=_ml_image,
+    volumes={_HF_CACHE: _hf_volume},
+    secrets=_hf_secrets,
+    timeout=600,  # generous: covers a cold-start model download + load
+    min_containers=_min_containers,
+)
+def run_stt(audio_bytes: bytes, language: str, model: str) -> str:
+    """Transcribe audio to text.
+
+    Two backends, chosen by the model identifier:
+      - a faster-whisper size tag (e.g. 'large-v3', no '/') → faster-whisper
+      - a HuggingFace repo (e.g. 'bangla-asr/whisper-medium-bn', has '/')
+        → transformers ASR pipeline (Bengali-specific models)
+
+    Args:
+        audio_bytes: Raw audio bytes (any format ffmpeg accepts).
+        language: 'en' or 'bn'.
+        model: stt_model (EN) or stt_bn_model (BN) from get_config().
+
+    Returns:
+        Transcribed text, or '' on failure (caller falls back to typed input).
+    """
+    if not audio_bytes:
+        return ""
+    try:
+        if "/" in model:
+            return _stt_transformers(audio_bytes, model)
+        return _stt_faster_whisper(audio_bytes, model, language)
+    except Exception as e:  # noqa: BLE001 — never raise; caller falls back to text
+        print(f"[modal_infra] run_stt failed: {e}")
+        return ""
+
+
+def transcribe_remote(audio_bytes: bytes, language: str, model: str) -> str:
+    """Plain-callable entry point used by core/stt.py.
+
+    Dispatches to the deployed Modal `run_stt` function. May raise — core/stt.py
+    wraps this and returns '' so the caller falls back to typed input.
+    """
+    fn = modal.Function.from_name("rupkotha", "run_stt")
+    return fn.remote(audio_bytes, language, model)

core/model_config.py

@@ -0,0 +1,177 @@
+# core/model_config.py
+
+import os
+from dataclasses import dataclass
+
+# UI-only mock mode for local development: set RUPKOTHA_MOCK=1 to make the core
+# wrappers return canned story/audio/transcript WITHOUT calling Modal (no GPU, no
+# cost). Lets you iterate on layout/CSS/aesthetics offline. Default off (real models).
+UI_MOCK: bool = os.environ.get("RUPKOTHA_MOCK", "") == "1"
+
+# ─────────────────────────────────────────────
+#  ACTIVE STACK — Stack A is the sole submission stack (OpenBMB prize path).
+#  The StackConfig machinery below is kept so a stack could be added/swapped, but
+#  only "A" is defined; Stacks B/C were dropped.
+ACTIVE_STACK: str = "A"
+# ─────────────────────────────────────────────
+#  WHERE INFERENCE RUNS
+#  "modal" → models served on Modal cloud GPUs (current; see CLAUDE.md infra note)
+#  "local" → models served on the local machine (legacy offline path; unsupported)
+COMPUTE_LOCATION: str = "modal"
+#  Modal GPU tier for inference containers. A10G (24 GB) covers Stack A's runtime.
+MODAL_GPU: str = "A10G"
+#  Keep one container warm to hide cold starts. =1 during a demo (no cold start,
+#  but bills idle GPU — incl. the A100 FT-serve container, so revert to 0 after).
+#  =0 scales to zero and only bills per request (cold start ~3-5 min on first call).
+MODAL_MIN_CONTAINERS: int = 0
+# ─────────────────────────────────────────────
+
+
+@dataclass(frozen=True)
+class StackConfig:
+    name: str
+    description: str
+    vision_model: str          # Ollama model tag
+    vision_backend: str        # "ollama" (all stacks use Ollama)
+    stt_model: str             # faster-whisper model size
+    stt_bn_model: str | None   # optional Bengali-specific STT model (HF repo)
+    tts_en_backend: str        # "voxcpm2"
+    tts_bn_backend: str        # "indic_tts" (chosen) | "indic_parler" (alt)
+    tts_bn_ref_audio: str | None  # unused (was for the removed IndicF5 voice-clone)
+    total_params_b: float      # informational — for README generation
+    openbmb_prize_eligible: bool
+
+
+STACKS: dict[str, StackConfig] = {
+    "A": StackConfig(
+        name="Stack A — OpenBMB Prize Path",
+        description="MiniCPM-V 4.5 + VoxCPM2 + AI4Bharat Indic-TTS. ~12.4B. OpenBMB prize eligible.",
+        # Default (~Q4, 6.1GB). q8_0 was tested and gave NO Bengali quality gain at
+        # higher cost/latency — precision is not the bottleneck, the 8B model's
+        # Bengali capability is. Bengali quality is addressed via two-pass (Lever C).
+        vision_model="openbmb/minicpm-v4.5",
+        vision_backend="ollama",
+        stt_model="large-v3",
+        stt_bn_model="bangla-asr/whisper-medium-bn",
+        tts_en_backend="voxcpm2",
+        # Bengali TTS: AI4Bharat Indic-TTS (FastPitch). Chosen over Indic Parler-TTS
+        # (sounded artificial) and IndicF5 (voice clone — removed: too slow even on
+        # A100, needs a reference clip). FastPitch is fast, no reference needed.
+        tts_bn_backend="indic_tts",
+        tts_bn_ref_audio=None,
+        total_params_b=12.4,
+        openbmb_prize_eligible=True,
+    ),
+}
+
+
+def get_config() -> StackConfig:
+    """Returns the active stack config. Import this everywhere model details are needed."""
+    if ACTIVE_STACK not in STACKS:
+        raise ValueError(
+            f"ACTIVE_STACK='{ACTIVE_STACK}' is not valid. Stack A is the only defined stack."
+        )
+    return STACKS[ACTIVE_STACK]
+
+
+def get_all_stacks() -> dict[str, StackConfig]:
+    """Returns all defined stacks (currently just Stack A)."""
+    return STACKS
+
+
+# HF repo IDs for the TTS backends. Model names live ONLY in this file — the
+# StackConfig stores the backend *key* ('voxcpm2' | 'indic_parler' | 'indic_tts');
+# this maps that key to the actual repo passed to core/modal_infra.py.
+TTS_BACKEND_REPOS: dict[str, str] = {
+    "voxcpm2": "openbmb/VoxCPM2",            # English (Voice Design)
+    "indic_parler": "ai4bharat/indic-parler-tts",
+    # AI4Bharat Indic-TTS (FastPitch + HiFi-GAN) — no HF repo; the value is the
+    # GitHub-release checkpoint zip (per-language). Bengali = bn.zip (~1.5 GB).
+    # Dedicated, MOS-tuned, no reference clip; fixed voice (no persona control).
+    "indic_tts": "https://github.com/AI4Bharat/Indic-TTS/releases/download/v1-checkpoints-release/bn.zip",
+}
+
+
+def get_tts_repo(backend: str) -> str:
+    """Resolve a TTS backend key to its HuggingFace repo ID."""
+    try:
+        return TTS_BACKEND_REPOS[backend]
+    except KeyError:
+        raise ValueError(
+            f"Unknown TTS backend '{backend}'. Known: {list(TTS_BACKEND_REPOS)}"
+        )
+
+
+# Per-language decoding params for the vision/story model (passed to Ollama).
+# Bengali uses a more conservative profile: lower temperature + min_p floor +
+# repetition penalty suppress the wrong-script (Latin) tokens, invented non-words,
+# and phrase repetition that high-temperature sampling produces in a lower-resource
+# language. English can afford a livelier profile. Tune these here only.
+VISION_GEN_OPTIONS: dict[str, dict] = {
+    "en": {
+        "temperature": 0.8,
+        "top_p": 0.95,
+        "repeat_penalty": 1.1,
+        "num_predict": 500,  # bound the response; a bedtime story is short
+    },
+    "bn": {
+        "temperature": 0.45,
+        "top_p": 0.9,
+        "top_k": 40,
+        "min_p": 0.05,
+        "repeat_penalty": 1.18,
+        "repeat_last_n": 64,
+        "num_predict": 700,  # Bengali uses more tokens per word; still bounded
+    },
+}
+
+
+def get_vision_options(language: str) -> dict:
+    """Return a copy of the decoding params for the given language ('en'|'bn')."""
+    return dict(VISION_GEN_OPTIONS.get(language, VISION_GEN_OPTIONS["en"]))
+
+
+# Translation-pivot path (research option #1): MiniCPM writes the story in English
+# (its strength), then IndicTrans2 translates it to fluent Bengali. Model name lives
+# here only. 1B (gated, MIT) for quality; dist-200M is the faster/smaller option.
+TRANSLATION_MODEL = "ai4bharat/indictrans2-en-indic-1B"
+# IndicTrans2 FLORES-style language codes.
+INDICTRANS_LANG_CODES: dict[str, str] = {"en": "eng_Latn", "bn": "ben_Beng"}
+
+
+def get_indictrans_code(language: str) -> str:
+    """Map our 'en'/'bn' codes to IndicTrans2's FLORES codes."""
+    try:
+        return INDICTRANS_LANG_CODES[language]
+    except KeyError:
+        raise ValueError(f"No IndicTrans2 code for language '{language}'.")
+
+
+# ── Bengali distillation fine-tuning (see finetune/) ────────────────────────
+# Teacher that writes native Bengali story labels from a drawing. Gemma 3 is
+# multimodal and writes excellent Bengali (চাঁদমামা/পুকুর register). 27B gives the
+# best labels (fewer code-switch leaks); 12B is faster. Label quality caps the
+# student, and data-gen is a one-time job, so quality is prioritised here.
+TEACHER_MODEL = "gemma3:27b"
+# Base student that gets fine-tuned (the Stack A vision model, HF repo form for
+# training — Ollama tag form for serving is in the StackConfig).
+STUDENT_BASE_REPO = "openbmb/MiniCPM-V-4_5"
+# Once a LoRA is trained + merged and served via vLLM, set this to the merged
+# model path/repo and route the vision path to it. None = use the base model.
+# Set 2026-06-13 after the held-out eval (finetune/eval_ft.py): the fine-tuned
+# model decisively beats the base on Bengali (base output was garbled + looping;
+# FT is coherent native রূপকথা), confirmed by a Bengali speaker. Bengali now routes
+# to the FT model served by finetune/serve_vllm.py (app `rupkotha-ft-serve`).
+FINETUNED_VISION_MODEL: str | None = "/data/out/minicpm-v-bengali-merged"
+
+
+def get_compute() -> dict:
+    """Returns the active compute-location settings (Modal infra). Import this in
+    core/modal_infra.py and the core/ wrappers — never hardcode GPU tier or location."""
+    if COMPUTE_LOCATION not in ("modal", "local"):
+        raise ValueError(f"COMPUTE_LOCATION='{COMPUTE_LOCATION}' is not valid. Use 'modal' or 'local'.")
+    return {
+        "location": COMPUTE_LOCATION,
+        "gpu": MODAL_GPU,
+        "min_containers": MODAL_MIN_CONTAINERS,
+    }

core/prompts.py

@@ -0,0 +1,167 @@
+# core/prompts.py
+"""All prompt templates (EN + BN, per style).
+
+Stories must read like a grandmother's bedtime tale: warm, simple, rhythmic,
+120–150 words, gentle sleepy ending, never scary, never a cliffhanger.
+
+Templates use {instruction}, {name_line}, {words}, {style_rule} placeholders.
+Bengali prompts are written natively in বাংলা — never "translate to Bengali".
+
+NOTE: This is a Step 1 skeleton. Real templates are filled in at Build Order
+steps 2 (EN) and 3 (BN).
+"""
+
+# Story-style keys exposed to the UI. EN and BN have separate style sets.
+STYLES: dict[str, dict[str, str]] = {
+    "en": {
+        "Adventure 🌟": "a gentle adventure where nothing truly scary happens",
+        "Funny 😄": "a silly, giggly tale full of light-hearted fun",
+        "Magical ✨": "a soft, wondrous tale with a touch of quiet magic",
+    },
+    "bn": {
+        "রূপকথা": "একটি কোমল রূপকথা যেখানে ভয় পাওয়ার কিছু নেই",
+        "মজার": "একটি হাসিখুশি, মজার গল্প",
+        "জাদু": "একটি কোমল, জাদুমাখা গল্প",
+    },
+}
+
+DEFAULT_WORDS = "120 to 150"
+
+# Hard cap on images woven into one story (CLAUDE.md §1: 1–4 images).
+MAX_IMAGES = 4
+
+
+def _images_rule(language: str, num_images: int) -> str:
+    """Count-aware instruction for weaving 0–4 pictures into ONE story cast."""
+    n = max(0, min(num_images, MAX_IMAGES))
+    if language == "bn":
+        if n == 0:
+            return "এবার কোনো ছবি নেই — তুমি নিজেই একটা আরামদায়ক ছোট্ট দৃশ্য কল্পনা করে নাও।"
+        if n == 1:
+            return "ছবিটি ভালো করে দেখো আর তাতে যা সত্যিই আছে — শিশুর নিজের আঁকা ও খেলনা — গল্পে বুনে দাও।"
+        return (
+            f"{n}টি ছবি আছে। সবগুলোকে একটিই গল্পে বুনে দাও — প্রতিটি ছবি যেন একই "
+            "গল্পের এক একটি চরিত্র, বন্ধু বা জায়গা হয়, সবাই একসাথে মিলেমিশে।"
+        )
+    if n == 0:
+        return "There is no picture this time — gently imagine a cosy little scene yourself."
+    if n == 1:
+        return "Use what is actually in the picture — the child's own drawing or toy."
+    return (
+        f"There are {n} pictures. Weave them ALL into a SINGLE story — let each "
+        "picture become a character, friend, or place that meets in the same gentle tale."
+    )
+
+# Gemma (Stacks B/C) overshoots word counts and writes lush prose; MiniCPM (Stack A)
+# runs terse. Nudge the cap per stack family. See CLAUDE.md §8.
+_WORDS_BY_STACK = {"A": "130 to 160", "B": "110 to 140", "C": "110 to 140"}
+
+# Bengali: a shorter target than English. A long Bengali span pushes the model
+# past its reliable range and invites invented words / script drift — keep it tight.
+_BN_WORDS = "80 to 110"
+
+# A tiny clean রূপকথা anchor (few-shot): locks Bengali script, register, and the
+# soft sleepy rhythm so the model imitates good বাংলা instead of drifting.
+_BN_EXAMPLE = (
+    "যেমন: “এক যে ছিল ছোট্ট খরগোশ, থাকত পুকুরপাড়ে। সারাদিন মাঠে খেলত, "
+    "সন্ধ্যা হলে মায়ের কোলে ঘুমিয়ে পড়ত। শুভরাত্রি।”"
+)
+
+# Lever C — two-pass. Pass 1 asks the model to DESCRIBE the image(s) in English
+# (its vision strength); pass 2 narrates from that text so the model spends all
+# its capacity on Bengali prose, not on perceiving the image at the same time.
+SCENE_SENTINEL = "[[SCENE_DESCRIPTION]]"
+
+DESCRIBE_PROMPT_EN = (
+    "Look carefully at the picture(s) a child has shared. In 3–5 simple English "
+    "sentences, describe ONLY what you actually see: the objects, their colours, "
+    "and the overall mood. Do not tell a story — just describe what is in the picture(s)."
+)
+
+_TEMPLATE_EN = """You are a warm, loving grandmother telling a bedtime story to a small child.
+Look at the picture(s) the child has shared and weave what you see into {style_rule}.
+{name_line}The child asked: "{instruction}"
+{scene}
+Rules:
+- Write {words} words. Simple words a young child knows. Gentle, rhythmic, soothing.
+- {images_rule}
+- Never anything scary, sad, or a cliffhanger.
+- End with everyone safe, cosy, and settling down to sleep.
+
+Tell the story now, in English:"""
+
+_TEMPLATE_BN = """তুমি একজন স্নেহময়ী ঠাকুমা, ছোট্ট এক শিশুকে ঘুমপাড়ানি গল্প বলছ।
+শিশুটি যে ছবি(গুলি) দেখিয়েছে তা দেখো এবং সেটিকে গল্পে বুনে দাও — {style_rule}।
+{name_line}শিশুটি বলেছে: "{instruction}"
+{scene}
+নিয়ম:
+- শুধু বিশুদ্ধ বাংলা অক্ষরে লেখো — কোনো ইংরেজি বা রোমান হরফ ব্যবহার করবে না।
+- কেবল চেনা, সঠিক বাংলা শব্দ ব্যবহার করো — কোনো বানানো বা অর্থহীন শব্দ নয়।
+- {words}টি শব্দে লেখো। ছোট শিশুর চেনা সহজ শব্দ। কোমল, ছন্দময়, আদুরে।
+- {images_rule}
+- পুকুর, মাঠ, জোনাকি, চাঁদমামার মতো চেনা বাংলা ছবি ব্যবহার করো।
+- কখনও ভয়ের, দুঃখের বা অসমাপ্ত কিছু নয়।
+- সবাই নিরাপদে, আরামে, ঘুমিয়ে পড়ার মধ্য দিয়ে গল্প শেষ করো।
+
+{example}
+
+এবার বাংলায় গল্পটি বলো:"""
+
+
+def _scene_block(language: str, scene_description: str) -> str:
+    """Two-pass: a grounding block holding the (English) scene description for
+    pass 2. Empty in single-pass mode."""
+    if not scene_description:
+        return ""
+    if language == "bn":
+        return f"\nছবিতে যা আছে তার বর্ণনা: {scene_description}\n"
+    return f"\nWhat is in the picture: {scene_description}\n"
+
+
+def _scene_rule(language: str) -> str:
+    """Two-pass replacement for the images_rule: weave the described scene in."""
+    if language == "bn":
+        return "উপরের বর্ণনায় থাকা জিনিসগুলো গল্পে বুনে দাও।"
+    return "Weave the things in the description above into the story."
+
+
+def build_story_prompt(
+    instruction: str,
+    language: str,
+    style: str,
+    child_name: str = "",
+    stack_key: str = "A",
+    num_images: int = 1,
+    scene_description: str = "",
+) -> str:
+    """Build the full story prompt for the vision model. language: 'en' | 'bn'.
+
+    num_images (0–4) drives how the prompt asks the model to weave the pictures
+    into one shared story cast.
+
+    scene_description: two-pass (Lever C). When set, this is the pass-2 prompt —
+    it injects the (English) scene description as text and the model narrates from
+    it with NO image attached. Pass a sentinel (SCENE_SENTINEL) here and have the
+    Modal layer substitute the real description after the describe pass."""
+    lang = language if language in ("en", "bn") else "en"
+    styles = STYLES[lang]
+    style_rule = styles.get(style) or next(iter(styles.values()))
+    words = _WORDS_BY_STACK.get(stack_key, DEFAULT_WORDS)
+    # In two-pass mode the story is grounded in the description, not the raw image.
+    images_rule = _scene_rule(lang) if scene_description else _images_rule(lang, num_images)
+    scene = _scene_block(lang, scene_description)
+
+    if lang == "bn":
+        name_line = f"শিশুটির নাম {child_name}; গল্পে তার নাম বুনে দাও।\n" if child_name else ""
+        instr = instruction.strip() or "আমাকে একটা গল্প বলো"
+        return _TEMPLATE_BN.format(
+            style_rule=style_rule, name_line=name_line, instruction=instr,
+            words=_BN_WORDS, images_rule=images_rule, example=_BN_EXAMPLE, scene=scene,
+        )
+
+    name_line = f"The child's name is {child_name}; weave their name in warmly.\n" if child_name else ""
+    instr = instruction.strip() or "tell me a bedtime story"
+    return _TEMPLATE_EN.format(
+        style_rule=style_rule, name_line=name_line, instruction=instr,
+        words=words, images_rule=images_rule, scene=scene,
+    )

core/stt.py

@@ -0,0 +1,30 @@
+# core/stt.py
+"""Whisper audio → text (EN/BN).
+
+Uses get_config().stt_model (EN) or stt_bn_model (BN, if set), transcribed on
+Modal. Returns '' on failure so the caller can fall back to typed input — the
+primary, reliable path for Bengali especially (CLAUDE.md §2, §14). Never raises.
+"""
+
+from core.model_config import UI_MOCK, get_config
+from core.modal_infra import transcribe_remote
+
+
+def transcribe(audio_path: str, language: str) -> str:
+    """language is 'en' or 'bn'. Returns '' on failure. Never raises."""
+    if UI_MOCK:  # local UI dev — no Modal/GPU
+        return "tell me a story about my dragon"
+    if not audio_path:
+        return ""
+    cfg = get_config()
+    if language == "bn" and cfg.stt_bn_model:
+        model = cfg.stt_bn_model
+    else:
+        model = cfg.stt_model
+    try:
+        with open(audio_path, "rb") as f:
+            audio_bytes = f.read()
+        return transcribe_remote(audio_bytes, language, model) or ""
+    except Exception as e:  # noqa: BLE001 — never raise; caller uses typed input
+        print(f"[stt.py] transcription failed: {e}")
+        return ""

core/tts.py

@@ -0,0 +1,92 @@
+# core/tts.py
+"""Story text → grandmother-voice audio (EN/BN).
+
+EN → VoxCPM2 Voice Design persona (Build Order step 5 — still stubbed).
+BN → get_config().tts_bn_backend ('indic_parler' or 'indic_tts'), synthesised on
+     Modal via Indic Parler-TTS. The grandmother persona is controlled by the
+     caption below — tune it for warmth and storytelling pace (CLAUDE.md §9).
+Returns (None, label) if TTS unavailable or failed. Never raises.
+"""
+
+import tempfile
+from pathlib import Path
+
+from core.model_config import UI_MOCK, get_config
+from core.modal_infra import synthesize_bengali_remote, synthesize_english_remote
+
+# A pre-generated sample clip for RUPKOTHA_MOCK=1, so the audio player has something
+# to show during local UI dev (no Modal/GPU). None if the sample isn't present.
+_MOCK_WAV = next(
+    (str(p) for p in [
+        Path("finetune/data/sample/demo_story_ai4bharat.wav"),
+        Path("finetune/data/sample/demo_story_output_sampled.wav"),
+    ] if p.exists()),
+    None,
+)
+
+# Grandmother voice caption — controls the Indic Parler-TTS speaker persona.
+# Tune for warmth and pace; run 5–10 samples and keep the best (CLAUDE.md §9).
+GRANDMOTHER_CAPTION_BN = (
+    "Aditi speaks in the warm, tender voice of a loving elderly Bengali "
+    "grandmother telling a bedtime story to her beloved grandchild. Her pace is "
+    "very slow, gentle, and unhurried, pausing softly at every sentence. Her "
+    "delivery is affectionate and expressive, rising and falling with the natural "
+    "soothing melody of Bengali storytelling, full of warmth and calm. The "
+    "recording is very clear, close, and intimate, with no background noise."
+)
+
+# Grandmother Voice Design persona for VoxCPM2 (English). This is prefixed to the
+# story text in parentheses by core/modal_infra.py — keep it short and concrete.
+# Tune for warmth and a slow, storytelling pace (CLAUDE.md §11, build step 5).
+GRANDMOTHER_VOICE_EN = (
+    "a warm, loving elderly grandmother reading a bedtime story, "
+    "slow and gentle pace, soft and soothing tone"
+)
+
+
+def _write_wav(audio_bytes: bytes) -> str:
+    """Persist WAV bytes to a temp file and return its path."""
+    tmp = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
+    try:
+        tmp.write(audio_bytes)
+    finally:
+        tmp.close()
+    return tmp.name
+
+
+def speak(
+    text: str,
+    language: str,            # 'en' or 'bn'
+    voice: str = "grandmother",
+) -> tuple[str | None, str]:
+    """Returns (wav_path_or_None, tts_model_label). Never raises."""
+    cfg = get_config()
+
+    if UI_MOCK:  # local UI dev — sample clip, no Modal/GPU
+        return _MOCK_WAV, "UI-mock"
+
+    if language == "bn":
+        label = cfg.tts_bn_backend
+        if not (text or "").strip():
+            return None, label
+        try:
+            audio_bytes = synthesize_bengali_remote(text, GRANDMOTHER_CAPTION_BN)
+            if not audio_bytes:
+                return None, label
+            return _write_wav(audio_bytes), label
+        except Exception as e:  # noqa: BLE001 — never raise to the UI
+            print(f"[tts.py] Bengali TTS failed: {e}")
+            return None, label
+
+    # English path: VoxCPM2 Voice Design.
+    label = cfg.tts_en_backend
+    if not (text or "").strip():
+        return None, label
+    try:
+        audio_bytes = synthesize_english_remote(text, GRANDMOTHER_VOICE_EN)
+        if not audio_bytes:
+            return None, label
+        return _write_wav(audio_bytes), label
+    except Exception as e:  # noqa: BLE001 — never raise to the UI
+        print(f"[tts.py] English TTS failed: {e}")
+        return None, label

core/vision_story.py

@@ -0,0 +1,160 @@
+# core/vision_story.py
+"""Image(s) + instruction → bedtime story text.
+
+Builds the prompt locally, then calls the Modal VisionStory function (Ollama
+serving get_config().vision_model). Never raises to the UI: returns a friendly
+fallback string on any error so the app always shows a story.
+"""
+
+import base64
+
+from core.model_config import (
+    ACTIVE_STACK,
+    FINETUNED_VISION_MODEL,
+    UI_MOCK,
+    get_config,
+    get_vision_options,
+)
+from core.modal_infra import (
+    generate_story_ft_remote,
+    generate_story_remote,
+    translate_remote,
+)
+from core.prompts import (
+    DESCRIBE_PROMPT_EN,
+    MAX_IMAGES,
+    SCENE_SENTINEL,
+    STYLES,
+    build_story_prompt,
+)
+
+# Languages that use two-pass generation (Lever C): describe the image in English
+# first, then narrate from that text. DISABLED — tested on Stack A and it did NOT
+# improve Bengali: pass 1 produces an excellent English description, but pass 2's
+# Bengali is still poor, proving the ceiling is MiniCPM-V's Bengali *generation*,
+# not perception. (Bengali now uses the fine-tuned model instead — see below.)
+# The two-pass plumbing is kept but off by default. Add "bn" back to enable.
+_TWO_PASS_LANGUAGES: set[str] = set()
+
+# Translation pivot (research option #1): MiniCPM writes the story in English, then
+# IndicTrans2 translates to Bengali. DISABLED — owner (a Bengali speaker) judged the
+# native Lever-#1 output better: the pivot is grammatically clean but reads translated
+# (stiff formal register তিনি/করুন, no native রূপকথা imagery like চাঁদমামা/পুকুর).
+# Authentic register beats textbook grammar for a bedtime tale. Plumbing kept (Modal
+# `run_translate` + this toggle) in case it's wanted later. Set True to re-enable.
+_PIVOT_BENGALI = False
+
+# Bengali style key → its positional English counterpart (রূপকথা→Adventure, etc.),
+# used to generate the English draft before translating.
+_BN_TO_EN_STYLE = dict(zip(STYLES["bn"].keys(), STYLES["en"].keys()))
+
+_FALLBACK = (
+    "Once upon a time, the stars grew sleepy and the moon pulled a soft cloud "
+    "over the world like a blanket. Everyone yawned, snuggled close, and drifted "
+    "off to dream. Goodnight. 💤"
+)
+
+_FALLBACK_BN = (
+    "এক যে ছিল রাত, তারারা ঘুমে ঢুলছিল আর চাঁদমামা নরম মেঘের কাঁথা টেনে দিল সারা "
+    "পৃথিবীর গায়ে। সবাই হাই তুলল, একসাথে গুটিসুটি মেরে ঘুমিয়ে পড়ল। শুভরাত্রি। 💤"
+)
+
+# Canned stories for RUPKOTHA_MOCK=1 (local UI dev — no Modal/GPU). Long enough to
+# exercise the story panel's layout and scrolling.
+_MOCK_STORY_EN = (
+    "Once upon a time, on a soft green hill, a little rabbit named Pip watched the "
+    "sun slip behind the mountains. The sky turned gold, then rosy pink. Pip hopped "
+    "past the sleepy pond where fireflies began to glow like tiny stars.\n\n"
+    "A gentle breeze hummed a lullaby through the tall grass. Pip curled up under the "
+    "big banyan tree, where mother rabbit waited with a warm hug. The moon climbed up "
+    "and pulled a soft cloud over the world like a blanket.\n\n"
+    "All the little creatures yawned and snuggled close. Pip closed both sleepy eyes "
+    "and drifted into a sweet, happy dream. Goodnight, little one. 💤"
+)
+_MOCK_STORY_BN = (
+    "এক যে ছিল ছোট্ট খরগোশ, নাম তার টুনি। সে থাকত সবুজ পাহাড়ের ধারে। সন্ধ্যা হলে "
+    "সূর্যিমামা পাহাড়ের পিছনে লুকিয়ে পড়ল, আকাশ হলো সোনালি আর গোলাপি।\n\n"
+    "টুনি পুকুরপাড়ে গেল, সেখানে জোনাকিরা মিটিমিটি জ্বলছিল তারার মতো। নরম বাতাস ঘাসের "
+    "মধ্যে দিয়ে ঘুমপাড়ানি গান গাইল। টুনি বড় বটগাছের নিচে মায়ের কোলে গিয়ে বসল।\n\n"
+    "চাঁদমামা উঠে এসে সারা পৃথিবীর গায়ে নরম মেঘের কাঁথা টেনে দিল। সব ছোট্ট প্রাণীরা "
+    "হাই তুলল, গুটিসুটি মেরে ঘুমিয়ে পড়ল। শুভরাত্রি, সোনা। 💤"
+)
+
+
+def _encode(path: str) -> str:
+    with open(path, "rb") as f:
+        return base64.b64encode(f.read()).decode()
+
+
+def generate_story(
+    image_paths: list[str],   # 1–4 image paths
+    instruction: str,         # transcribed or typed; may be ''
+    language: str,            # 'en' or 'bn'
+    style: str,               # key from prompts.STYLES
+    child_name: str = "",     # optional, woven into story if provided
+) -> tuple[str, str]:
+    """Returns (story_text, model_used_label). Never raises."""
+    cfg = get_config()
+    model_label = cfg.vision_model
+    fallback = _FALLBACK_BN if language == "bn" else _FALLBACK
+
+    if UI_MOCK:  # local UI dev — canned story, no Modal/GPU
+        return _MOCK_STORY_BN if language == "bn" else _MOCK_STORY_EN, f"{model_label} · UI-mock"
+
+    try:
+        # Cap to 1–4 images and weave them all into one story (CLAUDE.md §1, step 7).
+        paths = [p for p in (image_paths or []) if p][:MAX_IMAGES]
+        images_b64 = [_encode(p) for p in paths]
+
+        # Bengali translation pivot: write the story in English, then translate.
+        if _PIVOT_BENGALI and language == "bn":
+            en_prompt = build_story_prompt(
+                instruction=instruction,
+                language="en",
+                style=_BN_TO_EN_STYLE.get(style, next(iter(STYLES["en"]))),
+                child_name=child_name,
+                stack_key=ACTIVE_STACK,
+                num_images=len(paths),
+            )
+            en_story = (
+                generate_story_remote(images_b64, en_prompt, get_vision_options("en")) or ""
+            ).strip()
+            bn_story = (translate_remote(en_story, "en", "bn") or "").strip() if en_story else ""
+            return (bn_story or fallback), f"{cfg.vision_model} → IndicTrans2"
+
+        # Bengali fine-tuned model (see finetune/). When a merged FT model is
+        # configured, route Bengali to it — the held-out eval (finetune/eval_ft.py)
+        # showed it decisively beats the base's native Bengali. Served separately
+        # via finetune/serve_vllm.py; gets the same single-pass prompt the base would.
+        if FINETUNED_VISION_MODEL and language == "bn":
+            ft_prompt = build_story_prompt(
+                instruction=instruction,
+                language="bn",
+                style=style,
+                child_name=child_name,
+                stack_key=ACTIVE_STACK,
+                num_images=len(paths),
+            )
+            ft_story = (generate_story_ft_remote(images_b64, ft_prompt) or "").strip()
+            return (ft_story or fallback), f"{cfg.vision_model} · Bengali fine-tune"
+
+        two_pass = language in _TWO_PASS_LANGUAGES and len(paths) > 0
+        # In two-pass mode the story prompt is grounded in a sentinel that the Modal
+        # layer replaces with the English description produced by the describe pass.
+        prompt = build_story_prompt(
+            instruction=instruction,
+            language=language,
+            style=style,
+            child_name=child_name,
+            stack_key=ACTIVE_STACK,
+            num_images=len(paths),
+            scene_description=SCENE_SENTINEL if two_pass else "",
+        )
+        describe_prompt = DESCRIBE_PROMPT_EN if two_pass else None
+        options = get_vision_options(language)
+        story = generate_story_remote(images_b64, prompt, options, describe_prompt)
+        story = (story or "").strip()
+        return (story or fallback), model_label
+    except Exception as e:  # noqa: BLE001 — never raise to the UI
+        print(f"[vision_story.py] generation failed: {e}")
+        return fallback, model_label

finetune/README.md

@@ -0,0 +1,91 @@
+# Bengali distillation fine-tuning (`finetune/`)
+
+Improve Stack A's weak Bengali by **fine-tuning MiniCPM-V 4.5 itself** — the
+strongest OpenBMB-prize story (*"we improved the OpenBMB model for Bengali"*).
+This stays on Stack A; only the served model weights change.
+
+> Why fine-tune and not the IndicTrans2 pivot? The owner judged the pivot's
+> translated register worse than native output. The native ceiling is MiniCPM's
+> Bengali *generation*; fine-tuning on great Bengali raises that ceiling while
+> keeping the warm native register. See the `bengali-quality-investigation` memory.
+
+## The pipeline
+
+```
+drawings ──► [Gemma teacher] ──► Bengali stories ──► [purity gate] ──► train.json
+                                                                          │
+                              LoRA fine-tune MiniCPM-V (vision frozen) ◄───┘
+                                          │
+                                    merge adapter
+                                          │
+                                  serve via vLLM ──► wire into Bengali path
+```
+
+| Stage | File | Where | Notes |
+|---|---|---|---|
+| 0. Source images | (manual, see below) | local | the real bottleneck |
+| 1. Label + filter | `gen_labels.py` + `purity.py` | Modal (Gemma) | image → Bengali story, drop leaks |
+| 2. LoRA train | `train_lora.py` | Modal A100 | ms-SWIFT (`model_type=minicpmv4_5`) |
+| 3. Merge | `merge.py` | Modal A100 | adapter → full weights |
+| 4. Serve | `serve_vllm.py` | Modal A10G | vLLM, no GGUF |
+| 5. Wire | one branch in `core/vision_story.py` | local | route Bengali to the FT model |
+
+## Stage 0 — source the drawings (`finetune/data/images/`)
+Target ~300–500 diverse, on-distribution images. Mix all four:
+- **Team-drawn crayon pics** — photograph ~100–150. Most authentic; do these first.
+- **Quick, Draw!** — Google's doodle dataset (`https://quickdraw.withgoogle.com/data`);
+  pull a few categories (`.ndjson`/`.npy`), render strokes to PNG, optionally
+  composite 2–3 doodles into a scene.
+- **Synthetic crayon** — an image model (e.g. SDXL) with prompts like
+  *"a child's crayon drawing of a house and the sun, simple, on white paper"*.
+- **Toy / play photos** — stock or self-taken photos of toys/equipment.
+
+Quality > quantity for LoRA. Keep them varied (objects, colours, scenes).
+
+## Stage 1 — generate labels
+```bash
+modal deploy core/modal_infra.py        # teacher served via run_vision_story
+uv run python finetune/gen_labels.py --images finetune/data/images \
+    --out finetune/data/train.json --teacher gemma3:27b
+```
+Produces `train.json` (kept) and `train.json.rejected` (failed the gate). **Have a
+Bengali speaker spot-check `kept`** — distillation caps the student at label quality.
+Tune strictness in `purity.py` (`is_clean(max_foreign_letters=...)`).
+
+## Stage 2 — LoRA train
+Upload `finetune/data/` to the `rupkotha-finetune` Modal Volume, then validate
+cheaply before the full run:
+```bash
+uv run modal run finetune/train_lora.py --max-steps 4   # smoke test
+uv run modal run finetune/train_lora.py                 # full 3-epoch run
+```
+`train_lora.py` drives **ms-SWIFT** (`swift sft --model_type minicpmv4_5`), not
+OpenBMB's official `finetune.py` — the latter pins torch 2.1.2 while MiniCPM-V 4.5
+remote code needs torch>=2.4, and its finetune path doesn't target 4.5. SWIFT ships
+a maintained recipe for this model so the dep matrix is solved. It converts our
+MiniCPM `conversations` JSON to SWIFT's `messages`/`images` JSONL in-container.
+Vision encoder frozen (`--freeze_vit true`), LoRA r=16 on the LLM `q/k/v/o_proj`.
+SWIFT's nested checkpoint is copied to `out/lora-bengali/` so Stage 3 is unchanged.
+
+## Stage 3 — merge
+```bash
+uv run modal run finetune/merge.py      # → out/minicpm-v-bengali-merged/
+```
+
+## Stage 4 — serve + Stage 5 — wire
+```bash
+uv run modal deploy finetune/serve_vllm.py
+```
+Then set `FINETUNED_VISION_MODEL` in `core/model_config.py` to the merged path and
+add a one-line branch in `core/vision_story.py`: when it's set and `language=="bn"`,
+call `serve_vllm.generate_story_ft_remote(...)` instead of `generate_story_remote`.
+(`run_translate`/pivot plumbing stays available as a fallback.)
+
+## Evaluate
+Hold out ~20 drawings; compare FT Bengali vs native Lever-#1 (`demo_story_bn_v2.wav`)
+with a Bengali speaker. Ship the FT model only if it clearly wins.
+
+## Cost / risk (hackathon reality)
+Heaviest path: image curation (days) + ~1–3h A100 training + integration. Validated
+upside: the Gemma teacher writes genuinely native Bengali (চাঁদমামা/পুকুর/খই), so the
+student has an excellent target. Main risk is data volume/quality, not the method.

finetune/collect.py

@@ -0,0 +1,207 @@
+# finetune/collect.py
+"""Stage 0 — collect open-source images into finetune/data/images/<source>/.
+
+Zero-friction sources (no creds) are implemented: Google Quick, Draw! and
+file-based HF image datasets. Cred-gated sources (Kaggle, Roboflow) and generated
+ones (synthetic crayon, toy stock) are documented stubs — wire them once you drop
+in the relevant API key.
+
+Examples:
+    uv run python finetune/collect.py quickdraw --per 40
+    uv run python finetune/collect.py hf --dataset ironDong/Children_Drawings --per 150
+    uv run python finetune/collect.py hf --dataset 6chan/children-hand-drawn-style-transfer --per 150
+
+The later preprocessor (finetune/preprocess.py) sweeps the whole images/ pool.
+"""
+
+import argparse
+import json
+import os
+import urllib.parse
+import urllib.request
+from pathlib import Path
+
+# Redirect the HF cache to a writable dir BEFORE huggingface_hub/datasets import
+# (the global ~/.cache/huggingface is read-only in this sandbox).
+_HF_CACHE = str(Path(__file__).resolve().parent / ".hf_cache")
+os.environ.setdefault("HF_HOME", _HF_CACHE)
+os.environ.setdefault("HF_DATASETS_CACHE", _HF_CACHE + "/datasets")
+os.environ.setdefault("HF_HUB_CACHE", _HF_CACHE + "/hub")
+
+from PIL import Image, ImageDraw  # noqa: E402
+
+IMAGES = Path(__file__).resolve().parent / "data" / "images"
+
+# Quick, Draw! public bucket (no auth). Categories that fit bedtime-story scenes.
+_QD_BASE = "https://storage.googleapis.com/quickdraw_dataset/full/simplified/"
+_QD_CATS = [
+    "house", "tree", "sun","birthday cake", "cat","castle", "dog", "flower", "bird", "fish", "star",
+    "moon", "sailboat", "butterfly", "rainbow", "cloud", "duck", "elephant", "umbrella",
+]
+
+
+def _render_strokes(strokes, size=256, line=3) -> Image.Image:
+    """Render a Quick, Draw! simplified drawing (list of [xs, ys] strokes)."""
+    img = Image.new("RGB", (size, size), "white")
+    draw = ImageDraw.Draw(img)
+    for stroke in strokes:
+        xs, ys = stroke[0], stroke[1]
+        pts = list(zip(xs, ys))
+        if len(pts) >= 2:
+            draw.line(pts, fill="black", width=line, joint="curve")
+        elif pts:
+            draw.point(pts, fill="black")
+    return img
+
+
+def collect_quickdraw(categories: list[str], per: int) -> int:
+    out = IMAGES / "quickdraw"
+    out.mkdir(parents=True, exist_ok=True)
+    total = 0
+    for cat in categories:
+        url = _QD_BASE + urllib.parse.quote(cat) + ".ndjson"
+        try:
+            resp = urllib.request.urlopen(url, timeout=60)  # streams line-by-line
+        except Exception as e:  # noqa: BLE001
+            print(f"  {cat}: download failed ({e})")
+            continue
+        n = 0
+        for raw in resp:  # stop early — we don't pull the whole (huge) file
+            if n >= per:
+                break
+            try:
+                obj = json.loads(raw)
+                if obj.get("recognized") is False:
+                    continue
+                _render_strokes(obj["drawing"]).save(
+                    out / f"qd_{cat.replace(' ', '_')}_{n}.png"
+                )
+                n += 1
+                total += 1
+            except Exception:  # noqa: BLE001
+                continue
+        print(f"  {cat}: {n}")
+    print(f"Quick, Draw!: {total} images -> {out}")
+    return total
+
+
+def _example_image(example: dict):
+    """Find a PIL image in a datasets example (Image-feature columns decode to PIL)."""
+    for key in ("image", "img", "png", "jpg", "jpeg"):
+        v = example.get(key)
+        if isinstance(v, Image.Image):
+            return v
+    for v in example.values():  # fall back to any PIL value
+        if isinstance(v, Image.Image):
+            return v
+    return None
+
+
+def collect_hf(dataset: str, per: int) -> int:
+    """Pull images from an HF image dataset. Tries raw files first, then falls back
+    to parquet via `datasets` (run with: uv run --group finetune ...)."""
+    import shutil
+
+    from huggingface_hub import snapshot_download
+
+    out = IMAGES / ("hf_" + dataset.split("/")[-1].lower().replace("-", "_"))
+    out.mkdir(parents=True, exist_ok=True)
+
+    # 1) file-based datasets that ship raw images
+    try:
+        local = snapshot_download(
+            repo_id=dataset, repo_type="dataset", cache_dir=_HF_CACHE,
+            allow_patterns=["*.png", "*.jpg", "*.jpeg", "*.webp"],
+        )
+        imgs = [p for ext in ("png", "jpg", "jpeg", "webp") for p in Path(local).rglob(f"*.{ext}")][:per]
+    except Exception:  # noqa: BLE001
+        imgs = []
+    if imgs:
+        for i, p in enumerate(imgs):
+            shutil.copy(p, out / f"{p.stem}_{i}{p.suffix.lower()}")
+        print(f"HF {dataset}: {len(imgs)} image files -> {out}")
+        return len(imgs)
+
+    # 2) parquet datasets via `datasets` (streaming so we don't pull the whole thing)
+    try:
+        from datasets import load_dataset
+    except ImportError:
+        print(f"HF {dataset}: parquet dataset — run with `uv run --group finetune ...`")
+        return 0
+    n = 0
+    last_err = None
+    for split in ("train", "validation", "test"):
+        try:
+            ds = load_dataset(dataset, split=split, streaming=True)
+        except Exception as e:  # noqa: BLE001
+            last_err = e
+            continue
+        for ex in ds:
+            if n >= per:
+                break
+            img = _example_image(ex)
+            if img is None:
+                continue
+            img.convert("RGB").save(out / f"{n}.png")
+            n += 1
+        if n >= per:
+            break
+    if n == 0 and last_err is not None:
+        msg = str(last_err)
+        if "gated" in msg or "authenticated" in msg or "401" in msg:
+            print(f"HF {dataset}: GATED — accept its license on the Hub and set a local "
+                  f"HF_TOKEN (export HF_TOKEN=hf_xxx), then retry.")
+        else:
+            print(f"HF {dataset}: failed — {msg[:160]}")
+        return 0
+    print(f"HF {dataset}: {n} images (parquet) -> {out}")
+    return n
+
+
+# ── Cred-gated / generative sources: documented stubs ───────────────────────
+_STUBS = """
+Cred-gated and generated sources (wire once you have the key):
+
+  Kaggle  (needs ~/.kaggle/kaggle.json):
+    pip install kaggle
+    kaggle datasets download -d vishmiperera/children-drawings -p data/images/kaggle --unzip
+    kaggle datasets download -d lachin007/drawaperson-handdrawn-sketches-by-children \\
+        -p data/images/kaggle --unzip
+
+  Roboflow ESRA (needs Roboflow API key):
+    pip install roboflow  # then use the dataset's export snippet → data/images/roboflow
+
+  Synthetic crayon (SDXL on Modal — generate to fill volume/diversity):
+    prompt e.g. "a child's crayon drawing of <scene>, simple, on white paper"
+    → save to data/images/synthetic/   (a generate_synthetic.py can be added on request)
+
+  Toy / play photos (Unsplash/Pexels API):
+    fetch with your API key → data/images/toys/
+"""
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    qd = sub.add_parser("quickdraw", help="render Quick, Draw! doodles")
+    qd.add_argument("--per", type=int, default=40, help="images per category")
+    qd.add_argument("--categories", nargs="*", default=_QD_CATS)
+
+    hf = sub.add_parser("hf", help="pull a file-based HF image dataset")
+    hf.add_argument("--dataset", required=True)
+    hf.add_argument("--per", type=int, default=150)
+
+    sub.add_parser("sources", help="print cred-gated source instructions")
+
+    args = ap.parse_args()
+    if args.cmd == "quickdraw":
+        collect_quickdraw(args.categories, args.per)
+    elif args.cmd == "hf":
+        collect_hf(args.dataset, args.per)
+    elif args.cmd == "sources":
+        print(_STUBS)
+
+
+if __name__ == "__main__":
+    main()

finetune/eval_ft.py

@@ -0,0 +1,154 @@
+# finetune/eval_ft.py
+"""Stage 5 gate — side-by-side Bengali eval: fine-tuned MiniCPM-V (vLLM) vs the
+native base model (the current shipping "Lever #1" path).
+
+CLAUDE.md / finetune/README are explicit: ship the fine-tune ONLY if it clearly
+beats the native path in a human read. The train metrics (loss/token_acc) say the
+LoRA fit the distilled targets — they do NOT say Bengali quality improved. This
+script produces the artifact a Bengali speaker needs to make that call before
+FINETUNED_VISION_MODEL is ever set.
+
+Fairness: BOTH paths get the EXACT same app-built Bengali prompt (build_story_prompt)
+and the same image — exactly what each would receive in production. The only
+difference under test is the model weights.
+
+  native: core.modal_infra.generate_story_remote  → base openbmb/MiniCPM-V-4_5 (Ollama)
+  FT:     finetune.serve_vllm.generate_story_ft_remote → merged LoRA (vLLM)
+
+Held-out set: the 61 labelset images that the purity gate rejected, so they were
+NEVER trained on, yet are on-distribution. (Override with --images for your own.)
+
+Run:
+    uv run modal deploy finetune/serve_vllm.py          # FT server must be live
+    uv run python finetune/eval_ft.py --n 10            # 10 held-out images
+    uv run python finetune/eval_ft.py --images path/to/dir --n 8 --style রূপকথা
+Out:
+    finetune/eval_results/ft_vs_native_YYYYMMDD_HHMM.md
+"""
+
+import argparse
+import base64
+import json
+import os
+import sys
+import time
+from datetime import datetime
+from pathlib import Path
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from core.model_config import ACTIVE_STACK, get_vision_options
+from core.modal_infra import generate_story_remote
+from core.prompts import STYLES, build_story_prompt
+from finetune.serve_vllm import generate_story_ft_remote
+
+TRAIN_JSON = Path("finetune/data/train.json")
+LABELSET = Path("finetune/data/labelset")
+
+
+def held_out_images() -> list[Path]:
+    """Labelset images that are NOT in train.json — unseen but on-distribution."""
+    trained = {x["image"].split("/")[-1] for x in json.loads(TRAIN_JSON.read_text())}
+    imgs = [
+        p for p in sorted(LABELSET.glob("*"))
+        if p.suffix.lower() in (".jpg", ".jpeg", ".png") and p.name not in trained
+    ]
+    return imgs
+
+
+def _encode(path: Path) -> str:
+    return base64.b64encode(path.read_bytes()).decode()
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--images", default=None, help="dir of images (default: held-out labelset)")
+    ap.add_argument("--n", type=int, default=10, help="number of images to evaluate")
+    ap.add_argument("--style", default="রূপকথা", choices=list(STYLES["bn"].keys()))
+    ap.add_argument("--instruction", default="একটা গল্প বলো")
+    args = ap.parse_args()
+
+    if args.images:
+        imgs = [
+            p for p in sorted(Path(args.images).glob("*"))
+            if p.suffix.lower() in (".jpg", ".jpeg", ".png")
+        ]
+    else:
+        imgs = held_out_images()
+    imgs = imgs[: args.n]
+    if not imgs:
+        sys.exit("No images found to evaluate.")
+
+    print(f"Evaluating {len(imgs)} images · style={args.style} · stack={ACTIVE_STACK}", flush=True)
+    options = get_vision_options("bn")
+
+    # Precompute the (identical) prompt + encoded bytes per image.
+    items = []
+    for img in imgs:
+        prompt = build_story_prompt(
+            instruction=args.instruction,
+            language="bn",
+            style=args.style,
+            child_name="",
+            stack_key=ACTIVE_STACK,
+            num_images=1,
+        )
+        items.append((img, prompt, [_encode(img)]))
+
+    # Two phases so each serverless model cold-starts ONCE, not per image (the
+    # alternating native→FT loop kept scaling the other model back to zero).
+    print("Phase 1/2: native (base) ...", flush=True)
+    natives = []
+    for i, (img, prompt, b64) in enumerate(items, 1):
+        t0 = time.time()
+        story = (generate_story_remote(b64, prompt, options) or "").strip()
+        dt = round(time.time() - t0, 1)
+        print(f"  native [{i}/{len(items)}] {img.name}  {dt}s", flush=True)
+        natives.append((story, dt))
+
+    print("Phase 2/2: fine-tuned (vLLM) ...", flush=True)
+    fts = []
+    for i, (img, prompt, b64) in enumerate(items, 1):
+        t0 = time.time()
+        story = (generate_story_ft_remote(b64, prompt) or "").strip()
+        dt = round(time.time() - t0, 1)
+        print(f"  ft [{i}/{len(items)}] {img.name}  {dt}s", flush=True)
+        fts.append((story, dt))
+
+    rows = [
+        (items[i][0], natives[i][0], natives[i][1], fts[i][0], fts[i][1])
+        for i in range(len(items))
+    ]
+
+    out_dir = Path("finetune/eval_results")
+    out_dir.mkdir(exist_ok=True)
+    fname = out_dir / f"ft_vs_native_{datetime.now():%Y%m%d_%H%M}.md"
+    lines = [
+        f"# FT vs Native — Bengali story quality ({args.style})",
+        f"Generated: {datetime.now():%Y-%m-%d %H:%M} · stack {ACTIVE_STACK} · {len(rows)} held-out images\n",
+        "**Native** = base openbmb/MiniCPM-V-4_5 (current shipping Lever #1). ",
+        "**FT** = merged Bengali LoRA via vLLM. Same prompt + image for both.\n",
+        "> For the Bengali reviewer: for each image, which story reads more like a real "
+        "grandmother's bedtime tale (natural words, রূপকথা imagery, no English/garbled "
+        "words, calm sleepy ending)? Mark a winner per row.\n",
+        "---\n",
+    ]
+    for img, native, tn, ft, tf in rows:
+        lines += [
+            f"## {img.name}",
+            f"![{img.name}]({os.path.relpath(img, out_dir)})\n",
+            f"### Native (base) — {tn}s",
+            native or "_(empty)_", "",
+            f"### FT (LoRA) — {tf}s",
+            ft or "_(empty)_", "",
+            "**Winner (reviewer):** ☐ Native  ☐ FT  ☐ Tie  ·  notes: ____",
+            "\n---\n",
+        ]
+    fname.write_text("\n".join(lines))
+    print(f"\nReport written to {fname}")
+    print("Open it, have a Bengali speaker mark winners, and only set "
+          "FINETUNED_VISION_MODEL if FT clearly wins.")
+
+
+if __name__ == "__main__":
+    main()

finetune/gen_labels.py

@@ -0,0 +1,123 @@
+# finetune/gen_labels.py
+"""Stage 1 — generate the Bengali SFT dataset by distilling the Gemma teacher.
+
+For every drawing/photo in an images dir, ask the multimodal teacher
+(TEACHER_MODEL, e.g. gemma3:27b) to write a native Bengali bedtime story, run it
+through the purity gate, and emit MiniCPM-V fine-tuning JSON. The teacher does the
+SAME task as inference (image → Bengali story), so labels are image-grounded.
+
+Run (after `modal deploy core/modal_infra.py`):
+    uv run python finetune/gen_labels.py --images finetune/data/images \
+        --out finetune/data/train.json --teacher gemma3:27b
+
+Output:
+    <out>            kept examples (MiniCPM-V conversations format)
+    <out>.rejected   labels that failed the purity gate (for review)
+"""
+
+import argparse
+import itertools
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+import modal  # noqa: E402
+
+from core.model_config import TEACHER_MODEL, get_vision_options  # noqa: E402
+from core.prompts import STYLES, build_story_prompt  # noqa: E402
+from finetune.purity import foreign_words, is_clean  # noqa: E402
+
+IMAGE_EXTS = {".png", ".jpg", ".jpeg", ".webp"}
+# Light diversity so the student sees varied styles / instructions / names.
+_BN_STYLES = list(STYLES["bn"].keys())
+_INSTRUCTIONS = ["একটা গল্প বলো", "আমাকে একটা মজার গল্প শোনাও", "এই ছবি নিয়ে গল্প বলো", ""]
+_NAMES = ["রূপা", "অর্ক", "মিঠি", ""]
+
+
+def _teacher_story(fn, image_bytes: bytes, prompt: str) -> str:
+    """One image → Bengali story via the deployed Modal vision function."""
+    return fn.remote([image_bytes], prompt, TEACHER_MODEL, get_vision_options("bn"), None) or ""
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--images", required=True, help="dir of drawings/photos")
+    ap.add_argument("--out", default="finetune/data/train.json")
+    ap.add_argument("--teacher", default=TEACHER_MODEL, help="override TEACHER_MODEL")
+    ap.add_argument("--max-foreign", type=int, default=0, help="allowed foreign letters")
+    args = ap.parse_args()
+
+    images = sorted(
+        p for p in Path(args.images).rglob("*") if p.suffix.lower() in IMAGE_EXTS
+    )
+    if not images:
+        sys.exit(f"No images found under {args.images}")
+    print(f"Found {len(images)} images. Teacher = {args.teacher}")
+
+    out = Path(args.out)
+    rej_out = out.with_suffix(out.suffix + ".rejected")
+    out.parent.mkdir(parents=True, exist_ok=True)
+
+    # Resume: load any existing output and skip images already labelled. This makes
+    # a long run crash-safe — re-running continues where it stopped.
+    kept = json.loads(out.read_text(encoding="utf-8")) if out.exists() else []
+    rejected = json.loads(rej_out.read_text(encoding="utf-8")) if rej_out.exists() else []
+    done = {r["image"] for r in kept} | {r["image"] for r in rejected}
+    if done:
+        print(f"Resuming: {len(done)} already labelled, skipping those.")
+
+    def _flush() -> None:
+        out.write_text(json.dumps(kept, ensure_ascii=False, indent=2), encoding="utf-8")
+        rej_out.write_text(json.dumps(rejected, ensure_ascii=False, indent=2), encoding="utf-8")
+
+    fn = modal.Function.from_name("rupkotha", "run_vision_story")
+    style_cycle = itertools.cycle(_BN_STYLES)
+    instr_cycle = itertools.cycle(_INSTRUCTIONS)
+    name_cycle = itertools.cycle(_NAMES)
+
+    for i, img_path in enumerate(images):
+        if str(img_path) in done:
+            continue
+        style, instruction, name = next(style_cycle), next(instr_cycle), next(name_cycle)
+        prompt = build_story_prompt(
+            instruction=instruction, language="bn", style=style,
+            child_name=name, stack_key="B", num_images=1,
+        )
+        try:
+            story = _teacher_story(fn, img_path.read_bytes(), prompt).strip()
+        except Exception as e:  # noqa: BLE001
+            print(f"[{i}] {img_path.name}: teacher error {e}")
+            continue
+
+        ok, stats = is_clean(story, max_foreign_letters=args.max_foreign)
+        record = {
+            "id": str(i),
+            "image": str(img_path),
+            "conversations": [
+                {"role": "user", "content": "<image>\n" + prompt},
+                {"role": "assistant", "content": story},
+            ],
+        }
+        if ok:
+            kept.append(record)
+            print(f"[{i}] {img_path.name}: kept ({stats['bengali_ratio']:.2f})")
+        else:
+            record["_stats"] = stats
+            record["_leaks"] = foreign_words(story)
+            rejected.append(record)
+            print(f"[{i}] {img_path.name}: REJECTED leaks={foreign_words(story)}")
+
+        if (i + 1) % 20 == 0:  # checkpoint periodically
+            _flush()
+            print(f"  …checkpoint: {len(kept)} kept, {len(rejected)} rejected")
+
+    _flush()
+    print(f"\nKept {len(kept)} ({len(rejected)} rejected) of {len(images)} total")
+    print(f"Wrote {out}  and  {rej_out}")
+    print("⚠️  Have a Bengali speaker spot-check a sample of `kept` before training.")
+
+
+if __name__ == "__main__":
+    main()

finetune/merge.py

@@ -0,0 +1,90 @@
+# finetune/merge.py
+"""Stage 3 — merge the trained LoRA adapter into full MiniCPM-V weights.
+
+vLLM serves a full model, not a base+adapter pair, so we bake the adapter in.
+Output is a standalone HF model dir that finetune/serve_vllm.py loads.
+
+Run:
+    uv run modal run finetune/merge.py
+"""
+
+import modal
+
+from core.model_config import STUDENT_BASE_REPO
+
+app = modal.App("rupkotha-finetune-merge")
+
+_vol = modal.Volume.from_name("rupkotha-finetune", create_if_missing=True)
+_hf = modal.Volume.from_name("rupkotha-hf", create_if_missing=True)
+
+# Mirror finetune/train_lora.py's dependency matrix so the adapter loads against the
+# same stack it was trained on: transformers 4.x (the 5.x line breaks MiniCPM-V's
+# remote-code loader), peft<0.19 (SWIFT 3.12.6's pin), torch>=2.4 (4.5 remote code),
+# and timm/sentencepiece for the vision tower + tokenizer.
+_image = (
+    modal.Image.debian_slim(python_version="3.10")
+    .pip_install(
+        "torch>=2.4",
+        "transformers>=4.49,<4.58",
+        "peft>=0.11,<0.19",
+        "accelerate",
+        "timm",
+        "sentencepiece",
+        "pillow",
+    )
+    # merge.py imports core.model_config at module load; make `core` importable
+    # in the container (`modal run <file>` doesn't auto-mount the project).
+    .add_local_python_source("core")
+)
+
+
+@app.function(
+    gpu="A100-40GB",
+    image=_image,
+    volumes={"/data": _vol, "/root/.cache/huggingface": _hf},
+    secrets=[modal.Secret.from_name("algaeguard-secrets")],
+    timeout=60 * 30,
+)
+def merge() -> str:
+    import torch
+    from peft import PeftModel
+    from transformers import AutoModel, AutoProcessor, AutoTokenizer
+
+    adapter_dir = "/data/out/lora-bengali"
+    merged_dir = "/data/out/minicpm-v-bengali-merged"
+
+    # Defensive: ensure the base's remote-code .py files are present in the shared
+    # HF cache (they were cached weights-first by vLLM; train_lora.py repairs them,
+    # but keep merge self-contained). Cheap — weights stay cached, only code refetched.
+    from huggingface_hub import snapshot_download
+    snapshot_download(
+        STUDENT_BASE_REPO,
+        allow_patterns=["*.py", "*.json", "*.txt", "*.model", "tokenizer*"],
+        force_download=True,
+    )
+
+    # MiniCPM-V loads with trust_remote_code; it's an AutoModel (custom class).
+    base = AutoModel.from_pretrained(
+        STUDENT_BASE_REPO, trust_remote_code=True, torch_dtype=torch.bfloat16
+    )
+    tokenizer = AutoTokenizer.from_pretrained(STUDENT_BASE_REPO, trust_remote_code=True)
+
+    model = PeftModel.from_pretrained(base, adapter_dir)
+    model = model.merge_and_unload()  # fold LoRA into the base weights
+
+    model.save_pretrained(merged_dir, safe_serialization=True)
+    tokenizer.save_pretrained(merged_dir)
+    # MiniCPM-V is multimodal: vLLM needs the image processor too (preprocessor_
+    # config.json + processing code). Without it vLLM finds only the tokenizer and
+    # rejects it ("Invalid type of HuggingFace processor"). Copy it from the base.
+    processor = AutoProcessor.from_pretrained(STUDENT_BASE_REPO, trust_remote_code=True)
+    processor.save_pretrained(merged_dir)
+    _vol.commit()
+    return merged_dir
+
+
+@app.local_entrypoint()
+def main():
+    path = merge.remote()
+    print(f"Merged model written to volume rupkotha-finetune at {path}")
+    print("Next: set FINETUNED_VISION_MODEL and serve via finetune/serve_vllm.py.")

finetune/preprocess.py

@@ -0,0 +1,139 @@
+# finetune/preprocess.py
+"""Stage 0.5 — clean the collected image pool into a training-ready set.
+
+Sweeps finetune/data/images/**, and for each image:
+  1. optional fixed fractional crop (per source) — strips page furniture like the
+     "Kye Drawing for student" header / signature footer on the village scans;
+  2. auto-trim near-uniform borders (white margins on doodles & scans);
+  3. resize so the longest side ≤ MAX_SIZE;
+  4. RGB-normalise and re-encode;
+  5. drop near-duplicates via a 64-bit average hash (Hamming distance).
+
+Outputs finetune/data/processed/<source>/<name>.jpg + a manifest.json. The
+labeler (gen_labels.py) can then point at finetune/data/processed.
+
+Run:
+    uv run python finetune/preprocess.py            # process everything
+    uv run python finetune/preprocess.py --max-size 1024 --dedupe-distance 4
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+from PIL import Image, ImageChops
+
+ROOT = Path(__file__).resolve().parent
+SRC = ROOT / "data" / "images"
+DST = ROOT / "data" / "processed"
+
+# Per-source fractional crop (top, bottom, left, right) to remove page furniture.
+# "root" = images placed directly in data/images/ (the village drawing-book scans).
+CROP_FRACTIONS: dict[str, tuple[float, float, float, float]] = {
+    "root": (0.07, 0.06, 0.02, 0.02),  # trim header text + signature/date footer
+}
+
+
+def _source_of(path: Path) -> str:
+    rel = path.relative_to(SRC)
+    return rel.parts[0] if len(rel.parts) > 1 else "root"
+
+
+def _frac_crop(img: Image.Image, fracs: tuple[float, float, float, float]) -> Image.Image:
+    t, b, l, r = fracs
+    w, h = img.size
+    box = (int(w * l), int(h * t), int(w * (1 - r)), int(h * (1 - b)))
+    return img.crop(box) if box[2] > box[0] and box[3] > box[1] else img
+
+
+def _autotrim(img: Image.Image, tol: int = 18) -> Image.Image:
+    """Trim a near-uniform border using the top-left pixel as the background."""
+    bg = Image.new("RGB", img.size, img.getpixel((0, 0)))
+    diff = ImageChops.difference(img, bg).convert("L").point(lambda p: 255 if p > tol else 0)
+    bbox = diff.getbbox()
+    return img.crop(bbox) if bbox else img
+
+
+def _resize(img: Image.Image, max_size: int) -> Image.Image:
+    w, h = img.size
+    scale = max_size / max(w, h)
+    if scale < 1:
+        img = img.resize((max(1, int(w * scale)), max(1, int(h * scale))), Image.LANCZOS)
+    return img
+
+
+def _ahash(img: Image.Image) -> int:
+    """64-bit average hash for near-duplicate detection (no extra deps)."""
+    small = img.convert("L").resize((8, 8), Image.LANCZOS)
+    px = list(small.getdata())
+    avg = sum(px) / len(px)
+    bits = 0
+    for i, p in enumerate(px):
+        if p >= avg:
+            bits |= 1 << i
+    return bits
+
+
+def _hamming(a: int, b: int) -> int:
+    return bin(a ^ b).count("1")
+
+
+def process(max_size: int, dedupe_distance: int) -> None:
+    exts = {".png", ".jpg", ".jpeg", ".webp"}
+    paths = sorted(p for p in SRC.rglob("*") if p.suffix.lower() in exts)
+    if not paths:
+        raise SystemExit(f"No images under {SRC}")
+
+    DST.mkdir(parents=True, exist_ok=True)
+    hashes: list[int] = []
+    manifest, kept, dups, errors = [], 0, 0, 0
+
+    for p in paths:
+        source = _source_of(p)
+        try:
+            img = Image.open(p).convert("RGB")
+            img = _frac_crop(img, CROP_FRACTIONS.get(source, (0, 0, 0, 0)))
+            img = _autotrim(img)
+            img = _resize(img, max_size)
+        except Exception as e:  # noqa: BLE001
+            print(f"  skip {p.name}: {e}")
+            errors += 1
+            continue
+
+        h = _ahash(img)
+        if any(_hamming(h, prev) <= dedupe_distance for prev in hashes):
+            dups += 1
+            continue
+        hashes.append(h)
+
+        out_dir = DST / source
+        out_dir.mkdir(parents=True, exist_ok=True)
+        out_path = out_dir / f"{p.stem}.jpg"
+        img.save(out_path, "JPEG", quality=90)
+        manifest.append({"processed": str(out_path.relative_to(ROOT)),
+                         "source_image": str(p.relative_to(ROOT)), "source": source})
+        kept += 1
+
+    (DST / "manifest.json").write_text(
+        json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8"
+    )
+    print(f"\nProcessed {len(paths)} → kept {kept}, dropped {dups} dups, {errors} errors")
+    by_src: dict[str, int] = {}
+    for m in manifest:
+        by_src[m["source"]] = by_src.get(m["source"], 0) + 1
+    for s, c in sorted(by_src.items()):
+        print(f"  {s}: {c}")
+    print(f"Output: {DST}  (+ manifest.json)")
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--max-size", type=int, default=1024)
+    ap.add_argument("--dedupe-distance", type=int, default=4,
+                    help="max aHash Hamming distance to treat as duplicate (0=exact)")
+    args = ap.parse_args()
+    process(args.max_size, args.dedupe_distance)
+
+
+if __name__ == "__main__":
+    main()

finetune/purity.py

@@ -0,0 +1,82 @@
+# finetune/purity.py
+"""Bengali script-purity quality gate for teacher labels.
+
+The teacher (Gemma) writes mostly clean Bengali, but occasionally code-switches
+(e.g. a stray Latin or Cyrillic word — we saw `зеленая` leak once). Distillation
+caps the student at the label quality, so we filter/repair bad labels before
+training. This is the gate referenced in the Bengali-quality investigation.
+
+Heuristics only — no model needed. A human Bengali speaker should still spot-check
+a sample of what passes.
+"""
+
+import re
+import unicodedata
+
+# Bengali Unicode block: U+0980–U+09FF.
+_BENGALI = re.compile(r"[ঀ-৿]")
+# Letters from other scripts that must NOT appear in a Bengali story body.
+_LATIN = re.compile(r"[A-Za-z]")
+_CYRILLIC = re.compile(r"[Ѐ-ӿ]")
+# Allowed non-letter noise: digits, whitespace, common punctuation, emoji, danda.
+_ALLOWED_NONLETTER = re.compile(r"[\s\d\.,!?…\"'“”‘’—\-–—:;()।॥☀-➿\U0001F300-\U0001FAFF]")
+
+
+def script_stats(text: str) -> dict:
+    """Counts of Bengali vs foreign letters and the Bengali-letter ratio."""
+    text = unicodedata.normalize("NFC", text or "")
+    bengali = len(_BENGALI.findall(text))
+    latin = len(_LATIN.findall(text))
+    cyrillic = len(_CYRILLIC.findall(text))
+    letters = bengali + latin + cyrillic
+    ratio = (bengali / letters) if letters else 0.0
+    return {
+        "bengali": bengali,
+        "latin": latin,
+        "cyrillic": cyrillic,
+        "foreign": latin + cyrillic,
+        "bengali_ratio": round(ratio, 4),
+    }
+
+
+def is_clean(
+    text: str,
+    min_bengali_ratio: float = 0.98,
+    max_foreign_letters: int = 0,
+    min_length: int = 40,
+) -> tuple[bool, dict]:
+    """Decide whether a teacher label is clean enough to train on.
+
+    Defaults are strict: essentially zero foreign-script letters. Loosen
+    max_foreign_letters to 1–2 if you'd rather repair than drop.
+
+    Returns (ok, stats).
+    """
+    stats = script_stats(text)
+    ok = (
+        len((text or "").strip()) >= min_length
+        and stats["foreign"] <= max_foreign_letters
+        and stats["bengali_ratio"] >= min_bengali_ratio
+    )
+    return ok, stats
+
+
+def foreign_words(text: str) -> list[str]:
+    """Return whitespace tokens that contain any Latin/Cyrillic letter — useful
+    for eyeballing exactly what leaked (e.g. ['зеленая'])."""
+    out = []
+    for tok in (text or "").split():
+        if _LATIN.search(tok) or _CYRILLIC.search(tok):
+            out.append(tok)
+    return out
+
+
+if __name__ == "__main__":
+    samples = {
+        "good": "আচ্ছা রূপা, চোখ বুজে নাও। চাঁদমামা হাসছে, পুকুরের ধারে ঘাস দুলছে। শুভরাত্রি।",
+        "leak": "দেখছো зеленая গোল বলটা? ওটা রূপার প্রিয় খেলনা সবুজ আপেল!",
+        "english": "Once upon a time there was a small red house under the sun.",
+    }
+    for name, s in samples.items():
+        ok, stats = is_clean(s)
+        print(f"{name:8} ok={ok!s:5} {stats} leaks={foreign_words(s)}")

finetune/serve_vllm.py

@@ -0,0 +1,119 @@
+# finetune/serve_vllm.py
+"""Stage 4 — serve the merged fine-tuned MiniCPM-V via vLLM on Modal.
+
+Chosen over GGUF→Ollama because custom MiniCPM-V → GGUF has known vision (mmproj)
+breakage. vLLM loads the merged HF model directly with trust_remote_code and
+handles the multimodal input natively.
+
+This exposes `run_vision_story_ft(image_bytes_list, prompt)` — a drop-in for the
+Bengali vision path. To route the app to it, in core/vision_story.py call this
+function (via a small dispatcher in core/modal_infra.py) when a fine-tuned model
+is active, i.e. when model_config.FINETUNED_VISION_MODEL is set.
+
+Run / deploy:
+    uv run modal deploy finetune/serve_vllm.py
+"""
+
+import modal
+
+from core.model_config import MODAL_MIN_CONTAINERS
+
+app = modal.App("rupkotha-ft-serve")
+
+_vol = modal.Volume.from_name("rupkotha-finetune", create_if_missing=True)
+_MERGED_DIR = "/data/out/minicpm-v-bengali-merged"
+
+# vLLM 0.10.2 is the version the official MiniCPM-V CookBook pins for 4.5. It only
+# requires transformers>=4.55.2 (no upper bound), so pip otherwise grabs transformers
+# 5.x — whose tokenizer backend breaks vLLM (TokenizersBackend.all_special_tokens_
+# extended) AND MiniCPM-V's remote code. Pin transformers==4.57.6 (same 4.x the merge
+# step used with this model) to stay compatible with both.
+_vllm_image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .pip_install("vllm==0.10.2", "transformers==4.57.6", "pillow")
+    # serve_vllm.py imports core.model_config at module load; make `core`
+    # importable in the container (current Modal doesn't auto-mount the project).
+    .add_local_python_source("core")
+)
+
+_engine: dict = {}
+
+
+@app.function(
+    # A100-40GB, not A10G: the full bf16 8B weights (~16GB) + vision encoder + vLLM
+    # overhead leave no room for the KV cache on a 24GB A10G (CUDA OOM at startup).
+    # 40GB gives comfortable headroom at max_model_len=4096. Scales to zero when idle.
+    gpu="A100-40GB",
+    image=_vllm_image,
+    volumes={"/data": _vol},
+    timeout=600,
+    min_containers=MODAL_MIN_CONTAINERS,  # keep-warm honors model_config toggle
+)
+def run_vision_story_ft(image_bytes_list: list[bytes], prompt: str) -> str:
+    """Generate a story from the fine-tuned model via vLLM. Mirrors the contract
+    of core/modal_infra.run_vision_story (returns text, '' on failure)."""
+    import io
+
+    from PIL import Image
+    from transformers import AutoTokenizer
+    from vllm import LLM, SamplingParams
+
+    if "llm" not in _engine:
+        # max_model_len 4096 matches the SWIFT training length: the verbose Bengali
+        # prompt + expanded image tokens exceed 2048 for some inputs.
+        _engine["llm"] = LLM(
+            model=_MERGED_DIR,
+            trust_remote_code=True,
+            max_model_len=4096,
+            disable_mm_preprocessor_cache=True,
+            limit_mm_per_prompt={"image": 4},
+        )
+        tok = AutoTokenizer.from_pretrained(_MERGED_DIR, trust_remote_code=True)
+        _engine["tok"] = tok
+        # Stop on MiniCPM-V's chat-end tokens so we don't bleed past the story.
+        stop = ["<|im_end|>", "<|endoftext|>"]
+        _engine["sp"] = SamplingParams(
+            temperature=0.45, top_p=0.9, max_tokens=768,
+            stop_token_ids=[tok.convert_tokens_to_ids(t) for t in stop],
+        )
+
+    llm = _engine["llm"]
+    tok = _engine["tok"]
+    images = [Image.open(io.BytesIO(b)).convert("RGB") for b in (image_bytes_list or [])]
+    # MiniCPM-V 4.5: one (<image>./</image>) placeholder per image, then the text.
+    # Wrap via the model's chat template (add_generation_prompt) so inference matches
+    # the minicpmv4_5 template the LoRA was trained under — critical for quality.
+    placeholders = "".join("(<image>./</image>)\n" for _ in images)
+    messages = [{"role": "user", "content": placeholders + prompt}]
+    # MiniCPM-V 4.5 has a hybrid thinking mode that prepends <think>...</think> and
+    # eats the token budget. Disable it (best-effort — older templates lack the kwarg).
+    try:
+        text = tok.apply_chat_template(
+            messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
+        )
+    except TypeError:
+        text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    try:
+        out = llm.generate(
+            {"prompt": text, "multi_modal_data": {"image": images}},
+            _engine["sp"],
+        )
+        story = (out[0].outputs[0].text or "").strip()
+        # Strip any residual think block the model still emits.
+        import re
+        story = re.sub(r"^\s*<think>.*?</think>\s*", "", story, flags=re.DOTALL).strip()
+        return story
+    except Exception as e:  # noqa: BLE001
+        print(f"[serve_vllm] generation failed: {e}")
+        return ""
+
+
+def generate_story_ft_remote(images_b64: list[str], prompt: str) -> str:
+    """Plain-callable dispatcher (import into core/modal_infra.py to route the
+    Bengali path here when FINETUNED_VISION_MODEL is set). Mirrors
+    generate_story_remote so core/vision_story.py needs only a one-line branch."""
+    import base64
+
+    image_bytes = [base64.b64decode(b) for b in (images_b64 or [])]
+    fn = modal.Function.from_name("rupkotha-ft-serve", "run_vision_story_ft")
+    return fn.remote(image_bytes, prompt)

finetune/train_lora.py

@@ -0,0 +1,166 @@
+# finetune/train_lora.py
+"""Stage 2 — LoRA fine-tune MiniCPM-V 4.5 on the distilled Bengali dataset.
+
+Uses **ms-SWIFT** (modelscope/ms-swift) rather than OpenBMB's official finetune
+scripts. The official path hit a hard dependency wall: its requirements pin torch
+2.1.2, but MiniCPM-V 4.5's remote code needs torch>=2.4 (`torch.library.custom_op`),
+and the repo's finetune.py targets older MiniCPM releases (4.0/2.6/2.5), not 4.5.
+SWIFT ships a maintained recipe for this exact model — `model_type=minicpmv4_5`,
+`template=minicpmv4_5`, deps `timm, transformers>=4.36, decord` — so the version
+matrix is solved for us.
+
+Vision encoder frozen (`--freeze_vit true`); LoRA on the LLM self-attention layers
+only (`q/k/v/o_proj`) — that's the weak-Bengali part. ViT and aligner stay frozen.
+
+Inputs  (Modal Volume `rupkotha-finetune`):  /train.json  +  /labelset/*.jpg
+        (train.json 'image' fields are container-absolute: /data/labelset/<name>)
+        Converted in-container to SWIFT's messages/images JSONL schema.
+Output  (same volume):  /out/lora-bengali/   (PEFT adapter — feed to finetune/merge.py)
+
+Run a SHORT validation first to surface integration errors cheaply:
+    uv run modal run finetune/train_lora.py --max-steps 4
+Then the full run:
+    uv run modal run finetune/train_lora.py
+"""
+
+import modal
+
+from core.model_config import STUDENT_BASE_REPO  # openbmb/MiniCPM-V-4_5
+
+app = modal.App("rupkotha-finetune")
+
+_vol = modal.Volume.from_name("rupkotha-finetune", create_if_missing=True)
+_hf = modal.Volume.from_name("rupkotha-hf", create_if_missing=True)
+
+# ms-SWIFT brings its own pinned transformers/peft/accelerate; we only add the
+# MiniCPM-V 4.5 model extras (timm, decord) and force torch>=2.4 for its remote
+# code. No deepspeed/nvcc needed — a single A100-80GB fits an 8B base + LoRA, and
+# attn defaults to sdpa (no flash-attn build), so debian_slim + pip wheels suffice.
+# Pin ms-swift 3.12.6 (last 3.x): it lists model_type `minicpmv4_5` AND pins
+# transformers>=4.33,<4.58. The 4.x line pulls transformers 5.x, whose remote-code
+# loader follows the HF-cache symlink into blobs/ and fails to resolve MiniCPM-V's
+# relative imports (modeling_navit_siglip.py). 4.x transformers loads it cleanly.
+_train_image = (
+    modal.Image.debian_slim(python_version="3.10")
+    .apt_install("git")
+    .pip_install(
+        "ms-swift==3.12.6",
+        "torch>=2.4",
+        "timm",
+        "decord",
+        "pillow",
+        "sentencepiece",
+    )
+    # train_lora.py imports core.model_config at module load; `modal run <file>`
+    # doesn't auto-mount the project, so make `core` importable in the container.
+    .add_local_python_source("core")
+)
+
+
+@app.function(
+    gpu="A100-80GB",  # headroom for an 8B base + LoRA on one GPU
+    image=_train_image,
+    volumes={"/data": _vol, "/root/.cache/huggingface": _hf},
+    secrets=[modal.Secret.from_name("algaeguard-secrets")],  # HF_TOKEN for the base
+    timeout=60 * 60 * 6,
+)
+def train(max_steps: int = 0) -> str:
+    import glob
+    import json
+    import os
+    import shutil
+    import subprocess
+
+    # SWIFT defaults to ModelScope; force HuggingFace so it pulls openbmb/MiniCPM-V-4_5
+    # (and reuses the HF_TOKEN from the algaeguard-secrets secret).
+    os.environ["USE_HF"] = "1"
+    # Mirror the original max_slice_nums=9 (SWIFT reads this env for MiniCPM-V).
+    os.environ.setdefault("MAX_SLICE_NUMS", "9")
+
+    # Repair the shared HF cache: the base model was cached weights-first (vLLM),
+    # leaving the MiniCPM-V remote-code .py files as dangling symlinks (e.g.
+    # modeling_navit_siglip.py). Refetch the small code/config files cleanly so
+    # trust_remote_code loads; the .safetensors weights stay cached untouched.
+    from huggingface_hub import snapshot_download
+    snapshot_download(
+        STUDENT_BASE_REPO,
+        allow_patterns=["*.py", "*.json", "*.txt", "*.model", "tokenizer*"],
+        force_download=True,
+    )
+
+    # ── Convert MiniCPM conversations format → SWIFT messages/images JSONL ──
+    # Source rows already carry role/content turns with a leading <image> in the
+    # user content and a container-absolute image path. SWIFT wants `messages` +
+    # an `images` list (one path per <image> placeholder).
+    src = "/data/train.json"  # volume copy: image paths already /data/labelset/<name>
+    swift_data = "/data/train_swift.jsonl"
+    rows = json.load(open(src))
+    with open(swift_data, "w") as f:
+        for r in rows:
+            f.write(json.dumps(
+                {"messages": r["conversations"], "images": [r["image"]]},
+                ensure_ascii=False,
+            ) + "\n")
+    print(f"Converted {len(rows)} examples → {swift_data}")
+
+    adapter_dir = "/data/out/lora-bengali"     # canonical path merge.py expects
+    swift_out = "/data/out/swift-runs"         # SWIFT writes vX-<ts>/checkpoint-N here
+
+    # Mirror finetune_lora.sh intent via SWIFT's CLI: vision frozen, LoRA r=16 on
+    # the LLM self-attention projections only.
+    cmd = [
+        "swift", "sft",
+        "--model", STUDENT_BASE_REPO,
+        "--model_type", "minicpmv4_5",
+        "--train_type", "lora",
+        "--dataset", swift_data,
+        "--freeze_vit", "true",
+        "--target_modules", "q_proj", "k_proj", "v_proj", "o_proj",
+        "--lora_rank", "16", "--lora_alpha", "32", "--lora_dropout", "0.05",
+        "--torch_dtype", "bfloat16",
+        # 4096, not 2048: the verbose Bengali prompt + MiniCPM image slices
+        # (max_slice_nums=9) push some rows to ~2083 tokens. At 2048 SWIFT raises
+        # MaxLengthError per over-long row and silently drops it from training;
+        # 4096 keeps all 389 samples (peak mem was only ~28 GiB of 80).
+        "--max_length", "4096",
+        "--per_device_train_batch_size", "1",
+        "--gradient_accumulation_steps", "8",
+        "--learning_rate", "1e-4",
+        "--gradient_checkpointing", "true",
+        "--save_strategy", "steps", "--save_steps", "200", "--save_total_limit", "2",
+        "--logging_steps", "5", "--report_to", "none",
+        "--dataloader_num_workers", "4",
+        "--output_dir", swift_out,
+    ]
+    # Short validation run vs full run.
+    if max_steps and max_steps > 0:
+        cmd += ["--max_steps", str(max_steps)]
+    else:
+        cmd += ["--num_train_epochs", "3"]
+
+    print("Running:", " ".join(cmd))
+    subprocess.run(cmd, check=True)
+
+    # SWIFT nests output under output_dir/<version>/checkpoint-<step>/. Find the
+    # latest dir that actually holds a PEFT adapter and copy it to the canonical
+    # path so finetune/merge.py (which loads /data/out/lora-bengali) works unchanged.
+    adapters = glob.glob(f"{swift_out}/**/adapter_config.json", recursive=True)
+    if not adapters:
+        raise RuntimeError(f"No adapter produced under {swift_out}")
+    final_ckpt = max(
+        (os.path.dirname(p) for p in adapters), key=os.path.getmtime
+    )
+    print(f"Final adapter checkpoint: {final_ckpt}")
+    if os.path.exists(adapter_dir):
+        shutil.rmtree(adapter_dir)
+    shutil.copytree(final_ckpt, adapter_dir)
+
+    _vol.commit()
+    return adapter_dir
+
+
+@app.local_entrypoint()
+def main(max_steps: int = 0):
+    path = train.remote(max_steps=max_steps)
+    print(f"LoRA adapter written to volume rupkotha-finetune at {path}")
+    print("Next: finetune/merge.py to fold the adapter into full weights.")

pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "rupkatha"
+version = "0.1.0"
+requires-python = ">=3.10,<3.13"
+dependencies = [
+    "gradio>=6.17.3",
+    "modal>=1.5.0",
+]
+
+[dependency-groups]
+finetune = [
+    "datasets>=5.0.0",
+]

requirements.txt

@@ -0,0 +1,8 @@
+# Hugging Face Space (thin client) dependencies ONLY.
+# The Space runs app.py, which calls Modal for all inference — it holds zero model
+# weights and imports no ML libraries locally. So this is intentionally minimal:
+# just Gradio (the UI) and the Modal client (to invoke the deployed functions).
+# The heavy deps (torch, vllm, transformers, coqui-tts, …) live in the Modal images
+# in core/modal_infra.py + finetune/, NOT here. Local dev uses uv (pyproject/uv.lock).
+gradio==6.17.3
+modal>=1.5