finetune/README.md

# Bengali distillation fine-tuning (`finetune/`)

Improve Stack A's weak Bengali by **fine-tuning MiniCPM-V 4.5 itself** — the
strongest OpenBMB-prize story (*"we improved the OpenBMB model for Bengali"*).
This stays on Stack A; only the served model weights change.

> Why fine-tune and not the IndicTrans2 pivot? The owner judged the pivot's
> translated register worse than native output. The native ceiling is MiniCPM's
> Bengali *generation*; fine-tuning on great Bengali raises that ceiling while
> keeping the warm native register. See the `bengali-quality-investigation` memory.

## The pipeline

```
drawings ──► [Gemma teacher] ──► Bengali stories ──► [purity gate] ──► train.json
                                                                          │
                              LoRA fine-tune MiniCPM-V (vision frozen) ◄───┘
                                          │
                                    merge adapter
                                          │
                                  serve via vLLM ──► wire into Bengali path
```

| Stage | File | Where | Notes |
|---|---|---|---|
| 0. Source images | (manual, see below) | local | the real bottleneck |
| 1. Label + filter | `gen_labels.py` + `purity.py` | Modal (Gemma) | image → Bengali story, drop leaks |
| 2. LoRA train | `train_lora.py` | Modal A100 | ms-SWIFT (`model_type=minicpmv4_5`) |
| 3. Merge | `merge.py` | Modal A100 | adapter → full weights |
| 4. Serve | `serve_vllm.py` | Modal A10G | vLLM, no GGUF |
| 5. Wire | one branch in `core/vision_story.py` | local | route Bengali to the FT model |

## Stage 0 — source the drawings (`finetune/data/images/`)
Target ~300–500 diverse, on-distribution images. Mix all four:
- **Team-drawn crayon pics** — photograph ~100–150. Most authentic; do these first.
- **Quick, Draw!** — Google's doodle dataset (`https://quickdraw.withgoogle.com/data`);
  pull a few categories (`.ndjson`/`.npy`), render strokes to PNG, optionally
  composite 2–3 doodles into a scene.
- **Synthetic crayon** — an image model (e.g. SDXL) with prompts like
  *"a child's crayon drawing of a house and the sun, simple, on white paper"*.
- **Toy / play photos** — stock or self-taken photos of toys/equipment.

Quality > quantity for LoRA. Keep them varied (objects, colours, scenes).

## Stage 1 — generate labels
```bash
modal deploy core/modal_infra.py        # teacher served via run_vision_story
uv run python finetune/gen_labels.py --images finetune/data/images \
    --out finetune/data/train.json --teacher gemma3:27b
```
Produces `train.json` (kept) and `train.json.rejected` (failed the gate). **Have a
Bengali speaker spot-check `kept`** — distillation caps the student at label quality.
Tune strictness in `purity.py` (`is_clean(max_foreign_letters=...)`).

## Stage 2 — LoRA train
Upload `finetune/data/` to the `rupkotha-finetune` Modal Volume, then validate
cheaply before the full run:
```bash
uv run modal run finetune/train_lora.py --max-steps 4   # smoke test
uv run modal run finetune/train_lora.py                 # full 3-epoch run
```
`train_lora.py` drives **ms-SWIFT** (`swift sft --model_type minicpmv4_5`), not
OpenBMB's official `finetune.py` — the latter pins torch 2.1.2 while MiniCPM-V 4.5
remote code needs torch>=2.4, and its finetune path doesn't target 4.5. SWIFT ships
a maintained recipe for this model so the dep matrix is solved. It converts our
MiniCPM `conversations` JSON to SWIFT's `messages`/`images` JSONL in-container.
Vision encoder frozen (`--freeze_vit true`), LoRA r=16 on the LLM `q/k/v/o_proj`.
SWIFT's nested checkpoint is copied to `out/lora-bengali/` so Stage 3 is unchanged.

## Stage 3 — merge
```bash
uv run modal run finetune/merge.py      # → out/minicpm-v-bengali-merged/
```

## Stage 4 — serve + Stage 5 — wire
```bash
uv run modal deploy finetune/serve_vllm.py
```
Then set `FINETUNED_VISION_MODEL` in `core/model_config.py` to the merged path and
add a one-line branch in `core/vision_story.py`: when it's set and `language=="bn"`,
call `serve_vllm.generate_story_ft_remote(...)` instead of `generate_story_remote`.
(`run_translate`/pivot plumbing stays available as a fallback.)

## Evaluate
Hold out ~20 drawings; compare FT Bengali vs native Lever-#1 (`demo_story_bn_v2.wav`)
with a Bengali speaker. Ship the FT model only if it clearly wins.

## Cost / risk (hackathon reality)
Heaviest path: image curation (days) + ~1–3h A100 training + integration. Validated
upside: the Gemma teacher writes genuinely native Bengali (চাঁদমামা/পুকুর/খই), so the
student has an excellent target. Main risk is data volume/quality, not the method.