finetune/README.md

Bengali distillation fine-tuning (finetune/)

Improve Stack A's weak Bengali by fine-tuning MiniCPM-V 4.5 itself — the strongest OpenBMB-prize story ("we improved the OpenBMB model for Bengali"). This stays on Stack A; only the served model weights change.

Why fine-tune and not the IndicTrans2 pivot? The owner judged the pivot's translated register worse than native output. The native ceiling is MiniCPM's Bengali generation; fine-tuning on great Bengali raises that ceiling while keeping the warm native register. See the bengali-quality-investigation memory.

The pipeline

drawings ──► [Gemma teacher] ──► Bengali stories ──► [purity gate] ──► train.json
                                                                          │
                              LoRA fine-tune MiniCPM-V (vision frozen) ◄───┘
                                          │
                                    merge adapter
                                          │
                                  serve via vLLM ──► wire into Bengali path

Stage	File	Where	Notes
0. Source images	(manual, see below)	local	the real bottleneck
1. Label + filter	gen_labels.py + purity.py	Modal (Gemma)	image → Bengali story, drop leaks
2. LoRA train	train_lora.py	Modal A100	ms-SWIFT (model_type=minicpmv4_5)
3. Merge	merge.py	Modal A100	adapter → full weights
4. Serve	serve_vllm.py	Modal A10G	vLLM, no GGUF
5. Wire	one branch in core/vision_story.py	local	route Bengali to the FT model

Stage 0 — source the drawings (finetune/data/images/)

Target ~300–500 diverse, on-distribution images. Mix all four:

• Team-drawn crayon pics — photograph ~100–150. Most authentic; do these first.
• Quick, Draw! — Google's doodle dataset (https://quickdraw.withgoogle.com/data); pull a few categories (.ndjson/.npy), render strokes to PNG, optionally composite 2–3 doodles into a scene.
• Synthetic crayon — an image model (e.g. SDXL) with prompts like "a child's crayon drawing of a house and the sun, simple, on white paper".
• Toy / play photos — stock or self-taken photos of toys/equipment.

Quality > quantity for LoRA. Keep them varied (objects, colours, scenes).

Stage 1 — generate labels

modal deploy core/modal_infra.py        # teacher served via run_vision_story
uv run python finetune/gen_labels.py --images finetune/data/images \
    --out finetune/data/train.json --teacher gemma3:27b

Produces train.json (kept) and train.json.rejected (failed the gate). Have a Bengali speaker spot-check kept — distillation caps the student at label quality. Tune strictness in purity.py (is_clean(max_foreign_letters=...)).

Stage 2 — LoRA train

Upload finetune/data/ to the rupkotha-finetune Modal Volume, then validate cheaply before the full run:

uv run modal run finetune/train_lora.py --max-steps 4   # smoke test
uv run modal run finetune/train_lora.py                 # full 3-epoch run

train_lora.py drives ms-SWIFT (swift sft --model_type minicpmv4_5), not OpenBMB's official finetune.py — the latter pins torch 2.1.2 while MiniCPM-V 4.5 remote code needs torch>=2.4, and its finetune path doesn't target 4.5. SWIFT ships a maintained recipe for this model so the dep matrix is solved. It converts our MiniCPM conversations JSON to SWIFT's messages/images JSONL in-container. Vision encoder frozen (--freeze_vit true), LoRA r=16 on the LLM q/k/v/o_proj. SWIFT's nested checkpoint is copied to out/lora-bengali/ so Stage 3 is unchanged.

Stage 3 — merge

uv run modal run finetune/merge.py      # → out/minicpm-v-bengali-merged/

Stage 4 — serve + Stage 5 — wire

uv run modal deploy finetune/serve_vllm.py

Then set FINETUNED_VISION_MODEL in core/model_config.py to the merged path and add a one-line branch in core/vision_story.py: when it's set and language=="bn", call serve_vllm.generate_story_ft_remote(...) instead of generate_story_remote. (run_translate/pivot plumbing stays available as a fallback.)

Evaluate

Hold out ~20 drawings; compare FT Bengali vs native Lever-#1 (demo_story_bn_v2.wav) with a Bengali speaker. Ship the FT model only if it clearly wins.

Cost / risk (hackathon reality)

Heaviest path: image curation (days) + ~1–3h A100 training + integration. Validated upside: the Gemma teacher writes genuinely native Bengali (চাঁদমামা/পুকুর/খই), so the student has an excellent target. Main risk is data volume/quality, not the method.