docs/FINETUNE_VISION.md

Fine-Tuning the Rune Goblin Vision Model (MiniCPM-V 4.6)

How to fine-tune a vision-language model to read hand-drawn RuneLang glyphs from a canvas image + game state and emit visual_reading + spell JSON.

TL;DR

1. The GGUF you downloaded is inference-only — fine-tuning needs the full safetensors model openbmb/MiniCPM-V-4.6 (~2.6 GB).
2. Convert the dataset to absolute image paths before training; the Modal notebook includes the conversion cell.
3. Train LoRA with ms-swift (one command) or LLaMA-Factory (ready config).
4. (Optional) merge LoRA → convert to GGUF to serve in the game via llama.cpp.

Running on Modal? Use the ready notebook notebooks/MODAL-run-p1Jun2026.ipynb — a single Run-All notebook (ms-swift based) for a Modal Notebook GPU, with an appendix modal run script for unattended jobs. This doc is the framework-agnostic reference behind it.

---

0. Critical: GGUF cannot be fine-tuned

Artifact	What it's for
models/MiniCPM-V-4.6-gguf/MiniCPM-V-4_6-Q8_0.gguf + mmproj-model-f16.gguf	Serving / inference with llama.cpp / Ollama. Quantized — not trainable.
openbmb/MiniCPM-V-4.6 (safetensors, 2.6 GB)	The fine-tuning base. This is what you train.

So the flow is: train on the safetensors model → (optionally) export back to GGUF for fast game-time inference.

Download the trainable base into models/:

cd /home/ashu/github/goblin
source .env   # HF_TOKEN
uv run python - <<'PY'
import os
from huggingface_hub import snapshot_download
snapshot_download("openbmb/MiniCPM-V-4.6",
                  local_dir="models/MiniCPM-V-4.6",
                  token=os.environ.get("HF_TOKEN"),
                  ignore_patterns=["*.pth","*.onnx"])
PY

Model facts (drives VRAM planning)

openbmb/MiniCPM-V-4.6 is the lightweight / on-device member of the family:

• Total ≈ 2.6 B params (single model.safetensors, 2.6 GB bf16)
• Text backbone: qwen3_5_text — hidden 1024, 24 layers, vocab 248k
• Vision encoder: SigLIP-style — hidden 1152, 27 layers, max image 980px
• Arch class: MiniCPMV4_6ForConditionalGeneration (needs trust_remote_code)

This easily fits LoRA in bf16 on your RTX 4070 Ti SUPER (16 GB) — no 4-bit required. Our canvases are only 256×256, so each image produces few vision tokens, keeping memory low. (Full fine-tuning is also feasible with care.)

---

1. The dataset

Location: data/rune_goblin_visual_dataset_5000/rune_goblin_visual_dataset/

• 5,000 samples, 256×256 RGB JPEG canvases of wobbly, hand-drawn ink runes on a parchment background (plus scattered dot-noise to mimic real canvas mess).
• Train/val already split 4,500 / 500. Assistant JSON is 100 % valid.
• Category mix: basic_clean 1000, messy_handdrawn 1200, combo_rules 1100, cursed_broken_mark 700, enemy_specific 700, invalid_ambiguous 300.

Task

canvas image  +  game-state text  →  { visual_reading, spell }  (JSON only)

Provided formats (pick one; image paths are relative)

File	Shape
*_full.jsonl	everything: id, category, image, runes_ground_truth, game_state, messages
train/validation_messages.jsonl	{image, messages} with <image> token in the user turn
train/validation_hf_vision_messages.jsonl	HF/TRL content-parts: {images:[...], messages:[{content:[{type:image},{type:text}]}]}
train/validation_llava_style.jsonl	{image, conversations:[{from,value}], system}

Output JSON the model must learn

{
  "visual_reading": {
    "detected_runes": ["spiral","eye","broken_mark"],
    "ambiguous_runes": [],
    "drawing_style": "wobbly ink marks",
    "layout": "left-to-right chain",
    "confidence": 0.91,
    "notes": ["broken_mark_adds_cursed_side_effect"]
  },
  "spell": {
    "spell_name": "Cursed Foresight Loop",
    "spell_type": "prophecy_loop_curse",
    "flavor": "...",
    "effect": "Deals 2 damage to Mirror Fungus; confuses for 1 turn.",
    "side_effect": "player loses a little courage.",
    "enemy_hp_delta": -2,    // range -4..0
    "player_hp_delta": -1,   // range -3..2
    "status_effects": ["enemy_confused","spell_cursed"],
    "chaos": 8               // range 1..10
  }
}

Game-design signal baked into the data: clearer drawings → higher confidence, lower chaos; messy/ambiguous drawings still cast but raise chaos and can turn cursed. Bad player art is part of the game loop.

Step 1 — convert to absolute paths + framework formats

Frameworks need absolute image paths. The Modal notebook includes a conversion cell that writes:

Output	Purpose
vision_swift_{train,val}.jsonl	ms-swift
vision_sharegpt_{train,val}.jsonl	LLaMA-Factory ShareGPT

If you run outside Modal, use the same mapping: resolve each record's image field under the dataset root, skip or repair any missing image, and emit the framework-specific JSONL with absolute image paths.

---

2. Option A — ms-swift (recommended, simplest)

MiniCPM-V 4.6 has native ms-swift support. One CLI, handles the vision plumbing and LoRA target modules for you.

uv pip install "ms-swift>=3.0" transformers accelerate peft   # into the project venv
# (or: pip install ms-swift in a dedicated env)

Train LoRA:

source .env
swift sft \
  --model models/MiniCPM-V-4.6 \
  --model_type minicpm-v-4_6 \
  --dataset data/vision_prepared/vision_swift_train.jsonl \
  --val_dataset data/vision_prepared/vision_swift_val.jsonl \
  --train_type lora \
  --lora_rank 16 --lora_alpha 32 \
  --torch_dtype bfloat16 \
  --num_train_epochs 3 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 8 \
  --learning_rate 1e-4 \
  --freeze_vit true \
  --max_length 2048 \
  --gradient_checkpointing true \
  --eval_steps 200 --save_steps 200 \
  --output_dir models/rune-goblin-vision-lora

Notes:

• --freeze_vit true trains only the LLM + resampler adapters (faster, lighter, and enough since the glyph vocabulary is small). Set false to also adapt the vision encoder if recognition of messy drawings underperforms.
• Effective batch = 2 × 8 = 16. Drop per_device_train_batch_size to 1 (and raise grad-accum) if you hit OOM.
• Inference after training:

  swift infer --adapters models/rune-goblin-vision-lora \
              --load_data_args true --val_dataset data/vision_prepared/vision_swift_val.jsonl
  

• Merge LoRA into the base:

  swift export --adapters models/rune-goblin-vision-lora --merge_lora true
  

---

2. Option B — LLaMA-Factory (ready-made config)

OpenBMB ships a MiniCPM-V 4.6 LoRA recipe for LLaMA-Factory.

git clone https://github.com/hiyouga/LLaMA-Factory && cd LLaMA-Factory
pip install -e ".[torch,metrics]"

1. Register the dataset — add to data/dataset_info.json:

   "rune_goblin_vision": {
     "file_name": "/home/ashu/github/goblin/data/vision_prepared/vision_sharegpt_train.jsonl",
     "formatting": "sharegpt",
     "columns": { "messages": "conversations", "images": "images", "system": "system" },
     "tags": { "role_tag": "from", "content_tag": "value",
               "user_tag": "human", "assistant_tag": "gpt" }
   }
   

2. Train (start from the official example and point dataset at rune_goblin_vision):

   llamafactory-cli train \
     --stage sft --do_train true \
     --model_name_or_path /home/ashu/github/goblin/models/MiniCPM-V-4.6 \
     --trust_remote_code true \
     --dataset rune_goblin_vision \
     --template minicpm_v \
     --finetuning_type lora --lora_rank 16 --lora_target all \
     --freeze_vision_tower true \
     --cutoff_len 2048 \
     --per_device_train_batch_size 2 --gradient_accumulation_steps 8 \
     --learning_rate 1e-4 --num_train_epochs 3 \
     --bf16 true --gradient_checkpointing true \
     --output_dir models/rune-goblin-vision-lora-lf
   

   Reference config: https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/finetune/llamafactory_minicpmv46.md

---

3. VRAM guidance (16 GB)

Setup	Fits 16 GB?	Notes
LoRA bf16, ViT frozen, bs=2, len=2048	✅ comfortable	recommended starting point
LoRA bf16, ViT trainable	✅ likely	a bit more memory; helps messy-drawing recall
QLoRA 4-bit	✅ easy	only needed if you also raise batch/seq a lot
Full fine-tune	⚠️ tight	possible for a ~2.6B model with bs=1 + grad-ckpt

If you OOM: per_device_train_batch_size=1, raise gradient_accumulation_steps, keep gradient_checkpointing true, keep freeze_vit true.

---

4. Evaluate like a game engine

Beyond loss, the metrics that matter (mirrors rune_goblin.evaluate):

• Valid JSON rate (must parse to the visual_reading+spell schema) — target > 95 %
• Rune-recognition accuracy — compare visual_reading.detected_runes to runes_ground_truth in *_full.jsonl
• Delta-range validity — enemy_hp_delta ∈ [-4,0], player_hp_delta ∈ [-3,2], chaos ∈ [1,10]
• Weakness usage — weakness-hitting runes should produce meaningful effects

Quick recognition check after training (sketch):

import json
from rune_goblin.schema import _extract_json   # reuse the JSON repair
# load *_full.jsonl for ground-truth runes, run model.infer(image, state),
# parse visual_reading.detected_runes, compare to runes_ground_truth.

---

5. Serve the fine-tune in the game

Two paths:

1. transformers (simplest): load models/MiniCPM-V-4.6 + the LoRA adapter (or the merged model) with trust_remote_code=True and call its chat method with the canvas image + state prompt. Wire this into a vision variant of rune_goblin.inference.
2. llama.cpp / Ollama (fast, matches your GGUF download): after swift export --merge_lora true, convert the merged HF model to GGUF with llama.cpp/convert_hf_to_gguf.py, regenerate the mmproj projector, then serve. This produces a fine-tuned analogue of the Q8_0 + mmproj pair you already have in models/MiniCPM-V-4.6-gguf/.

The game prompt must match training exactly (see the system + user strings in the dataset): system = "You are Rune Goblin, a tiny vision spell engine…", user = <image>\nSTATE: player_hp=… enemy=… …\nLook at the drawn RuneLang spell….

---

6. Pitfalls

• Don't point training at the GGUF — use openbmb/MiniCPM-V-4.6 safetensors.
• Relative image paths in the raw dataset will fail mid-training; always convert them to absolute paths first. The Modal notebook's preparation cell does this and skips records with missing images.
• Prompt drift: keep the exact system/user template from the dataset at game time or accuracy drops.
• The dataset is synthetic — after launch, log real Gradio-canvas drawings and append them as a second-stage dataset (the README suggests this).
• trust_remote_code=True is required (custom MiniCPMV4_6 architecture).