| --- |
| license: apache-2.0 |
| base_model: openbmb/MiniCPM-V-4 |
| tags: |
| - lora |
| - peft |
| - vision-language |
| - floorplan |
| - object-detection |
| - minicpm-v |
| datasets: |
| - Voxel51/FloorPlanCAD |
| library_name: peft |
| --- |
| |
| # MiniCPM-V-4 Floor-Plan Element Detection (LoRA + vision tuning) |
|
|
| Fine-tune of **MiniCPM-V-4 (4B)** for structured extraction from CAD floor |
| plans: walls, doors, windows, stairs, fixtures, and furniture as JSON with |
| bounding boxes normalized to `[0, 1000]`. |
|
|
| ```json |
| {"elements": [{"type": "double_door", "bbox": [623, 730, 710, 763]}, ...]} |
| ``` |
|
|
| ## Results (held-out FloorPlanCAD, detection F1 @ IoU 0.5, greedy decoding) |
|
|
| | | JSON valid | Precision | Recall | F1 | |
| |---|---|---|---|---| |
| | MiniCPM-V-4 zero-shot | 32% | 0.027 | 0.010 | 0.015 | |
| | LoRA, LLM-only (300 steps, 800 samples) | 16% | 0.0 | 0.0 | 0.0 | |
| | **LoRA + vision tuning (800 steps, 3.3k samples)** | **40%** | **0.439** | 0.060 | **0.105** | |
|
|
| The decisive factor was **unfreezing the vision tower**: LLM-only LoRA |
| learned the output format but stayed image-blind (high-confidence repetition |
| of dataset priors). With vision tuning, precision rose 16x over zero-shot — |
| the model genuinely grounds boxes in the drawing. Recall remains the open |
| weakness (long element lists; outputs sometimes truncate before the JSON |
| closes). |
|
|
| ## Training |
|
|
| - Base: `openbmb/MiniCPM-V-4`, official MiniCPM-V finetune harness |
| (`llm_type` ChatML; note: the harness's qwen2 target-span detection needs a |
| patch for V-4's tokenizer — spans located by token-id lookup of |
| `'<|im_start|>'`/`'assistant'` only exist in Qwen2's vocab) |
| - LoRA on LLM attention projections (q/k/v/o) + full vision-tower tuning |
| - 3,281 train / held-out eval from |
| [FloorPlanCAD](https://huggingface.co/datasets/Voxel51/FloorPlanCAD) |
| (FiftyOne detections converted to conversation JSON) |
| - 800 steps, effective batch 8, lr 1e-5 cosine, bf16, single NVIDIA L4 |
| (Modal), ~3.5 h |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoTokenizer |
| from peft import PeftModel |
| |
| base = "openbmb/MiniCPM-V-4" |
| tokenizer = AutoTokenizer.from_pretrained(base, trust_remote_code=True) |
| model = AutoModel.from_pretrained(base, trust_remote_code=True, |
| torch_dtype=torch.bfloat16, device_map="cuda") |
| model = PeftModel.from_pretrained(model, "Barath/minicpmv4-floorplan-lora") |
| model = model.merge_and_unload().eval() |
| |
| from PIL import Image |
| img = Image.open("floorplan.png").convert("RGB") |
| prompt = ('Detect the architectural elements in this floor plan (walls, doors, ' |
| 'windows, stairs, fixtures, furniture). Return only JSON: ' |
| '{"elements": [{"type": str, "bbox": [x1, y1, x2, y2]}]} with integer ' |
| 'coordinates normalized to [0, 1000].') |
| out = model.chat(msgs=[{"role": "user", "content": [img, prompt]}], |
| tokenizer=tokenizer, sampling=False, |
| max_new_tokens=1500, repetition_penalty=1.1) |
| print(out) |
| ``` |
|
|
| ## Limitations |
|
|
| - Recall is low (0.06): dense plans with dozens of elements are only |
| partially enumerated, and long outputs may truncate mid-JSON. Use a |
| truncation-tolerant parser in production. |
| - Trained on CAD-style monochrome drawings (FloorPlanCAD); photographed or |
| hand-drawn plans are out of distribution. |
| - Coordinates are model estimates, not measurements. |
|
|
| Trained for the Hugging Face **Build Small Hackathon 2026**. |
|
|
