ukanwat/custral-14b-lora

custral — GUI Grounding LoRA (Ministral 3 14B)

LoRA adapters (r=32, a=64, LLM-only QLoRA on a 4-bit base) that fine-tune Ministral 3 14B Instruct for GUI grounding: screenshot + instruction -> click(x, y) with coordinates normalized to 0-1000. Trained entirely locally on Apple Silicon with mlx-vlm.

Full code, data pipeline, training logs and findings: https://github.com/ukanwat/custral

Files

File	What	ScreenSpot (n=150)
adapter_sft_cum4000.safetensors	mix-SFT (13k web+desktop+action examples, 4000 iters)	30.0% (peak 34.0% mid-run)
adapter_rft.safetensors	+ rejection-sampling fine-tune (best-of-4 in-box clicks, 1400 iters)	28.0% (null vs SFT)

Un-tuned base scores 22.0%. The recipe (frozen vision encoder, 13k examples, lr 2e-5) converges to low-30s ScreenSpot and plateaus — see the repo README for the full analysis.

Usage (mlx-vlm, quant-preserving load)

import mlx.core as mx
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.trainer.utils import get_peft_model, find_all_linear_names

model, processor = load("mlx-community/Ministral-3-14B-Instruct-2512-4bit")
model = get_peft_model(model, find_all_linear_names(model.language_model),
                       rank=32, alpha=64, dropout=0.0)
model.load_weights(list(mx.load("adapter_sft_cum4000.safetensors").items()), strict=False)

prompt = apply_chat_template(processor, model.config,
    "Click on: add to cart button\nRespond with one action only: click(x, y) "
    "where x and y are integers in 0-1000.", num_images=1)
print(generate(model, processor, prompt, image=["screenshot.png"], max_tokens=24))

Do not load via --adapter-path / apply_lora_layers on the 4-bit base — it dequantizes to ~28 GB. Use the injection pattern above (or eval/eval_grounding.py --lora-checkpoint in the repo).