README.md

metadata

base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
license: apache-2.0
datasets:
  - ruotian/ContrastGround
tags:
  - gui-grounding
  - computer-use
  - qwen3-vl
  - lora
  - selectground

SelectGround-8B

SelectGround-8B maps a screenshot and a natural-language instruction to one click point. This release replaces the earlier ClickContrast-trained checkpoint with a checkpoint trained from the pinned plain Qwen3-VL-8B-Instruct base on the selectground-8b configuration of ruotian/ContrastGround. It is a single directly trained LoRA checkpoint, not a model soup or weight aggregate.

Direct grounding results

Benchmark	Accuracy	Semantic error
ScreenSpot-Pro	65.09	29.35
UI-Vision	37.12	49.10
OSWorld-G	69.41	20.00

UI-Vision is the equal-weight macro over its basic, functional, and spatial element-grounding subsets. OSWorld-G uses its 510 target-bearing examples; refusal-only rows are excluded. These public benchmarks were used during model selection, so results are test-tuned rather than held-out validation estimates.

Self-Contrastive Grounding

The release also includes Self-Contrastive Grounding, a training-free extension of the paper's contrast-mining principle. Training mines observed hard distractors from disagreement between models. At inference, Self-Contrast mines latent distractors from disagreement between deterministic views of the same model, then asks every other view to verify each visible coordinate. A proposal is never scored by the view that generated it, which prevents self-confirmation.

Inference	ScreenSpot-Pro	UI-Vision	OSWorld-G
Direct	65.09	37.12	69.41
Self-Contrast	71.16	44.09	72.75

The method uses one full-screen view, one 40% incumbent-centered revisit, and four fixed overlapping 60% views. All crops are enlarged by 2x. Within each view, coordinate-string mean token log-likelihoods are standardized; evidence is averaged across non-source views and combined at equal weight with proximity to the incumbent revisit. This one configuration is shared by all three benchmarks: there is no benchmark-specific gate, router, prompt, or threshold.

The implementation retains each view's visual prefix after greedy candidate generation and reuses its KV cache for batched coordinate scoring. It therefore uses six visual prefills, rather than the twelve prefills of a naive generate-then-rescore implementation, and requires no weight update.

python evaluate.py \
  --model ruotian/SelectGround-8B \
  --benchmark screenspot_pro \
  --data data/screenspot-pro \
  --self-contrast \
  --output outputs/screenspot-pro-self-contrast.jsonl

Full 8B ablations, using the same benchmark protocols, are:

Variant	ScreenSpot-Pro	UI-Vision	OSWorld-G
Full	71.16	44.09	72.75
no latent distractors	71.22	43.44	69.61
one latent distractor	70.97	43.44	70.59
no recurrent anchor	68.82	43.23	72.75
no cross-view evidence	71.16	43.46	69.41
no anchor proximity	70.15	44.10	72.94

self_contrast_manifest.json records the exact protocol, full counts, split metrics, artifact checksums, and cache-reuse equivalence test.

Direct inference

The repository includes the exact loader and evaluator. visual_merger.pt must be loaded in addition to the PEFT adapter; selectground.py does this.

python evaluate.py \
  --model ruotian/SelectGround-8B \
  --benchmark screenspot_pro \
  --data data/screenspot-pro \
  --output outputs/screenspot-pro.jsonl

Inference uses the full native screenshot, the prompt in selectground.py, Qwen smart resize with min_pixels=3136 and max_pixels=8847360, greedy decoding for at most 32 tokens, and normalized 0–1000 point coordinates.

Reproduce training from the plain base

hf download ruotian/ContrastGround --repo-type dataset \
  --local-dir data/ContrastGround

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
accelerate launch --mixed_precision bf16 --num_processes 2 train.py \
  --model 8b \
  --data data/ContrastGround \
  --pairs-file data/ContrastGround/data/selectground-8b/train_pairs.jsonl \
  --replay-file data/ContrastGround/data/selectground-8b/train_replays.jsonl \
  --output outputs/SelectGround-8B \
  --steps 240 --gpus 2 --accumulation 64 \
  --learning-rate 3e-5 --selector-learning-rate 1e-4 \
  --aux-weight 0.1 --ground-coordinate-weight 1.0 \
  --margin 0.3 --pair-weight 0.5 \
  --warmup-steps 10 --scheduler-steps 384 \
  --holdout-fraction 0.02 --seed 20260819

This is SFT coordinate cross-entropy on pair and replay rows plus the paper's auxiliary competitor-selection loss on pair rows. Pair and replay microbatches alternate. LoRA uses rank 64, alpha 128, dropout 0.05 on q/k/v/o/gate/up/down projections. The selector reads semantic attention from layers 18–23. See training_manifest.json for the complete recipe and artifact SHA-256 checksums.

The reference environment used PyTorch 2.11.0+cu128, Transformers 4.57.1, PEFT 0.19.1, Accelerate 1.13.0, and qwen-vl-utils 0.0.14. CUDA kernels are not bitwise deterministic; clean runs should be expected to be close rather than byte-identical.

License and data

The adapter follows the Apache-2.0 license of the base model. Dataset assets retain their upstream terms; consult the ContrastGround data card and its row-level provenance.