UI-Venus-1.5-2B-R5A (unofficial)

Unofficial. Not affiliated with, reviewed by, or endorsed by inclusionAI or the UI-Venus authors. The UI-Venus name is used only to say where the weights came from, as Apache-2.0 §6 allows.

This is a small LoRA fine-tune of inclusionAI/UI-Venus-1.5-2B, merged back into full bf16 weights. It looks at a screenshot and a short description of a control ("the Settings button") and answers with either a point to click or "not on screen":

{"x": 515, "y": 93}      # 0-1000 normalized to the image width / height
{"absent": true}         # the target is not visible in this screenshot

We trained it for one job: suggest where a control is, and say so when it is not visible, so that a person (or a tightly fenced script) can decide what to do. It is a hobby project run on home hardware, with small test sets. Read the weaknesses section before you use it for anything.

What is in this repo

path what
model.safetensors, config.json, tokenizer and processor files, chat_template.jinja Merged bf16 weights (Transformers layout, qwen3_vl).
gguf/UI-Venus-1.5-2B-r5.Q8_0.gguf Q8_0 GGUF of the same model. Every number on this card was measured with this file.
gguf/UI-Venus-1.5-2B-r5.mmproj-f16.gguf F16 vision projector for llama.cpp. Use this one, not the base model's: the vision-language aligner was trained too.
harness/ The suggest-only shim, our serving scripts (cleaned of our own hosts and paths), example systemd units, the click grammar, and offline tests. See harness/RUN-GUIDE.md.
LICENSE, LICENSE-CHECK.md Apache-2.0 text and our notes on the upstream licenses.
WEIGHTS-INVENTORY.md Sizes and sha256 of the weight files.
SHA256SUMS-UPLOAD sha256 of every file in this repo. Check with sha256sum -c SHA256SUMS-UPLOAD.

We only tested the GGUF pair with llama.cpp. We did not test or benchmark the safetensors tree in Transformers or vLLM.

Intended use

  • Suggest-only UI element locate. Given a screenshot and a target description, return a click point or abstain. The shim in harness/ runs a deny-list first (login, password, checkout, payment, delete, post and similar words are blocked before the model is called), then one model call with a grammar that only allows the two answer shapes above. If the model's log-odds for "absent" over "click" are above 3.5, or it answers absent, the shim abstains. It never clicks.
  • Optional bounded click mode (harness/venus_click.py). This is for experiments on frozen local copies of pages, not live sites. Its guardrails:
    • off unless a CLICK_MODE file says 1; VENUS_CLICK_MODE=0 or a missing file means suggest-only (kill switch);
    • loopback only: pages are served GET-only on 127.0.0.1, and every request to any other origin, and every non-GET/HEAD request, is failed in the browser;
    • clicks only, never types (so it cannot enter a PIN, password or card number);
    • denies and stops on login/password/PIN/OTP/payment fields, form submits, file inputs, downloads, links off the page origin, and buttons whose text is an action verb (send, buy, pay, delete, post, submit, sign in...);
    • 15 steps, 120 s, one task at a time; fresh browser profile per task; logs keep screenshot hashes only;
    • the HTTP wrapper only accepts callers listed in ALLOWED_CLIENTS.

Not for: unattended clicking, filling forms, logging in, paying, sending messages, or any action you cannot undo. Do not point it at real accounts.

Fine-tuning we did

Base: inclusionAI/UI-Venus-1.5-2B (built on Qwen3-VL-2B-Instruct). Trained with ms-swift 4.6.0, model_type qwen3_vl.

Recipe (from the training run's args.json):

setting value
method LoRA, rank 32, alpha 64, dropout 0.05, target all-linear
frozen vision tower (ViT) frozen; aligner trained at lr 1e-5; language model LoRA at lr 1e-4
schedule cosine, warmup 0.05, weight decay 0.1, max grad norm 1.0, 2 epochs
batch 1 per device × 8 gradient accumulation, max length 4096, bf16, gradient checkpointing, seed 42
steps 2,518 (2 epochs)
checkpoint the one with the lowest validation loss: step 2518, eval loss 0.17607 (eval loss went 0.3025 at step 100 → 0.2251 at 1000 → 0.1837 at 2000 → 0.1761 at 2518)
output LoRA merged into bf16 safetensors, then converted to GGUF Q8_0 plus an F16 mmproj

Data recipe. All training screenshots are of synthetic web pages that we generated: made-up or generic labels on many page templates (lists, menus, cards, forms, panels, tabs, drawers), rendered locally in headless Chrome at 1280×800 from file:// with the network blocked. In R3–R5 each example is used in three views: the full page, a 1024-wide resize, and a 256-px crop. The answer is either the target's center (0–1000) or exactly {"absent":true}. Hard cases we built on purpose:

  • look-alike decoys (near-identical labels, case, homoglyphs, extra letters) next to the real target;
  • target only below the fold → the correct answer is absent (it is not in the screenshot);
  • target hidden in a closed drawer, collapsed accordion, inactive tab or display:none while a look-alike stays visible → absent;
  • "twin-below" sets of four matched pages that differ only in whether the target is visible.

Rounds. This is round 5 (R5) of our own data rounds (R1, R2a, R2b, R3, R4, R5). R5 was a fresh LoRA from the base weights, not a continuation of an older adapter. It replays the full R4 training set verbatim (7,683 rows: 5,466 replayed from R3 plus 2,217 from R4) and adds new R5 rows on new template families. The final build was 10,071 training rows and 549 validation rows; the share of "absent" rows went from 14.45% to 15.25%. "R5A" is our name for this round-5 checkpoint as served.

Kept out of training. Every evaluation set below and all earlier held-out sets (by seed, image hash, perceptual hash and label words); three page families were held out entirely (weather, events, podcast); a quarantined "do not train" folder; any page from our sandbox tests and their misses; and a sealed gate set that was never opened. A leakage audit before training reported 0 leaks. No live websites, no real account screens and no training data are published here.

Training hardware: one NVIDIA RTX 5090 (32 GB), peak about 7 GB, about 2.1–2.5 s per step. The first run stopped at step 2,375 because of a GPU fault (not a training error); it was resumed from the step-2,300 checkpoint with the same data and settings and finished cleanly.

Sources: r5-args.json (sha256 8752e159…), PLAN-VENUS2B-R5.md (a4acee34…), RESULT-VENUS2B-R5.md (9b9a39c8…), RESULT-VENUS2B-R5-RESUME2.md (e7e89607…), PLAN-VENUS2B-R3.md (20765e69…).

Evaluation: base vs R5A

Every number below is copied from one of our result files (file name and sha256 prefix in the last column). "not run" means we never ran that model on that set; we do not estimate it. All runs used the Q8_0 GGUF with llama.cpp b11669 on one AMD BC-250 board, one seed, temperature 0. The base model ran as a Q8_0 GGUF (320a1d2f…) with its own F16 mmproj (df84ad76…), so each column is model plus its own projector.

Sets (all small and local; pages are frozen copies, not live sites):

  • L0b: 42 tasks on 21 pages. 30 tasks where the target is visible (5 of them with a look-alike decoy), 7 where it is only below the fold (correct answer: abstain), 5 deny-list tasks (blocked before the model).
  • L4: 40 of the L0b tasks through the final shim (30 visible, 7 below-fold, 3 deny).
  • SIM v2: 24 tasks: 16 visible (4 with look-alikes), 4 below-fold, 4 deny.
  • R5-HO: our synthetic held-out set from round 5, 345 rows: 210 target visible, 135 target absent. Never trained on.
set metric base UI-Venus-1.5-2B R5A source
L0b visible target correct 29/30 29/30 RESULT-BASE-L0B.md 3d0535d5…; RESULT-L0B.md a3edc6b1…; l0b.json d6e786cb…
L0b wrong click on a visible target 1 0 same
L0b look-alike wrong click 0/5 0/5 same
L0b below-fold: correct abstain / wrong click 1/7 / 6/7 5/7 / 2/7 same
L0b deny-list escapes 0/5 0/5 same
L0b abstains (of 37 model calls) 1/37 6/37 same
L0b latency p50 / p95 (ms), 37 calls 3934.0 / 4159.2 3759.0 / 3940.2 same
L0b timeouts > 15 s 0/37 0/37 same
L4 visible target correct not run 29/30 (0 wrong clicks) RESULT-L4.md a0bfb409…
L4 look-alike wrong click not run 0/5 same
L4 below-fold: correct abstain / wrong click not run 5/7 / 2/7 same
L4 deny-list escapes not run 0/3 same
L4 abstains (of 37 model calls) not run 6/37 same
L4 latency p50 / p95 (ms), 37 calls through the shim not run 2396.0 / 2490.6 same
SIM v2 visible target correct (forced-click rule, τ 3.5) not run 15/16 RESULT-SIM.md d6f89361…
SIM v2 look-alike wrong click not run 0/4 same
SIM v2 below-fold correct abstain not run 4/4 same
SIM v2 deny-list escapes not run 0/4 same
SIM v2 latency p50 / p95 (ms), 20 calls not run 4465.5 / 12874.0 same
R5-HO visible target hit, greedy (τ 0) not run 198/210 RESULT-PROBE.md ec754783…; RESULT-VERIFIER.md d0bc618f…
R5-HO false "absent" on a visible target, greedy not run 1/210 same
R5-HO correct abstain when the target is absent, greedy not run 110/135 same
R5-HO visible target hit, τ 3.5 with forced click not run 199/210 same
R5-HO false "absent" on a visible target, τ 3.5 with forced click not run 0/210 same
R5-HO correct abstain when the target is absent, τ 3.5 with forced click not run 91/135 same

How to read this:

  • On the only like-for-like set (L0b) both models found 29 of 30 visible targets. R5A is not more accurate on visible targets there. The measured differences are 0 vs 1 wrong click on visible targets and 5/7 vs 1/7 correct abstains when the target is below the fold. The two misses differ: the base clicked the wrong control on S05; R5A abstained on a look-alike task (L01) where the base clicked correctly.
  • L4 and SIM v2 are R5A-only checks of the serving path. SIM v2 used an older forced-click rule (a second pass forces a click when the absent margin is small), which is why its p95 latency is higher.
  • R5-HO is synthetic and from the same generator family as the training data (with held-out templates), so it flatters the model. We did not score the shipped shim rule (abstain on any "absent", no forced click) on R5-HO; only the two rules shown were scored there.
  • One more base number: on an early 16-task desk test the base scored 11/16 (full frame plus crop) and 9/16 (single 1024-wide pass) (RESULT-VENUS2B.md 3a177d55…). R5A was not run on that set.
  • Verifier-v2 (fresh draw). On a new 345-row draw (210 visible, 135 absent), with the official bars locked before inference, the official arm (τ 3.5 G0 + verifier v2) scored 0/210 false "absent" on visible targets (bar 0), 131/135 correct abstains when the target is absent (bar ≥124), and 208/210 visible-target hits (bar ≥196). The verifier caused 0 false actions and 0/345 parser fails (RESULT-V2.md 6cd6c48b…). Comparison arms (false "absent" / kept / hit): G0 alone 0 / 103 / 201; greedy 1 / 122 / 200; greedy + v2 1 / 135 / 207. The one greedy false "absent" is a visible target the model called absent; v2 cannot turn an already-absent answer into a click. This ran on a llama.cpp CPU build (b11454), not the served build (b11669). A small check found one decision that differed between the two builds on a non-official row, so these are not served or production numbers.

Weaknesses (please read)

  • Below-fold false clicks. When the target is only below the fold, R5A still clicked something on screen in 2 of 7 L0b/L4 tasks (B06, B07) instead of abstaining.
  • Look-alike twins and false "absent". On our internal round-5 gate (500 visible-target rows with twins and below-fold decoys), R5A said "absent" on 7/500 visible targets; our own limit was 3/500, so R5 failed that gate (the previous round, R4, had 23/500) (RESULT-VENUS2B-R5-RESUME2.md e7e89607…). It also abstained on a visible look-alike target in L0b/L4 (L01).
  • Text-identical decoys: the open gap. On R5-HO, greedy R5A clicked a decoy on 25 of 135 target-absent screens. 20 of those 25 decoys had the same text as the target. We tried two fixes and both stopped:
    • a linear probe on the model's decision-token states (Path A) kept 0/25 of those hard cases at a zero false-absent threshold; it matched the τ 3.5 gate exactly (0 / 91 / 199) and added nothing (RESULT-PROBE.md ec754783…);
    • a rule-based region/role verifier (verifier-v1) caught all 25 but wrongly vetoed 13 rows, 4 of them correct clicks, so it was stopped (RESULT-VERIFIER.md d0bc618f…). In short, R5A often cannot tell "the right button" from "a button with the same text in the wrong place".
  • Abstain rate. With the shipped rule R5A abstained on 6 of 37 model calls on L0b and L4 (base: 1 of 37). Most are correct (below-fold), but one was a visible target. On R5-HO, the rules we scored kept only 91–110 of 135 absent screens as abstains.
  • Latency on a BC-250-class AMD APU board (Vulkan, Q8_0, one request at a time): p50 about 2.4 s and p95 about 2.5 s per suggestion through the shim (L4); 3.8 s / 3.9 s on the L0b client path; up to 12.9 s p95 with the old forced-click second pass (SIM v2). The click lane adds browser time per step and caps a task at 120 s.
  • Small eval sets. 16 to 42 tasks per sandbox set, on a handful of local pages, one seed, one board. These are not ScreenSpot or any public benchmark, and confidence intervals on sets this small are wide.
  • Synthetic training pages. Real sites have more varied layouts, icons, languages and themes than our generator.
  • Not production-safe for unattended clicking. Use it to suggest, and have a person confirm. The click lane exists for supervised experiments on frozen local pages only.

Other notes

llama.cpp quick start (file names as in this repo):

llama-server \
  -m gguf/UI-Venus-1.5-2B-r5.Q8_0.gguf \
  --mmproj gguf/UI-Venus-1.5-2B-r5.mmproj-f16.gguf \
  --host 127.0.0.1 --port 18480 \
  -c 8192 -ngl 99 --parallel 1 --jinja --cache-ram 0 \
  --grammar-file harness/click.gbnf

Then send an OpenAI-style chat request with the screenshot as an image_url (data URL) and this prompt:

Return JSON {"x": int, "y": int} (0-1000 normalized of image width/height) of the center of: <TARGET>, inside the web page content. If it is not visible, return {"absent":true}.

Use temperature: 0. Add "logprobs": true, "top_logprobs": 20 if you want the shim's absent-vs-click log-odds. Or run harness/venus_suggest.py, which does all of this (see harness/RUN-GUIDE.md).

Hardware we tested on: training on one NVIDIA RTX 5090; serving and every evaluation on one AMD BC-250 board (an AMD APU board) with llama.cpp b11669 and the Vulkan backend; the Path A probe ran on a desktop CPU in fp32.

Unofficial. This is a personal fine-tune. It is not an official UI-Venus or inclusionAI release, and the UI-Venus authors have not reviewed it.

License and attribution. Apache-2.0, the same license as the base model (inclusionAI/UI-Venus-1.5-2B) and its upstream Qwen3-VL-2B-Instruct. The weights are modified: LoRA fine-tune, merged, and converted to GGUF. See LICENSE and LICENSE-CHECK.md. If you use the base work, please cite:

  • UI-Venus-1.5 Technical Report, arXiv:2602.09082
  • UI-Venus Technical Report, arXiv:2508.10833
Downloads last month
112
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jlay843/UI-Venus-1.5-2B-R5A-unofficial

Finetuned
(2)
this model
Adapters
1 model

Papers for jlay843/UI-Venus-1.5-2B-R5A-unofficial