| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-E2B |
| library_name: peft |
| language: |
| - en |
| tags: |
| - web-agents |
| - gui-agents |
| - mind2web |
| - lora |
| - pointer-network |
| - selective-prediction |
| - calibration |
| - decision-making |
| - fast-decision-model |
| - system-1 |
| - model-routing |
| - gui-grounding |
| datasets: |
| - osunlp/Multimodal-Mind2Web |
| --- |
| |
| # Gemma 4 E2B + LoRA pointer head for web element selection (Multimodal-Mind2Web) |
|
|
| **A small, fast "which element do I click next?" model with a confidence score. Research release, not a ready-to-use web agent.** |
|
|
| Given a task, the previous actions, a full-page screenshot and a list of candidate DOM elements, the model points at one candidate (or at a learned "none of these" slot) and predicts the operation type (click / type / select). It does **not** generate the text to type. |
|
|
| - Backbone: `google/gemma-4-E2B` (frozen) + LoRA (rank 32, attention + MLP, layers 0-29, 180 modules) + a small pointer head trained from scratch. |
| - Trained on Multimodal-Mind2Web for about 8 hours on one A100-40GB. |
| - Speed: in the live form test below (short pages, one screenshot strip, about 33 candidates) a decision took about 0.2 s for the model plus 0.1 s for candidate extraction and screenshot on an A100. Inputs of Mind2Web size (several strips, up to 4600 tokens) are slower; we did not benchmark that separately. |
| - Honest summary of what it is good and bad at is in [Summary of results](#summary-of-results) and [Limitations](#limitations-and-what-this-model-is-not). |
|
|
| **Role: a fast decision module ("System 1").** The model is meant as the quick, cheap first decision in a larger agent: it picks the next element or abstains with a confidence score, and low-confidence steps are escalated to a stronger model ("System 2", for example a large VLM) or a human. It is a decision/routing component, not a standalone agent. |
|
|
| **Approach: a calibrated decision head, "Kev-like".** Instead of generating the next action as text, the model makes a *decision over a closed set*: a query vector (from the task, history and screenshot) is scored against one key per candidate element plus a learned "none of these" key, and a softmax gives a probability for every option. This pointer-style, "Kev-like" decision head has three consequences we care about: (1) one forward pass picks the element, no token-by-token decoding; (2) the top-1 probability is a ready-made confidence score, and "none" is an explicit option, so the model can abstain; (3) because decisions and confidences are explicit numbers, they can be evaluated and calibrated with standard tools (ECE, Brier score, risk-coverage curves, temperature scaling) and used to route uncertain steps to a stronger model. In this release the decision quality and the confidence ranking are measured (see results); the confidences themselves are still raw and need temperature scaling on held-out data before being read as probabilities. The "Kev-like" design follows [Kev](https://github.com/jaredpalmer/kev) by Jared Palmer (open-source family of "System One" decision models: a small LoRA adapter plus a pointer head on a Qwen base, [weights on Hugging Face](https://huggingface.co/collections/jaredpalmer/kev), Apache-2.0), which in turn is described as Jev-like; see [Related work](#related-work-and-credit). |
|
|
| > **Status:** single training run, single seed. Confidence scores are **raw (not temperature-scaled)** for this checkpoint. Numbers are not directly comparable with published Mind2Web numbers (see [Evaluation protocol](#evaluation-protocol)). |
|
|
| ## Summary of results |
|
|
| Element accuracy on Multimodal-Mind2Web test splits (`n` = steps; 95% bootstrap CI). A step counts as correct if the top-1 candidate is a ground-truth element. A "none" prediction counts as an error. |
|
|
| | Split | n | Accuracy (CI95) | Accuracy if the answer is in the list | Filter ceiling | ECE (raw) | AUROC of confidence | Acc. on most confident 20% | |
| |---|---|---|---|---|---|---|---| |
| | test_task | 1339 | **0.465** [0.439, 0.493] | 0.556 | 0.837 | 0.189 | 0.710 | 0.761 | |
| | test_website | 1019 | **0.384** [0.355, 0.413] | 0.483 | 0.795 | 0.189 | 0.698 | 0.662 | |
| | test_domain | 4060 | **0.385** [0.370, 0.400] | 0.456 | 0.845 | 0.199 | 0.737 | 0.703 | |
| |
| "Filter ceiling" is the fraction of steps in which our rule-based candidate filter keeps a correct element at all (see below). Nobody can score above it with this candidate list. Validation (579 steps carved from the training websites): accuracy 0.368 [0.326, 0.408]. |
| |
| Progression across our own runs (same candidate filter, same splits): |
| |
| | Run | test_task | test_website | test_domain | |
| |---|---|---|---| |
| | First text-only run, LoRA r16 attention | 0.375 | 0.275 | 0.308 | |
| | + full-page screenshot, LoRA r16 attention, 24-token candidate cap | 0.416 | 0.317 | 0.351 | |
| | **This model**: screenshot, LoRA r32 attention+MLP, tree-format candidates, gold candidate always present in training | **0.465** | **0.384** | **0.385** | |
|
|
| Risk-coverage (accuracy if the model only answers its most confident x% of steps; raw confidences): |
|
|
| | Split | 100% | 80% | 60% | 40% | 20% | |
| |---|---|---|---|---|---| |
| | test_task | 0.465 | 0.515 | 0.583 | 0.660 | 0.761 | |
| | test_website | 0.384 | 0.426 | 0.471 | 0.574 | 0.662 | |
| | test_domain | 0.385 | 0.445 | 0.513 | 0.591 | 0.703 | |
| |
| Other metrics (test_task / test_website / test_domain): operation-type accuracy 0.881 / 0.866 / 0.864; Brier 0.256 / 0.249 / 0.240; fraction of steps where the model answers "none" 0.120 / 0.119 / 0.092; "none" predicted when no correct element is in the list (abstention success) 0.367 / 0.373 / 0.237. |
|
|
| **Take-aways** |
|
|
| 1. The screenshot helps, and the gain is driven by the pixels. In an ablation on the earlier vision run (not repeated for this checkpoint), replacing the screenshot with a gray image or another page removed the gain (paired CI of the gain over the text-only model: +3.2 to +5.4 points). |
| 2. The confidence score carries information: accuracy rises from 0.38-0.47 on all steps to 0.66-0.76 on the most confident 20%. This is the main reason to use this model (selective prediction, routing hard cases to a stronger model), not its raw accuracy. |
| 3. Raw confidences are over-confident (ECE about 0.19-0.20). On the earlier vision model temperature scaling fitted on validation reduced ECE to roughly 0.05-0.09; **we have not measured this for this checkpoint**. Fit a temperature on your own held-out data before using the numbers as probabilities. |
| 4. Absolute accuracy is modest. About 15-20% of steps lose the correct element in our candidate filter, and even when the correct element is in the list the model finds it only 46-56% of the time. |
|
|
| ## Intended use |
|
|
| - Research on small, fast GUI/web element grounding with calibrated abstention. |
| - A cheap first-pass element selector whose low-confidence steps are escalated to a larger model or a human (cascade/routing experiments). |
| - A baseline and a set of documented negative results for pointer-style heads on top of a small VLM. |
|
|
| ## Model details |
|
|
| **Architecture.** The full Gemma 4 E2B model (35 layers, hidden size 1536) runs on the whole sequence; we read the hidden states of **layer 30** and attach: |
|
|
| - a **query** from the last token (LayerNorm -> linear, d_k = 256), |
| - one **key per candidate** from the mean of the candidate's token hidden states (LayerNorm -> linear), |
| - a learned **"none" key**, so the softmax is over `n_candidates + 1` slots, |
| - an **operation head** (linear, 3 classes: CLICK / TYPE / SELECT) on the query. |
|
|
| LoRA is applied to layers 0-29 (q/k/v/o and gate/up/down projections where they exist; the last 20 layers of Gemma 4 E2B share KV and have no k/v projections), rank 32, alpha 64, dropout 0.05. The vision tower and the language model weights are frozen. Hidden states of layers 30-34 are not used by the head (they are still computed in the forward pass; truncating the model at layer 30 would be an easy speed-up we did not implement). |
|
|
| **Input.** `[BOS] [screenshot strips] Task: ... Previous actions: ... Candidates: - <candidate 1> - <candidate 2> ... Task: ... Previous actions: ... Next element to interact with:` |
|
|
| - *Screenshot*: the full-page screenshot is cut top-to-bottom into 1280x1280 strips (a trailing piece under 200 px is merged), at most 5 strips (6400 px), 280 vision tokens per strip. About 19% of Mind2Web pages are taller than that and are truncated at the bottom. No target-dependent cropping is used (it would leak the answer). |
| - *Candidates*: text only, in "tree" format: `role "accessible-ish name" [state]`, for example `textbox "Where from?"`, `link "Flights"`, `button "Search"`. At most 64 tokens per candidate and 4600 tokens in total per step (candidate lines are shortened uniformly when a step is too long). **The model is not given element positions**, so it must match candidate text to what it sees in the screenshot. |
| - *Candidate list*: Multimodal-Mind2Web provides 331-743 candidates per step. We order them by TF-IDF similarity to the task and recent actions, then apply a rule filter (visible, interactive tags first, then text-bearing common tags) and keep at most K = 160. |
|
|
| **Loss.** Multi-positive softmax cross-entropy over candidates plus "none" (target is "none" if no correct candidate is in the list), plus 0.3 x cross-entropy for the operation type. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Data | Multimodal-Mind2Web `train` (7775 steps): 7196 for training, 579 for validation (validation = websites whose `crc32(website) % 8 == 0`) | |
| | Candidate handling in training | rule filter K = 160, then the dropped correct candidate is **added back** to the list (`--always_pos`). Validation and test lists are never modified | |
| | Epochs | 4 run, **checkpoint of epoch index 2 (third epoch) kept** (best validation accuracy: 0.249 / 0.363 / **0.368** / 0.358; ECE rose 0.119 -> 0.164 -> 0.260 -> 0.362, i.e. over-fitting in the last epoch) | |
| | Optimiser | AdamW, LoRA lr 2e-4, head lr 1e-3, cosine schedule, 3% warm-up, gradient clipping 1.0, micro-batch 1 with accumulation 8, bf16 | |
| | Candidate order | random permutation in training; original TF-IDF order at test time | |
| | Hardware / time | 1x A100-SXM4-40GB, 29,604 s for 4 epochs (about 1.03 s per example), peak memory 38.6 GB | |
| | Skipped | 4 training steps skipped for out-of-memory (inputs of about 13.8k tokens); 11 steps without a screenshot | |
| | Seeds | one (seed 0) | |
| | Base model revision | `google/gemma-4-E2B` @ `d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f` | |
|
|
| Before this run we swept LoRA capacity, learning rate, candidate text format, and the query layer on a 1500-example, 2-epoch text-only setup (val 579 steps, single seed): rank-32 attention+MLP was the only clear winner (about +6.6 points on validation over rank-16 attention-only); rank 64, lr 1e-4 and 4e-4, and query layer 35 were not better; the tree format was as accurate as the long attribute format and shorter; a 24-token candidate cap did not measurably hurt in the short regime. These differences are from one seed and a small validation set, treat them as hints. |
|
|
| ## Evaluation protocol |
|
|
| - Splits: the official Multimodal-Mind2Web `test_task`, `test_website`, `test_domain`. |
| - Metric: top-1 element accuracy over **our** candidate list (K = 160, rule filter). This is **not** the official candidate list (the Mind2Web paper uses a DeBERTa ranker's top-50), and steps with no correct element in the list count as errors. Our numbers must not be compared with published element accuracy without this caveat; we make no claim of parity or superiority with any published system. |
| - "Accuracy if the answer is in the list" is conditional on the correct element being among the candidates and is **higher by construction**; it should not be compared with unconditional numbers. |
| - ECE uses 15 equal-width bins on the top-1 probability. AUROC measures how well the top-1 probability separates correct from incorrect steps. |
| - Bootstrap confidence intervals are over steps. Several steps come from the same task or website, so the true uncertainty is somewhat larger than the CIs suggest. |
|
|
| ## A stress test outside the training distribution: live form filling (negative result) |
|
|
| We ran the model in a real browser (Playwright) on 12 synthetic forms and, when reachable, 5 public practice pages (no submit buttons were pressed). One step per field; the field label and value come from the task sentence; the model only picks the element. We compare two candidate styles: **as in the dataset** (raw attributes, no `<label>` binding) and **with labels** (label text of the associated `<label>` added as the accessible name), and a trivial no-model baseline (TF-IDF match of the field label against candidate text; this baseline is told the target label directly). |
|
|
| | Setting (86 steps, 17 forms) | Step accuracy (history = ground truth) | Fields correctly filled when the model drives (free run) | Forms fully correct | |
| |---|---|---|---| |
| | Gemma pointer, dataset-style candidates | 0.36 | 0.29 | 2/17 | |
| | Gemma pointer, label-bound candidates | 0.55 | 0.49 | 3/17 | |
| | Lexical-match baseline (knows the label) | 0.78 | 0.70 | 6/17 | |
|
|
| So: **this model is not a usable form filler.** What we found: |
|
|
| - Candidate lines like `textbox "fld3"` (opaque id/name) cannot be matched with the visible label, because the model never sees element positions. Adding the label text gave +19 points. Placeholder-based forms (text is in the candidate) scored 0.77, forms whose label is an unbound `<div>` 0.21. |
| - By type: text 0.50-0.75, select 0.36-0.57, checkbox/radio close to 0. |
| - The model often picks `button "Submit"` for fields that are not done yet, with 96-99% confidence (a learned "what comes next" prior). Confidence ranking is still informative overall (label-bound free run: AUROC 0.88, 0.94 accuracy on the most confident 20%), but confidently wrong steps exist. |
| - The operation head dropped from 0.88 on Mind2Web to 0.56-0.63 here: the task phrasing (many fields in one sentence) and the pages differ from the training distribution. |
| - 86 steps is a small sample: differences of about 10 points are within noise. |
|
|
| ## Limitations and what this model is not |
|
|
| - **Not an agent.** It selects an element and an operation type. It does not produce typed text, does not plan, and does not recover from errors. |
| - **Accuracy is modest** (0.38-0.47 end-to-end on our lists) and below what larger systems report; the candidate filter alone caps it at 0.80-0.85. |
| - **Over-confident out of the box** (ECE about 0.19) and over-confident on some systematic mistakes (see the form test). Do not trigger irreversible actions (payments, deletions, sending messages) based on its confidence. |
| - **No element positions in the input**, full-page screenshots only (no viewport/scroll handling), pages truncated after 6400 px. |
| - **Distribution:** English web pages from the 2022-2023 Mind2Web snapshot; modern single-page apps, pop-ups, shadow DOM and iframes are not covered. Live DOM to tree conversion in `code/` is an approximation of the dataset format (the dataset has almost no state attributes). |
| - **One seed, one run.** No ablation isolates the effect of `always_pos`, the tree format, and rank 32 in the final configuration separately. |
| - **Possible contamination** with the pre-training data of the Gemma backbone (web pages) cannot be ruled out. |
|
|
| ## How to use |
|
|
| Requirements: a GPU with bf16 support (inference memory was not measured; training peaked at 38.6 GB with gradients), `torch`, `transformers` (with Gemma 4 support), `peft`, `playwright`, `lxml`, `scikit-learn`, `pillow`. The code lives in `code/`; `inference_example.py` shows a complete loop. |
|
|
| ```python |
| import sys; sys.path.insert(0, "code") |
| from m2w_form_demo import Gemma, aday_listesi, softmax |
| |
| model = Gemma("senpaicq/m2w-gemma4-e2b-pointer", shots_dir="tmp_shots") # downloads adapter + head |
| |
| # `page` is a Playwright page (viewport width 1280). Candidates are extracted from the live DOM: |
| cands, raw = aday_listesi(page, task="Find flights from NYC to Paris", history=[], arm="etiketli") |
| png = page.screenshot(full_page=True) |
| ex = {"task": "Find flights from NYC to Paris", "history": [], "cands": cands, "pos_idx": [], "op": "CLICK"} |
| logits, op_logits = model(ex, png, None) # logits has n_candidates + 1 entries; the last one is "none" |
| p = softmax(logits, T=1.0) # T = 1.0 is raw; fit your own temperature on held-out data |
| j = int(p.argmax()) |
| print("none" if j == len(cands) else cands[j]["text"], float(p[j])) |
| ``` |
|
|
| Files in this repository: `adapter_model.safetensors` and `adapter_config.json` (LoRA), `head.pt` (pointer and operation heads), `m2w_config.json` (inference settings), `results/` (validation log, final metrics, per-step top-1 confidence and correctness for each split; no dataset content), `code/` (data preparation, filtering, formatting, training, evaluation and live-demo scripts), `inference_example.py`. |
|
|
| ## Reproducing |
|
|
| The scripts in `code/` reproduce the pipeline: `m2w_prep.py` (candidates, TF-IDF order) -> `m2w_filter.py` (K = 160) -> `m2w_format.py --fmt tree` -> `m2w_ekran_goruntuleri_cikar.py` (screenshots) -> `m2w_train_lora_gorsel.py` (arguments used: `--layer 30 --epochs 4 --accum 8 --rank 32 --lora_mlp --max_cand_tok 64 --max_seq 4600 --img_tokens 280 --max_strips 5 --always_pos`). Comments are partly in Turkish. The dataset is not redistributed here; download it from its original source. |
|
|
| ## License, data and attribution |
|
|
| - Our adapter weights and code are released under Apache-2.0. The base model `google/gemma-4-E2B` is listed as Apache-2.0 on its Hugging Face page; see its license terms. |
| - The model was trained on [Multimodal-Mind2Web](https://huggingface.co/datasets/osunlp/Multimodal-Mind2Web). **We did not verify the dataset's license for this card; check the dataset page and terms before any commercial use.** No dataset content is included in this repository. |
| - Screenshots and DOM snapshots in the dataset come from third-party websites. |
|
|
| ## Related work and credit |
|
|
| - **Kev** (Jared Palmer, 2026): open-source, Apache-2.0 family of small "System One" decision models that answer typed questions (choice, score, ...) with probabilities, built as a LoRA adapter plus a small pointer head on a Qwen base ([code](https://github.com/jaredpalmer/kev), [weights](https://huggingface.co/collections/jaredpalmer/kev)). The idea of a LoRA adapter with a pointer head over a closed set of options, and of reading calibrated probabilities from it, comes from this line of work. We are not affiliated with it, and this model is not a Kev checkpoint. |
| - **What is different here:** a different backbone (Gemma 4 E2B, a vision-language model), screenshots as input, a task-specific candidate set built from the DOM (hundreds of web elements per step), a learned "none" key and an operation head, LoRA on attention and MLP projections of layers 0-29 (rank 32), and an evaluation on Multimodal-Mind2Web with calibration and risk-coverage metrics. We did not compare against Kev checkpoints. |
| - **Benchmarks and data:** Mind2Web and Multimodal-Mind2Web (see Citation). |
|
|
| ## Citation |
|
|
| If you use this model, please also cite the benchmark and the multimodal extension: |
|
|
| ```bibtex |
| @inproceedings{deng2023mind2web, |
| title = {Mind2Web: Towards a Generalist Agent for the Web}, |
| author = {Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Samuel and Wang, Boshi and Sun, Huan and Su, Yu}, |
| booktitle = {NeurIPS Datasets and Benchmarks Track}, |
| year = {2023} |
| } |
| @inproceedings{zheng2024seeact, |
| title = {GPT-4V(ision) is a Generalist Web Agent, if Grounded}, |
| author = {Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu}, |
| booktitle = {ICML}, |
| year = {2024} |
| } |
| ``` |
|
|
