ai-ecoverse/webel-1b

WebEl 1B

WebEl ranks the interactive elements of a web page. Given a screenshot and the page's elements (box, tag, role, label), it answers one of two questions:

• click: where on this page are visitors most likely to click?
• goal: which element starts this task? (e.g. "Download the Acrobat Reader installer")

The output is a probability per element.

Architecture

1.15B parameters: Ettin-encoder-1b (ModernBERT, 28 layers, hidden 1792) reads one sequence per page and SigLIP2-base-patch16-512 encodes the screenshot:

[CLS] task text [SEP]  <64 image tokens per 512-px screenshot tile>  ([EL] element label) x n  [SEP]

• The page (viewport on top of the full-page screenshot, up to 2400 CSS px) is scaled to 3 tiles across on desktop (1 on mobile) and cut into 512x512 tiles; each tile becomes 64 tokens (4x4 pixel shuffle + MLP).
• Each [EL] token gets box, tag and role embeddings and the mean SigLIP2 patch feature under its box.
• A linear head on the [EL] states gives one logit per element; softmax(logits / T) with the device temperature in config.json (desktop 1.2, mobile 1.0) is the probability.

Usage

from PIL import Image
from modeling_webel import WebEl          # in this repo

model = WebEl.from_pretrained("ai-ecoverse/webel-1b")
ranked = model.predict({
    "device": "desktop",                   # or "mobile" (412 CSS px wide)
    "viewport": Image.open("viewport.png"),
    "full": Image.open("full.png"),        # optional full-page screenshot
    "height": 3200,                        # page height in CSS px
    "task": "click",                       # or a goal, e.g. "Find the pricing page"
    "elements": [{"box": [x, y, w, h], "tag": "a", "role": "", "text": "Pricing", "aria": "", "href": "/pricing"}, ...],
}, top=5)
# [{"index": 17, "p": 0.21}, ...]  index into "elements", most likely first

Elements are taken in reading order (top to bottom, left to right), at most 160 per page. ONNX exports for onnxruntime and slicc-ortllama are in ai-ecoverse/webel-1b-ONNX.

Training

• Click distributions from real-user monitoring of Adobe web properties, matched to elements of crawled pages.
• Goals: task descriptions generated by an LLM for crawled pages (Adobe and other public websites), each grounded to the element that starts the task.
• Two stages: goals only, then clicks and goals; weights are an exponential moving average. The training data is not released.

Evaluation

On pages of other websites not seen in training (desktop clicks: 1,421 pages; mobile: 1,128; goals: 4,247 tasks):

	hit@1	hit@5	Brier
desktop click	12.3%	38.4%	0.217
mobile click	13.5%	41.5%	0.243
goal	76.2%	93.4%

hit@k for clicks is the share of observed clicks whose element is among the k most likely ones; Brier is the squared error between the predicted and the observed click distribution (lower is better).

License

Apache-2.0. Base models: Ettin (MIT), SigLIP2 (Apache-2.0); see NOTICE.