README.md

---
license: mit
base_model: microsoft/deberta-v3-base
tags:
  - cross-encoder
  - web-agent
  - computer-use
  - mind2web
  - candidate-generation
  - sentence-transformers
datasets:
  - osunlp/Multimodal-Mind2Web
language:
  - en
library_name: sentence-transformers
pipeline_tag: text-ranking
---

# mind2web-candidate-ranker

A **DeBERTa-v3-base cross-encoder** fine-tuned for **candidate generation in web agents**:
given a natural-language task (plus action history) and a DOM element, it scores how likely
that element is the next action target. It is stage 1 of a two-stage computer-use agent
(rank DOM elements → LLM picks the action → Playwright executes).

Trained on [Mind2Web](https://osu-nlp-group.github.io/Mind2Web/) (2,000+ tasks across
137 real websites).

## Results

Evaluated on the three official Mind2Web generalization splits:

| split | acc@1 | recall@3 | recall@5 | recall@10 | recall@20 | MRR |
|---|---|---|---|---|---|---|
| test_task (unseen tasks) | 60.2% | 83.5% | 89.4% | 93.0% | 93.8% | 0.725 |
| test_website (unseen websites) | 50.9% | 76.1% | 85.3% | 92.6% | 95.3% | 0.656 |
| test_domain (unseen domains) | 55.4% | 79.8% | 87.1% | 92.8% | 94.5% | 0.689 |

~93% recall@10 on entirely unseen domains. The recall curve plateaus at k≈20, making
top-20 a good cost/recall operating point for the downstream LLM.

## Usage

```python
from sentence_transformers import CrossEncoder

model = CrossEncoder("torontodeveloper/mind2web-candidate-ranker", num_labels=1, max_length=512)

candidate = "tag: input | aria_label: Destination | placeholder: To?"
query = "task is: Search for a flight from Toronto to New York. Type NYC into the destination field.\nPrevious actions: "

score = model.predict([(candidate, query)])
```

Candidates are serialized DOM elements (`tag`, `aria_label`, `placeholder`, `name`,
`is_clickable`, bounding box, etc.); the query is the task description plus the last
few actions. Score all interactable elements on a page and keep the top-k.

## Training

- Base model: `microsoft/deberta-v3-base` (86M params)
- Data: Mind2Web train split — 7,775 actions, 1 positive + 3 sampled negatives each (31,100 pairs)
- 3 epochs, batch size 8, lr 3e-5, BCEWithLogitsLoss, linear warmup
- Single T4 GPU (~85 minutes), **fp32**

## ⚠️ Precision note

DeBERTa-v3's disentangled attention **overflows fp16** (NaN logits). Load and run this
model in **fp32** (or bf16 on Ampere+). Newer transformers versions may default to fp16
on GPU — pass `torch_dtype=torch.float32` explicitly if needed.

## Pipeline context

This model is the candidate generator from
[mind2web-computer-use-agent](https://github.com/torontodeveloper/mind2web-computer-use-agent):
stage 2 (action prediction) uses FLAN-T5 / GPT-class models over the top-k candidates, and
stage 3 executes via Playwright. Architecture follows
[MindAct](https://github.com/OSU-NLP-Group/Mind2Web) (Deng et al., 2023).