myjev-decision-4b

An open, Jev-compatible decision model for browser-use/jev-ultrafast. On every browser step jev-ultrafast asks a set of Choice questions: which operation to perform (CLICK, TYPE_TEXT, SELECT, SCROLL_DOWN, SCROLL_UP, WAIT, DONE, BLOCKED) and, for each element operation, which element or option to act on. This model answers all questions of a request in one forward pass and returns, per question, a choice, a probability for every option and a confidence, in the same shape as TypeSafe's hosted Jev model. It never generates text (jev-ultrafast writes typed text with a separate small LLM).

Code, training pipeline and server: github.com/KISHAN8888/myjev.

Architecture

• Body: Qwen/Qwen3-4B-Base transformer (hidden size 2560), no LM head; bf16 on GPU.
• Adapter: LoRA r = 16, α = 32, dropout 0.05 on all linear layers (q/k/v/o and gate/up/down projections), 252 modules, 33.0M parameters (adapter/).
• Scoring head (head.pt, fp32): each option's row ends with a <|box_end|> marker token (an unused Qwen3 vision token, so the vocabulary is unchanged); score = Linear(256→1)(GELU(Linear(2560→256)(h_marker))), softmax over the options of each question.
• Calibration: one temperature per question kind, fit on a held-out calibration split: T_operation = 1.131, T_target = 1.365 (config.json). Temperatures change confidences, never choices.
• Input: goal, last 10 actions, URL/title/visible page text, the element table, then each question with its options. The constant jev-ultrafast instruction text is omitted (include_rules: false). Budget 4,096 tokens; page text and non-candidate rows are trimmed first, candidates are never dropped. All questions share the state prefix and are packed into one sequence with a block attention mask.

Files: adapter/ (PEFT LoRA weights + config), head.pt (MLP head state dict), tokenizer/, config.json (base model, marker, max_len, temperatures, calibration stats, training hyperparameters).

Training

• Data: Mlchemist/myjev-browser-decisions v3 train split: 21,646 steps from 301 websites (Mind2Web train, WebLINX train, WebChain, NNetNav live/WebArena, Go-Browse), all converted to jev-ultrafast's exact request format; no step on any benchmark, validation or calibration website. Per-record weights: NNetNav 0.5, BLOCKED proxies 0.3, TYPE_TEXT ×1.25, SELECT ×1.5.
• Objective: per question, soft-target cross-entropy + 0.5 × Brier score; trained on the operation question and the gold operation's target question.
• Optimisation: AdamW, warmup 5% then cosine to 0, effective batch 16 records, grad-clip 1.0, bf16, gradient checkpointing, one H100 80GB (~5 s/step, peak 25.5 GB).
• Schedule: (1) one epoch on an earlier build of the data (22,320 steps; LR 2e-4 LoRA / 1e-3 head); (2) one continuation epoch on the label-fixed v3 data (LR 1e-4 / 5e-4, fresh optimizer), resumed exactly after an interruption. This checkpoint is the end of (2), calibrated on the 652-step calibration split.

Evaluation

Frozen benchmark: human-labelled steps on 84 websites unseen in training (Mind2Web test_website + test_domain, WebLINX test_web). The 106 steps Jev's API rejected as too large are excluded for both systems (3,239 scored). Step SR = right operation and right element.

Headline — Mind2Web (Cross-Website + Cross-Domain, 2,074 steps, 62 websites): Step SR 64.8% vs 57.8% for TypeSafe Jev 1.13 (+7.0, paired 95% CI 4.9–9.2); ahead on 44 of 62 websites.

	Cross-Website (660)			Cross-Domain (1,414)			WebLINX test_web (1,165)			All (3,239)
	Ele. Acc	Op. Acc	Step SR	Ele. Acc	Op. Acc	Step SR	Ele. Acc	Op. Acc	Step SR	Ele. Acc	Op. Acc	Step SR
TypeSafe Jev 1.13	66.2	79.5	63.2	56.4	80.6	55.3	51.9	47.9	32.6	57.2	68.6	48.7
This model	73.8	90.5	71.4	62.8	91.1	61.7	62.6	79.1	57.2	65.2	86.6	62.1

	ECE action	ECE element	Latency
Jev 1.13	0.062	0.077	321 ms p50 (API)
This model	0.017	0.021	~105 ms (H100, batched scoring; not like-for-like)

Caveats:

• The all-splits gap overstates the lead: WebLINX test_web penalises Jev for conventions it never saw (scroll-while-reading as gold, click-then-type labelled CLICK, chat-transcript goals), and this model was trained on WebLINX train. Dropping scroll-gold steps and accepting typing into a clicked field: 56.9 (Jev) vs 64.4 (ours) overall, 51.5 vs 61.5 on WebLINX.
• Candidate sets are viewport-filtered (median 34 elements), so numbers are not comparable to published Mind2Web results. Benchmark labels have one gold answer (plus audited equivalents); label ceiling ≈ 95%.
• Weaknesses: TYPE_TEXT steps 65.6% vs Jev 77.1%; SELECT 39.6% vs 81.2% (only 48 steps). Strong on CLICK (62.2% vs 47.2%) and SCROLL_DOWN (66.4% vs 5.1%).
• DONE and BLOCKED are trained but not benchmarked (no trustworthy human labels yet). Step accuracy is not live task success.

How to use

git clone https://github.com/KISHAN8888/myjev && cd myjev
pip install -e ".[model,serve]"
huggingface-cli download Mlchemist/myjev-decision-4b --local-dir ckpt/myjev-decision-4b

from myjev.model.predict import Predictor

pred = Predictor("ckpt/myjev-decision-4b")      # downloads Qwen/Qwen3-4B-Base; cuda (bf16) if available, else cpu (fp32)
request = {...}                                  # a jev-ultrafast request body: {"model", "state", "questions"}
answers = pred.answer([request])[0]
# {"operation": {"choice": "CLICK", "probabilities": {...}, "confidence": 0.93},
#  "click_target": {"choice": "25", "probabilities": {...}, "confidence": 0.71}, ...}

Drop-in server for jev-ultrafast (POST /v1/systemone, returns {"answers": ..., "model": ...}):

MYJEV_CKPT=ckpt/myjev-decision-4b uvicorn myjev.model.server:app --host 0.0.0.0 --port 8000

Then replace https://api.typesafe.ai/v1/systemone in jev-ultrafast's jev_ultrafast/model.py with http://<host>:8000/v1/systemone.

Batch scoring: python -m myjev.model.predict --ckpt ckpt/myjev-decision-4b --data benchmark.jsonl --out pred.jsonl.

The checkpoint is loaded unmerged (merge=False): the temperatures were fit on the unmerged bf16 model. Tested with transformers 5.18.0, peft 0.21.2, torch 2.14.1. The 4B body needs 8 GB of GPU memory in bf16 (16 GB RAM on CPU in fp32). Benchmark rows can reach ~6k tokens with all questions packed.

Limitations

• Trained and evaluated on single-step decisions from recorded datasets, not live browsing.
• ~45% of training data (NNetNav, Go-Browse, WebLINX) comes from accessibility trees rather than the real snapshot.js; Mind2Web pages were rendered without CSS. Real jev-ultrafast pages may look different.
• NNetNav/Go-Browse labels are agent-generated and noisier (down-weighted). WebLINX goals are chat transcripts.
• English-centric; mostly e-commerce, travel, information and WebArena sites.
• Not a safety layer: do not let it act on accounts, payments or irreversible actions without human oversight.

Licence

CC-BY-NC-SA-4.0 (non-commercial, share-alike), because the training data includes WebLINX (CC-BY-NC-SA-4.0). Other training sources: Mind2Web CC-BY-4.0, WebChain CC-BY-4.0, NNetNav Apache-2.0, Go-Browse MIT (repository licence). Base model Qwen3-4B-Base: Apache-2.0. Code: Apache-2.0 (GitHub repo). Not affiliated with TypeSafe or Browser Use; Jev was used only as a baseline through its public API.

Citation

@misc{myjev2026,
  title  = {myjev: an open decision model for jev-ultrafast browser agents},
  author = {KISHAN8888},
  year   = {2026},
  url    = {https://huggingface.co/Mlchemist/myjev-decision-4b}
}

Please also cite the data sources: Mind2Web (Deng et al., 2023), WebLINX (Lù et al., 2024), WebChain, NNetNav (Murty et al., 2024), Go-Browse (Gandhi & Neubig, 2025), and Qwen3 (Qwen Team, 2025).