Qwen3-VL-8B-Instruct-UI-Genie-scoring

A Bradley-Terry reward model fine-tuned from Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie (itself SFT'd from Qwen/Qwen3-VL-8B-Instruct) on the UI-Genie-RM-517k dataset. This is a merged sequence-classification checkpoint (task_type="seq_cls", num_labels=1, problem_type="regression") trained with the Bradley-Terry pairwise loss. A learned score.weight linear head projects the last non-padding token's hidden state to a scalar reward.

Score = score_head(last non-padding token hidden state) — a single float per input trajectory. Higher score = higher quality action.

2026-08-10 update: an earlier upload of this repo shipped with the untrained bootstrap score.weight (the swift trainer keeps the head out of the LoRA adapter and writes it only to deepspeed checkpoint shards, which a naive merge/export silently drops — see score_head_source.txt in this repo for provenance). score.weight's norm in the old upload matched the untrained bootstrap head exactly; it now differs by norm 0.099 after fine-tuning. This upload replaces it with the actual trained head and was re-validated with rm_eval before publishing (see Evaluation below).

Intended Use

Drop-in scalar reward signal for GUI agent training (e.g. PPO/GRPO, RLHF) or best-of-N selection. Unlike the discrete <|+|> / <|-|> classifier, this model outputs a continuous unbounded reward that is directly usable as a value signal.

Inference

Requirements

pip install transformers torch pillow safetensors

Note: Use HuggingFace transformers directly — vLLM's Qwen3VLForConditionalGeneration loader does not support the additional score.weight tensor.

Scoring with HuggingFace Transformers

import json
import os
import torch
from safetensors import safe_open
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from PIL import Image

MODEL_PATH = "Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring"
MAX_LEN = 8192
DEVICE = "cuda"

# HF tolerates the extra `score.weight` tensor in the checkpoint (warning,
# harmless) — it just isn't wired into the base CausalLM class, so we load
# it ourselves into a separate linear head below.
model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
).to(DEVICE).eval()
processor = AutoProcessor.from_pretrained(MODEL_PATH, max_pixels=1_048_576)
tokenizer = processor.tokenizer


def _load_score_head(model_path: str) -> torch.nn.Linear:
    from huggingface_hub import hf_hub_download

    index_path = hf_hub_download(model_path, "model.safetensors.index.json")
    with open(index_path) as f:
        weight_map = json.load(f)["weight_map"]
    shard_path = hf_hub_download(model_path, weight_map["score.weight"])
    with safe_open(shard_path, framework="pt") as f:
        weight = f.get_tensor("score.weight")
    head = torch.nn.Linear(weight.shape[1], weight.shape[0], bias=False)
    with torch.no_grad():
        head.weight.copy_(weight)
    return head.to(DEVICE, dtype=torch.bfloat16)


score_head = _load_score_head(MODEL_PATH)


def score_action(prefix_messages, response_text, images=None):
    """
    Score a GUI agent action.

    Args:
        prefix_messages: Chat messages up to (not including) the assistant turn.
                         E.g. [{"role": "system", "content": "..."},
                               {"role": "user", "content": [{"type":"text","text":"..."},
                                                             {"type":"image"}]}]
        response_text:   The assistant tool-call response to score.
        images:          List of PIL.Image objects matching image placeholders.

    Returns:
        float: Scalar BT reward (higher = better action).
    """
    structured_msgs = list(prefix_messages) + [
        {"role": "assistant", "content": response_text.strip()}
    ]
    # Mirrors training: chat template closes the assistant turn with
    # <|im_end|>; the score head reads the hidden state at that position.
    full_text = processor.apply_chat_template(
        structured_msgs, tokenize=False, add_generation_prompt=False
    )

    inputs = processor(
        text=[full_text],
        images=images or None,
        truncation=True,
        max_length=MAX_LEN,
        return_tensors="pt",
    ).to(DEVICE)

    with torch.inference_mode():
        outputs = model(**inputs, output_hidden_states=True, return_dict=True)

    last_hidden = outputs.hidden_states[-1]           # (1, seq_len, hidden_dim)
    last_idx = inputs["attention_mask"].sum(dim=1) - 1
    pooled = last_hidden[0, last_idx[0], :]            # (hidden_dim,)
    reward = score_head(pooled.unsqueeze(0).to(score_head.weight.dtype))
    return reward.squeeze().float().item()


# Example: compare two candidate actions
screenshot = Image.open("screenshot.png").convert("RGB")

SYSTEM_PROMPT = "You are a helpful assistant."  # use full mobile_use tool spec in practice

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user",   "content": [
        {"type": "text",  "text": "The user query: tap the search button\n"},
        {"type": "image"},
    ]},
]

action_a = '{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [540, 120]}}'
action_b = '{"name": "mobile_use", "arguments": {"action": "click", "coordinate": [100, 800]}}'

score_a = score_action(messages, action_a, images=[screenshot])
score_b = score_action(messages, action_b, images=[screenshot])

print(f"Action A: {score_a:.4f}")
print(f"Action B: {score_b:.4f}")
print(f"Preferred: {'A' if score_a > score_b else 'B'}")

Pairwise evaluation with rm_eval

python eval_rm.py \
    --rm_path Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring \
    --datasets ui-genie \
    --mode bt \
    --uigenie_jsonl /path/to/reward_data_rm_pairs_last5.jsonl \
    --uigenie_images_dir /path/to/images \
    --output_dir results/

Evaluation

Pairwise accuracy on held-out (chosen, rejected) UI-Genie pairs:

Dataset	Pairs	Pairwise accuracy	Source
UI-Genie (held-out)	1000	90.6% (906/1000, 3 ties)	Fresh rm_eval run, 2026-08-11, on this uploaded checkpoint
UI-Genie (held-out)	1000	90.5% (905/1000, 3 ties)	rm_eval run, 2026-05-01, on this exact checkpoint (mtime-verified)
AndroidFlux (OOD multi-agent replay)	203	61.1% (124/203)	rm_eval run, 2026-04-30, on this exact checkpoint (mtime-verified)

The two UI-Genie runs agree to within 1 pair (906 vs. 905 correct, identical tie count), confirming the fix reproduces consistently.

AndroidFlux is out-of-domain relative to the UI-Genie training data (different agents, different action distribution) and is reported for reference, not as the primary validation signal.

Training Details

Field	Value
Base model	Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie
Training method	Bradley-Terry pairwise loss (LoRA, merged)
Architecture	Qwen3VLForConditionalGeneration + score.weight linear head
Task type	seq_cls (regression, num_labels=1)
Score	Last non-padding token hidden state → linear head → scalar
dtype	bfloat16

Related Models

• Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie — the SFT discrete preference classifier (<|+|> / <|-|>) this model's score.weight was fine-tuned on top of.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}