File size: 2,193 Bytes
fb502e6 a90a801 fb502e6 6cc8fb3 a90a801 fb502e6 6cc8fb3 a90a801 fb502e6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
base_model: Qwen/Qwen3-VL-2B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
---
# AF_RL — AndroidFlux GRPO LoRA adapters
GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error
recovery. Each adapter lives in its own subfolder.
| subfolder | data split | reward model | final reward |
|---|---|---|---|
| `scalar-firsterror-20260817` | first_error (2,662) | Bradley-Terry scalar head | +4.608 |
| `discrete-firsterror-20260817` | first_error (2,662) | judge-token margin | −2.273 |
| `scalar-preerror-20260818` | pre_error (2,669) | Bradley-Terry scalar head | +4.982 |
| `discrete-preerror-20260818` | pre_error (2,669) | judge-token margin | −5.053 |
**Reward numbers are not comparable across rows.** The two reward models emit
different scales, and the two splits are different states. Compare policies by
downstream task performance, not by these values.
- **first_error** — the screen *immediately after* the first erroneous action;
the policy must recover.
- **pre_error** — the screen *before* the erroneous action; the policy must avoid
the mistake.
```python
import torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel
base = AutoModelForImageTextToText.from_pretrained(
"Qwen/Qwen3-VL-2B-Instruct",
revision="89644892e4d85e24eaac8bacfd4f463576704203",
dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(
base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")
```
Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` /
`AutoPeftModelForCausalLM` do **not** work — use `AutoModelForImageTextToText`.
After loading, assert
`sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392`.
All four were trained from
`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`
for 1 epoch with identical hyperparameters — only the reward model and the data
split differ. See each subfolder's `README.md` for full details.
Intended comparison (run elsewhere, on held-out data):
base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.
|