README.md

---
base_model: Qwen/Qwen3-VL-2B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
---

# AF_RL — AndroidFlux GRPO LoRA adapters

GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error
recovery. Each adapter lives in its own subfolder.

| subfolder | data split | reward model | final reward |
|---|---|---|---|
| `scalar-firsterror-20260817` | first_error (2,662) | Bradley-Terry scalar head | +4.608 |
| `discrete-firsterror-20260817` | first_error (2,662) | judge-token margin | −2.273 |
| `scalar-preerror-20260818` | pre_error (2,669) | Bradley-Terry scalar head | +4.982 |
| `discrete-preerror-20260818` | pre_error (2,669) | judge-token margin | −5.053 |

**Reward numbers are not comparable across rows.** The two reward models emit
different scales, and the two splits are different states. Compare policies by
downstream task performance, not by these values.

- **first_error** — the screen *immediately after* the first erroneous action;
  the policy must recover.
- **pre_error** — the screen *before* the erroneous action; the policy must avoid
  the mistake.

```python
import torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct",
    revision="89644892e4d85e24eaac8bacfd4f463576704203",
    dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(
    base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")
```

Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` /
`AutoPeftModelForCausalLM` do **not** work — use `AutoModelForImageTextToText`.
After loading, assert
`sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392`.

All four were trained from
`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`
for 1 epoch with identical hyperparameters — only the reward model and the data
split differ. See each subfolder's `README.md` for full details.

Intended comparison (run elsewhere, on held-out data):
base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.