Add first_error GRPO LoRA adapters (scalar / discrete), 2026-08-17

README.md

@@ -0,0 +1,33 @@
+---
+base_model: Qwen/Qwen3-VL-2B-Instruct
+library_name: peft
+pipeline_tag: image-text-to-text
+tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
+---
+
+# AF_RL — AndroidFlux GRPO LoRA adapters
+
+GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error
+recovery. Each adapter lives in its own subfolder.
+
+| subfolder | reward model | final reward |
+|---|---|---|
+| `scalar-firsterror-20260817` | Bradley-Terry scalar head | +4.608 |
+| `discrete-firsterror-20260817` | judge-token `<\|+\|>` / `<\|-\|>` margin | −2.273 |
+
+Reward scales are **not comparable across the two** — they come from different
+reward models. Compare policies by downstream task performance, not by these numbers.
+
+```python
+from peft import AutoPeftModelForCausalLM
+model = AutoPeftModelForCausalLM.from_pretrained(
+    "YeMoKoo/AF_RL", subfolder="scalar-firsterror-20260817", torch_dtype="bfloat16")
+```
+
+Both were trained on the `first_error` split (2,662 samples) of
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
+for 1 epoch with identical hyperparameters — the reward model is the only
+difference. See each subfolder's `README.md` for full details.
+
+Intended comparison (run elsewhere, on held-out data):
+base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.

discrete-firsterror-20260817/README.md

@@ -0,0 +1,81 @@
+# discrete-firsterror-20260817
+
+GRPO-trained LoRA adapter for **Qwen3-VL-2B-Instruct**, optimised for GUI error
+recovery on Android. The policy sees the screen *immediately after the first
+erroneous action* of a trajectory and must produce the next recovery action.
+
+Reward model: **discrete** — judge-token `<|+|>` / `<|-|>` log-probability margin.
+
+## Load
+
+```python
+from peft import AutoPeftModelForCausalLM
+model = AutoPeftModelForCausalLM.from_pretrained(
+    "YeMoKoo/AF_RL", subfolder="discrete-firsterror-20260817", torch_dtype="bfloat16")
+```
+
+Or with the base model loaded explicitly:
+
+```python
+from transformers import AutoModelForCausalLM
+from peft import PeftModel
+base = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen3-VL-2B-Instruct",
+    revision="89644892e4d85e24eaac8bacfd4f463576704203",
+    torch_dtype="bfloat16")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="discrete-firsterror-20260817")
+```
+
+The base checkpoint is pinned in `adapter_config.json`
+(`Qwen/Qwen3-VL-2B-Instruct` @ `89644892e4d85e24eaac8bacfd4f463576704203`) and is
+downloaded automatically.
+
+> `target_modules` is a **regex** over `model.language_model.*`. Verify the adapter
+> actually attached (`sum("lora" in n for n, _ in model.named_parameters()) > 0`);
+> a different `transformers` version can silently match zero modules.
+> Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
+directory `data_rl_v0_from_error_only`, filtered to `sampling_category == "first_error"`.
+
+| | |
+|---|---|
+| samples | 2,662 (unique 2,662) |
+| images | 12,648 |
+| policy / reward history window | 10 / 5 |
+| hidden labels exported | false |
+| dataset SHA-256 | `d6a7463520feb40403f806640622435e4abacad7dcfacc119bc399a5edd8c6e7` |
+
+Prompts are label-free: no sampling category, error location, or recorded next
+action is present at runtime.
+
+## Training
+
+LoRA r=8, alpha=32, dropout=0.05, `target_modules=all-linear` (ViT and aligner
+frozen, LLM trainable). GRPO with `num_generations=8`, `generation_batch_size=8`,
+temperature 0.9, top_p 0.95, lr 1e-6, 1 epoch, warmup_ratio 0.03, weight_decay 0,
+beta 0.001, `scale_rewards=group`, max_length 8192, max_completion_length 256,
+seed 42, bf16. vLLM colocate rollout. 2×H100 DDP
+(`per_device_train_batch_size=1`, `gradient_accumulation_steps=4`) + 1×H100 for
+the reward model. 2,662 optimizer steps.
+
+Reward model: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie` @ `05bf461bed27a18ee18cf9eacc6706f424a53d5e`, invalid-action penalty −5.0.
+
+ms-swift `23e06ccd93ea00533c0e14d579095f45ac7976bc` (4.5.0.dev0), Python 3.11.15.
+
+## Final metrics
+
+| metric | value |
+|---|---|
+| reward (final) | -2.2728 |
+| reward, first 10 logs -> last 10 logs | -4.7019 -> -3.1675 |
+| reward_std (final) | 1.5304 |
+| frac_reward_zero_std (final) | 0.0000 |
+| train_loss | 0.00109 |
+| completion length min/mean/max | 128.0 / 166.9 / 217.0 |
+| clipped_ratio | 0.000 |
+| train_runtime | 32083 s |
+
+Trained 2026-08-17. Checkpoint `checkpoint-2662` (final, 1 full epoch).

discrete-firsterror-20260817/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-2B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": "89644892e4d85e24eaac8bacfd4f463576704203",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(up_proj|down_proj|v_proj|k_proj|gate_proj|q_proj|o_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

discrete-firsterror-20260817/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:498340cf3048290a1db48dcd3552246218831697fe35b136f4a76e2b9863c043
+size 17490168

scalar-firsterror-20260817/README.md

@@ -0,0 +1,81 @@
+# scalar-firsterror-20260817
+
+GRPO-trained LoRA adapter for **Qwen3-VL-2B-Instruct**, optimised for GUI error
+recovery on Android. The policy sees the screen *immediately after the first
+erroneous action* of a trajectory and must produce the next recovery action.
+
+Reward model: **scalar** — Bradley-Terry scalar head; raw scalar reward.
+
+## Load
+
+```python
+from peft import AutoPeftModelForCausalLM
+model = AutoPeftModelForCausalLM.from_pretrained(
+    "YeMoKoo/AF_RL", subfolder="scalar-firsterror-20260817", torch_dtype="bfloat16")
+```
+
+Or with the base model loaded explicitly:
+
+```python
+from transformers import AutoModelForCausalLM
+from peft import PeftModel
+base = AutoModelForCausalLM.from_pretrained(
+    "Qwen/Qwen3-VL-2B-Instruct",
+    revision="89644892e4d85e24eaac8bacfd4f463576704203",
+    torch_dtype="bfloat16")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="scalar-firsterror-20260817")
+```
+
+The base checkpoint is pinned in `adapter_config.json`
+(`Qwen/Qwen3-VL-2B-Instruct` @ `89644892e4d85e24eaac8bacfd4f463576704203`) and is
+downloaded automatically.
+
+> `target_modules` is a **regex** over `model.language_model.*`. Verify the adapter
+> actually attached (`sum("lora" in n for n, _ in model.named_parameters()) > 0`);
+> a different `transformers` version can silently match zero modules.
+> Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
+directory `data_rl_v0_from_error_only`, filtered to `sampling_category == "first_error"`.
+
+| | |
+|---|---|
+| samples | 2,662 (unique 2,662) |
+| images | 12,648 |
+| policy / reward history window | 10 / 5 |
+| hidden labels exported | false |
+| dataset SHA-256 | `d6a7463520feb40403f806640622435e4abacad7dcfacc119bc399a5edd8c6e7` |
+
+Prompts are label-free: no sampling category, error location, or recorded next
+action is present at runtime.
+
+## Training
+
+LoRA r=8, alpha=32, dropout=0.05, `target_modules=all-linear` (ViT and aligner
+frozen, LLM trainable). GRPO with `num_generations=8`, `generation_batch_size=8`,
+temperature 0.9, top_p 0.95, lr 1e-6, 1 epoch, warmup_ratio 0.03, weight_decay 0,
+beta 0.001, `scale_rewards=group`, max_length 8192, max_completion_length 256,
+seed 42, bf16. vLLM colocate rollout. 2×H100 DDP
+(`per_device_train_batch_size=1`, `gradient_accumulation_steps=4`) + 1×H100 for
+the reward model. 2,662 optimizer steps.
+
+Reward model: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring` @ `b7d953e59c49a3108416f860d430712d8b81094f`, invalid-action penalty −5.0.
+
+ms-swift `23e06ccd93ea00533c0e14d579095f45ac7976bc` (4.5.0.dev0), Python 3.11.15.
+
+## Final metrics
+
+| metric | value |
+|---|---|
+| reward (final) | +4.6084 |
+| reward, first 10 logs -> last 10 logs | +4.3925 -> +5.0904 |
+| reward_std (final) | 2.6275 |
+| frac_reward_zero_std (final) | 0.0000 |
+| train_loss | 0.00098 |
+| completion length min/mean/max | 103.5 / 139.0 / 189.0 |
+| clipped_ratio | 0.000 |
+| train_runtime | 31523 s |
+
+Trained 2026-08-17. Checkpoint `checkpoint-2662` (final, 1 full epoch).

scalar-firsterror-20260817/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-2B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": "89644892e4d85e24eaac8bacfd4f463576704203",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(q_proj|gate_proj|o_proj|k_proj|up_proj|v_proj|down_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

scalar-firsterror-20260817/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a3fff83b0171d0cfcf87ce20c87c534baa707438033094caf8aa5aeef6955dd
+size 17490168