Add pre_error GRPO LoRA adapters (scalar / discrete), 2026-08-18

README.md

@@ -10,13 +10,21 @@ tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
 GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error
 recovery. Each adapter lives in its own subfolder.
 
-| subfolder | reward model | final reward |
-|---|---|---|
-| `scalar-firsterror-20260817` | Bradley-Terry scalar head | +4.608 |
-| `discrete-firsterror-20260817` | judge-token `<\|+\|>` / `<\|-\|>` margin | −2.273 |
-
-Reward scales are **not comparable across the two** — they come from different
-reward models. Compare policies by downstream task performance, not by these numbers.
+| subfolder | data split | reward model | final reward |
+|---|---|---|---|
+| `scalar-firsterror-20260817` | first_error (2,662) | Bradley-Terry scalar head | +4.608 |
+| `discrete-firsterror-20260817` | first_error (2,662) | judge-token margin | −2.273 |
+| `scalar-preerror-20260818` | pre_error (2,669) | Bradley-Terry scalar head | +4.982 |
+| `discrete-preerror-20260818` | pre_error (2,669) | judge-token margin | −5.053 |
+
+**Reward numbers are not comparable across rows.** The two reward models emit
+different scales, and the two splits are different states. Compare policies by
+downstream task performance, not by these values.
+
+- **first_error** — the screen *immediately after* the first erroneous action;
+  the policy must recover.
+- **pre_error** — the screen *before* the erroneous action; the policy must avoid
+  the mistake.
 
 ```python
 import torch
@@ -28,7 +36,7 @@ base = AutoModelForImageTextToText.from_pretrained(
     revision="89644892e4d85e24eaac8bacfd4f463576704203",
     dtype=torch.bfloat16, device_map="cuda")
 model = PeftModel.from_pretrained(
-    base, "YeMoKoo/AF_RL", subfolder="scalar-firsterror-20260817")
+    base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")
 ```
 
 Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` /
@@ -36,10 +44,10 @@ Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` /
 After loading, assert
 `sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392`.
 
-Both were trained on the `first_error` split (2,662 samples) of
-`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
-for 1 epoch with identical hyperparameters — the reward model is the only
-difference. See each subfolder's `README.md` for full details.
+All four were trained from
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`
+for 1 epoch with identical hyperparameters — only the reward model and the data
+split differ. See each subfolder's `README.md` for full details.
 
 Intended comparison (run elsewhere, on held-out data):
 base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.

discrete-preerror-20260818/README.md

@@ -0,0 +1,85 @@
+# discrete-preerror-20260818
+
+GRPO-trained LoRA adapter for **Qwen3-VL-2B-Instruct** on Android GUI recovery.
+The policy sees a screen from the `pre_error` split - the state *before* the
+erroneous action of a trajectory - and must produce the next action.
+
+Reward model: **discrete** - judge-token `<|+|>` / `<|-|>` log-probability margin.
+
+Companion of `discrete-firsterror-20260817`, which is the same recipe on the
+`first_error` split (the screen *after* the first erroneous action).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-2B-Instruct", "89644892e4d85e24eaac8bacfd4f463576704203"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="discrete-preerror-20260818")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 392, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392
+```
+
+Qwen3-VL is a vision-language model (`Qwen3VLForConditionalGeneration`) and is
+**not** registered in `MODEL_FOR_CAUSAL_LM_MAPPING`, so `AutoModelForCausalLM` and
+`AutoPeftModelForCausalLM` do not work here - use `AutoModelForImageTextToText`.
+
+The base checkpoint is also pinned in `adapter_config.json`.
+
+> `target_modules` is a **regex** over `model.language_model.*`. A different
+> `transformers` version can rename modules so the regex matches **zero** of them;
+> the adapter then loads without any error and the model behaves exactly like the
+> base. Always assert the 392 LoRA parameter count above.
+> Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
+directory `data_rl_v0_from_error_only`, filtered to `sampling_category == "pre_error"`.
+
+| | |
+|---|---|
+| samples | 2,669 (unique 2,669) |
+| images | 10,916 |
+| policy / reward history window | 10 / 5 |
+| hidden labels exported | false |
+| dataset SHA-256 | `b9d33be41f9d9e1549221886917b653250d786e7c0d8cdfafba9675ed1a76503` |
+
+Prompts are label-free: no sampling category, error location, or recorded next
+action is present at runtime.
+
+## Training
+
+LoRA r=8, alpha=32, dropout=0.05, `target_modules=all-linear` (ViT and aligner
+frozen, LLM trainable). GRPO with `num_generations=8`, `generation_batch_size=8`,
+temperature 0.9, top_p 0.95, lr 1e-6, 1 epoch, warmup_ratio 0.03, weight_decay 0,
+beta 0.001, `scale_rewards=group`, max_length 8192, max_completion_length 256,
+seed 42, bf16. vLLM colocate rollout. Single H100 for the policy
+(`per_device_train_batch_size=1`, `gradient_accumulation_steps=8`) plus one H100
+for the reward model. 2,669 optimizer steps, one prompt per step.
+
+Reward model: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie` @ `05bf461bed27a18ee18cf9eacc6706f424a53d5e`, invalid-action penalty -5.0.
+
+ms-swift `23e06ccd93ea00533c0e14d579095f45ac7976bc` (4.5.0.dev0), Python 3.11.15.
+
+## Final metrics
+
+| metric | value |
+|---|---|
+| reward (final) | -5.0528 |
+| reward, first 10 logs -> last 10 logs | -3.9981 -> -4.0804 |
+| reward_std (final) | 1.3360 |
+| frac_reward_zero_std (final) | 0.0000 |
+| train_loss | 0.00084 |
+| completion length min/mean/max | 107.0 / 132.8 / 166.5 |
+| clipped_ratio | 0.000 |
+| train_runtime | 44590 s (12.39 h) |
+
+Trained 2026-08-18. Checkpoint `checkpoint-2669` (final, 1 full epoch).

discrete-preerror-20260818/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-2B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": "89644892e4d85e24eaac8bacfd4f463576704203",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(v_proj|down_proj|up_proj|k_proj|q_proj|o_proj|gate_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

discrete-preerror-20260818/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:284e4c99969e61d9c026996f5a016fa5d5af7181f096971d22a5657b89978d9a
+size 17490168

scalar-preerror-20260818/README.md

@@ -0,0 +1,85 @@
+# scalar-preerror-20260818
+
+GRPO-trained LoRA adapter for **Qwen3-VL-2B-Instruct** on Android GUI recovery.
+The policy sees a screen from the `pre_error` split - the state *before* the
+erroneous action of a trajectory - and must produce the next action.
+
+Reward model: **scalar** - Bradley-Terry scalar head; raw scalar reward.
+
+Companion of `scalar-firsterror-20260817`, which is the same recipe on the
+`first_error` split (the screen *after* the first erroneous action).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-2B-Instruct", "89644892e4d85e24eaac8bacfd4f463576704203"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 392, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392
+```
+
+Qwen3-VL is a vision-language model (`Qwen3VLForConditionalGeneration`) and is
+**not** registered in `MODEL_FOR_CAUSAL_LM_MAPPING`, so `AutoModelForCausalLM` and
+`AutoPeftModelForCausalLM` do not work here - use `AutoModelForImageTextToText`.
+
+The base checkpoint is also pinned in `adapter_config.json`.
+
+> `target_modules` is a **regex** over `model.language_model.*`. A different
+> `transformers` version can rename modules so the regex matches **zero** of them;
+> the adapter then loads without any error and the model behaves exactly like the
+> base. Always assert the 392 LoRA parameter count above.
+> Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`,
+directory `data_rl_v0_from_error_only`, filtered to `sampling_category == "pre_error"`.
+
+| | |
+|---|---|
+| samples | 2,669 (unique 2,669) |
+| images | 10,916 |
+| policy / reward history window | 10 / 5 |
+| hidden labels exported | false |
+| dataset SHA-256 | `b9d33be41f9d9e1549221886917b653250d786e7c0d8cdfafba9675ed1a76503` |
+
+Prompts are label-free: no sampling category, error location, or recorded next
+action is present at runtime.
+
+## Training
+
+LoRA r=8, alpha=32, dropout=0.05, `target_modules=all-linear` (ViT and aligner
+frozen, LLM trainable). GRPO with `num_generations=8`, `generation_batch_size=8`,
+temperature 0.9, top_p 0.95, lr 1e-6, 1 epoch, warmup_ratio 0.03, weight_decay 0,
+beta 0.001, `scale_rewards=group`, max_length 8192, max_completion_length 256,
+seed 42, bf16. vLLM colocate rollout. Single H100 for the policy
+(`per_device_train_batch_size=1`, `gradient_accumulation_steps=8`) plus one H100
+for the reward model. 2,669 optimizer steps, one prompt per step.
+
+Reward model: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring` @ `b7d953e59c49a3108416f860d430712d8b81094f`, invalid-action penalty -5.0.
+
+ms-swift `23e06ccd93ea00533c0e14d579095f45ac7976bc` (4.5.0.dev0), Python 3.11.15.
+
+## Final metrics
+
+| metric | value |
+|---|---|
+| reward (final) | +4.9824 |
+| reward, first 10 logs -> last 10 logs | +4.7519 -> +3.6207 |
+| reward_std (final) | 0.4867 |
+| frac_reward_zero_std (final) | 0.0000 |
+| train_loss | 0.00083 |
+| completion length min/mean/max | 92.5 / 114.3 / 144.2 |
+| clipped_ratio | 0.000 |
+| train_runtime | 43774 s (12.16 h) |
+
+Trained 2026-08-18. Checkpoint `checkpoint-2669` (final, 1 full epoch).

scalar-preerror-20260818/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-2B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": "89644892e4d85e24eaac8bacfd4f463576704203",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(down_proj|v_proj|up_proj|k_proj|q_proj|o_proj|gate_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

scalar-preerror-20260818/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a349950261e8ba93fbd794ceb8e39fac5a80c87c7f1fb0d8a3977f86b6508007
+size 17490168