Add four Qwen3-VL-4B GRPO adapters (UI-Genie / AndroidFlux mixes, checkpoint-150) + README section

README.md

@@ -1,5 +1,7 @@
 ---
-base_model: Qwen/Qwen3-VL-2B-Instruct
+base_model:
+- Qwen/Qwen3-VL-2B-Instruct
+- Qwen/Qwen3-VL-4B-Instruct
 library_name: peft
 pipeline_tag: image-text-to-text
 tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
@@ -51,3 +53,21 @@ split differ. See each subfolder's `README.md` for full details.
 
 Intended comparison (run elsewhere, on held-out data):
 base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.
+
+## Qwen3-VL-4B adapters (2026-09-24/25)
+
+Four offline-GRPO LoRA adapters for **Qwen3-VL-4B-Instruct**, trained with the same recipe, reward model
+(`Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced`, Bradley-Terry scalar + GPRS) and budget
+(2,400 prompts, 150 steps); only the training-data mix differs. HV65 success rate (65 human-verified AndroidWorld tasks):
+
+| subfolder | training data | clean | error_t | mean |
+|---|---|---:|---:|---:|
+| (base Qwen3-VL-4B-Instruct, no RL) | — | 20.0 | 21.5 | 20.8 |
+| `grpo4b-ui100-20260924` | UI-Genie 100% | 23.1 | 16.9 | 20.0 |
+| `grpo4b-t100-20260924` | AndroidFlux t 100% | 18.5 | 30.8 | 24.6 |
+| `grpo4b-ui50-t25-t1-25-20260924` | UI 50 / t 25 / t-1 25 | 29.2 | 21.5 | 25.4 |
+| `grpo4b-ui40-t35-t1-25-20260925` | UI 40 / t 35 / t-1 25 | 27.7 | 26.2 | 26.9 |
+
+Load with `subfolder=` exactly as above, but with base `Qwen/Qwen3-VL-4B-Instruct` @
+`ebb281ec70b05090aa6165b016eac8ec08e71b17`, and assert **504** LoRA tensors (not 392). Each subfolder's
+`README.md` has the full recipe and data provenance; `trainer_state.json` has the per-step logs.

grpo4b-t100-20260924/README.md

@@ -0,0 +1,52 @@
+# grpo4b-t100-20260924
+
+GRPO-trained LoRA adapter for **Qwen3-VL-4B-Instruct** (Android GUI agent), trained on **AndroidFlux t 100%**.
+Final checkpoint (step 150 of 150, one epoch).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-4B-Instruct", "ebb281ec70b05090aa6165b016eac8ec08e71b17"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="grpo4b-t100-20260924")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 504, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 504
+```
+
+Use `AutoModelForImageTextToText`; Qwen3-VL is not a causal-LM class. `target_modules` is a regex over
+`model.language_model.*`: if another `transformers` version renames modules, the adapter loads silently as a no-op,
+so always assert the 504 LoRA tensors. Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+2,400 AndroidFlux t prompts (2,080 unique + 320 repeated); 2,400 prompts, one epoch. Exact prompts and order: `Gyubeum/AndroidFlux_RL_Train`, `curated_set/0926-t100.json`.
+AndroidFlux contexts come from the `t` / `t_minus_1` slices of `Gyubeum/AndroidFlux_RL_Train` @ `419373f6`,
+keeping only rows whose goal matches their source trajectory. No rollout or reward-model filtering.
+
+## Training
+
+LoRA r=64, alpha=32, dropout 0, all linear layers of the LLM (vision tower and aligner frozen). Offline GRPO
+(ms-swift): 8 candidates per prompt, 16 prompts per step (batch 128), 150 steps, lr 5e-5 cosine to 0.1x,
+warmup 5%, KL beta 1e-3, temperature 0.9, top-p 0.95, max completion 512 tokens.
+Reward: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced` @ `4ce62423` (Bradley-Terry scalar), shaped with GPRS group win-share
+(no std scaling); candidate coordinates are rescaled from the 0-1000 grid to the RM's declared screen resolution.
+A `terminate` candidate gets 0 unless the context is labelled `is_next_terminate=True`.
+`trainer_state.json` has the per-step logs.
+
+## Evaluation
+
+HV65 (65 human-verified AndroidWorld tasks, greedy, one episode per task and condition), success rate:
+
+| clean | error_t | mean |
+|---:|---:|---:|
+| 18.5% | 30.8% | 24.6% |
+
+Base Qwen3-VL-4B-Instruct without RL: 20.0% / 21.5% / 20.8%. One seed per run.

grpo4b-t100-20260924/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(v_proj|k_proj|up_proj|q_proj|o_proj|down_proj|gate_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

grpo4b-t100-20260924/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e2f74e9aab368f20ca35bc5f713788f5faaf56e44c7af4e8c8a09d6ca32f3f4
+size 264316960

grpo4b-ui100-20260924/README.md

@@ -0,0 +1,52 @@
+# grpo4b-ui100-20260924
+
+GRPO-trained LoRA adapter for **Qwen3-VL-4B-Instruct** (Android GUI agent), trained on **UI-Genie 100%**.
+Final checkpoint (step 150 of 150, one epoch).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-4B-Instruct", "ebb281ec70b05090aa6165b016eac8ec08e71b17"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="grpo4b-ui100-20260924")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 504, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 504
+```
+
+Use `AutoModelForImageTextToText`; Qwen3-VL is not a causal-LM class. `target_modules` is a regex over
+`model.language_model.*`: if another `transformers` version renames modules, the adapter loads silently as a no-op,
+so always assert the 504 LoRA tensors. Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+2,400 UI-Genie prompts; 2,400 prompts, one epoch. The prompts are the first 2,400 of a seed-42 shuffle of a 12,000-prompt stratified reserve drawn from `Gyubeum/UI_Genie_128k` @ `d7987e1f`.
+AndroidFlux contexts come from the `t` / `t_minus_1` slices of `Gyubeum/AndroidFlux_RL_Train` @ `419373f6`,
+keeping only rows whose goal matches their source trajectory. No rollout or reward-model filtering.
+
+## Training
+
+LoRA r=64, alpha=32, dropout 0, all linear layers of the LLM (vision tower and aligner frozen). Offline GRPO
+(ms-swift): 8 candidates per prompt, 16 prompts per step (batch 128), 150 steps, lr 5e-5 cosine to 0.1x,
+warmup 5%, KL beta 1e-3, temperature 0.9, top-p 0.95, max completion 512 tokens.
+Reward: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced` @ `4ce62423` (Bradley-Terry scalar), shaped with GPRS group win-share
+(no std scaling); candidate coordinates are rescaled from the 0-1000 grid to the RM's declared screen resolution.
+A `terminate` candidate gets 0 unless the context is labelled `is_next_terminate=True`.
+`trainer_state.json` has the per-step logs.
+
+## Evaluation
+
+HV65 (65 human-verified AndroidWorld tasks, greedy, one episode per task and condition), success rate:
+
+| clean | error_t | mean |
+|---:|---:|---:|
+| 23.1% | 16.9% | 20.0% |
+
+Base Qwen3-VL-4B-Instruct without RL: 20.0% / 21.5% / 20.8%. One seed per run.

grpo4b-ui100-20260924/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(k_proj|v_proj|o_proj|down_proj|up_proj|gate_proj|q_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

grpo4b-ui100-20260924/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c49cfdffad584c9e451dea97d157ea5175dfab63bfd4f6e4bdad83913c5bf1b3
+size 264316960

grpo4b-ui40-t35-t1-25-20260925/README.md

@@ -0,0 +1,52 @@
+# grpo4b-ui40-t35-t1-25-20260925
+
+GRPO-trained LoRA adapter for **Qwen3-VL-4B-Instruct** (Android GUI agent), trained on **UI 40 / t 35 / t-1 25**.
+Final checkpoint (step 150 of 150, one epoch).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-4B-Instruct", "ebb281ec70b05090aa6165b016eac8ec08e71b17"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="grpo4b-ui40-t35-t1-25-20260925")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 504, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 504
+```
+
+Use `AutoModelForImageTextToText`; Qwen3-VL is not a causal-LM class. `target_modules` is a regex over
+`model.language_model.*`: if another `transformers` version renames modules, the adapter loads silently as a no-op,
+so always assert the 504 LoRA tensors. Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+960 UI-Genie + 840 AndroidFlux t + 600 AndroidFlux t-1; 2,400 prompts, one epoch. Exact prompts and order: `Gyubeum/AndroidFlux_RL_Train`, `curated_set/0926-ui40_t35_t1_25.json`.
+AndroidFlux contexts come from the `t` / `t_minus_1` slices of `Gyubeum/AndroidFlux_RL_Train` @ `419373f6`,
+keeping only rows whose goal matches their source trajectory. No rollout or reward-model filtering.
+
+## Training
+
+LoRA r=64, alpha=32, dropout 0, all linear layers of the LLM (vision tower and aligner frozen). Offline GRPO
+(ms-swift): 8 candidates per prompt, 16 prompts per step (batch 128), 150 steps, lr 5e-5 cosine to 0.1x,
+warmup 5%, KL beta 1e-3, temperature 0.9, top-p 0.95, max completion 512 tokens.
+Reward: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced` @ `4ce62423` (Bradley-Terry scalar), shaped with GPRS group win-share
+(no std scaling); candidate coordinates are rescaled from the 0-1000 grid to the RM's declared screen resolution.
+A `terminate` candidate gets 0 unless the context is labelled `is_next_terminate=True`.
+`trainer_state.json` has the per-step logs.
+
+## Evaluation
+
+HV65 (65 human-verified AndroidWorld tasks, greedy, one episode per task and condition), success rate:
+
+| clean | error_t | mean |
+|---:|---:|---:|
+| 27.7% | 26.2% | 26.9% |
+
+Base Qwen3-VL-4B-Instruct without RL: 20.0% / 21.5% / 20.8%. One seed per run.

grpo4b-ui40-t35-t1-25-20260925/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(gate_proj|v_proj|o_proj|q_proj|up_proj|k_proj|down_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

grpo4b-ui40-t35-t1-25-20260925/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88f0727f571d1fc28df1455db07bb76bf8e6f334af81767d78b3f6abc51e3d2f
+size 264316960

grpo4b-ui50-t25-t1-25-20260924/README.md

@@ -0,0 +1,52 @@
+# grpo4b-ui50-t25-t1-25-20260924
+
+GRPO-trained LoRA adapter for **Qwen3-VL-4B-Instruct** (Android GUI agent), trained on **UI 50 / t 25 / t-1 25**.
+Final checkpoint (step 150 of 150, one epoch).
+
+## Load
+
+```python
+import torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+from peft import PeftModel
+
+BASE, REV = "Qwen/Qwen3-VL-4B-Instruct", "ebb281ec70b05090aa6165b016eac8ec08e71b17"
+
+base = AutoModelForImageTextToText.from_pretrained(
+    BASE, revision=REV, dtype=torch.bfloat16, device_map="cuda")
+model = PeftModel.from_pretrained(base, "YeMoKoo/AF_RL", subfolder="grpo4b-ui50-t25-t1-25-20260924")
+processor = AutoProcessor.from_pretrained(BASE, revision=REV)
+
+# sanity check: must be 504, not 0
+assert sum("lora" in n.lower() for n, _ in model.named_parameters()) == 504
+```
+
+Use `AutoModelForImageTextToText`; Qwen3-VL is not a causal-LM class. `target_modules` is a regex over
+`model.language_model.*`: if another `transformers` version renames modules, the adapter loads silently as a no-op,
+so always assert the 504 LoRA tensors. Trained with `transformers==4.57.6`, `peft==0.19.1`, `torch==2.8.0`.
+
+## Training data
+
+1,200 UI-Genie + 600 AndroidFlux t + 600 AndroidFlux t-1; 2,400 prompts, one epoch. Exact prompts and order: `Gyubeum/AndroidFlux_RL_Train`, `curated_set/0926-ui50_t25_t1_25.json`.
+AndroidFlux contexts come from the `t` / `t_minus_1` slices of `Gyubeum/AndroidFlux_RL_Train` @ `419373f6`,
+keeping only rows whose goal matches their source trajectory. No rollout or reward-model filtering.
+
+## Training
+
+LoRA r=64, alpha=32, dropout 0, all linear layers of the LLM (vision tower and aligner frozen). Offline GRPO
+(ms-swift): 8 candidates per prompt, 16 prompts per step (batch 128), 150 steps, lr 5e-5 cosine to 0.1x,
+warmup 5%, KL beta 1e-3, temperature 0.9, top-p 0.95, max completion 512 tokens.
+Reward: `Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced` @ `4ce62423` (Bradley-Terry scalar), shaped with GPRS group win-share
+(no std scaling); candidate coordinates are rescaled from the 0-1000 grid to the RM's declared screen resolution.
+A `terminate` candidate gets 0 unless the context is labelled `is_next_terminate=True`.
+`trainer_state.json` has the per-step logs.
+
+## Evaluation
+
+HV65 (65 human-verified AndroidWorld tasks, greedy, one episode per task and condition), success rate:
+
+| clean | error_t | mean |
+|---:|---:|---:|
+| 29.2% | 21.5% | 25.4% |
+
+Base Qwen3-VL-4B-Instruct without RL: 20.0% / 21.5% / 20.8%. One seed per run.

grpo4b-ui50-t25-t1-25-20260924/adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-4B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": "ebb281ec70b05090aa6165b016eac8ec08e71b17",
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(gate_proj|down_proj|q_proj|up_proj|k_proj|o_proj|v_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

grpo4b-ui50-t25-t1-25-20260924/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:560ee4fea26fb6d660351432cca3c7a1a80ff3e0b09a55f534bda23795cb10b5
+size 264316960