README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
tags:
- reinforcement-learning
- lora
- scienceworld
- agent
- gigpo
---

# ScienceWorld — single-expert LoRA (GiGPO RL, gold-SFT warm start)

LoRA adapters for **Qwen/Qwen2.5-1.5B-Instruct**, trained as a single-expert agent on the
[ScienceWorld](https://sciworld.apps.allenai.org/) text environment (30 elementary-science
tasks) with GiGPO reinforcement learning, warm-started from behaviour cloning on gold paths.

Run: `scienceworld_single_warmstart_ms100_1k_seed0` (seed 0).

## Checkpoints

| folder | step | val test_score | val success_rate | note |
|---|---|---|---|---|
| `single_expert_warmstart/final_step1000/`  | 1000 | 5.11 | 0.271 | final policy (1000 RL steps) |
| `single_expert_warmstart/best_step140/`    | 140  | 5.50 | **0.314** | best checkpoint by `val/success_rate` |

`val/text/test_score` is the mean ScienceWorld raw score / 10 over a fixed, stratified dev
validation set (140 episodes, all 30 task types, greedy decode). So test_score ≈ 5.1 means an
average raw score of ~51/100; success_rate is the fraction of episodes fully solved (raw
score = 100).

> Note: intermediate checkpoints (e.g. the test_score peaks at steps 210 / 510, both ~5.6–5.7)
> were not retained; only the final and the best-by-success-rate checkpoints are available.

## Training setup

- **Base model:** Qwen/Qwen2.5-1.5B-Instruct
- **Adapter:** LoRA, r = 64, α = 64, target = all-linear (q,k,v,o,gate,up,down)
- **Warm start:** gold-path behaviour-cloning SFT adapter (replay of
  `env.getGoldActionSequence()`; ~6k (prompt, `<think>…</think><action>…</action>`) pairs;
  3 epochs). Cold-start RL never solves a task (val success ≈ 0), so the warm start is
  required to give GiGPO a learning signal.
- **RL:** GiGPO (`adv_estimator=gigpo`, γ = 0.95), lr = 3e-6, 1000 steps.
  - `train_batch_size = 8`, GiGPO group `rollout.n = 8`, `ppo_mini_batch_size = 64`
  - invalid-action penalty coef = 0.1
- **Env:** ScienceWorld, `max_steps = 100`, `history_length = 2` (last 2 obs+action pairs in
  the prompt). Reward = per-step ScienceWorld score delta / 10 (dense).
- **Validation:** fixed stratified dev set (140 episodes, 30 tasks × ~5 variations), greedy
  (temperature 0), every 10 steps.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-1.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(
    model,
    "efficient-moe-agent-project/scienceworld",
    subfolder="single_expert_warmstart/best_step140",  # or final_step1000
)
```