leandermaben/werewolf-rl-gspo-v2-trial1_lower_diff

werewolf-rl-gspo-v2 — trial1_lower_diff

Qwen3-4B-Instruct-2507 fine-tuned as a werewolf RL agent using GSPO (sequence-level importance sampling) on the AReaL framework.

• Full run name: trial1_lower_diff_20260420_115050
• Base model: Qwen/Qwen3-4B-Instruct-2507
• Algorithm: GSPO + GRPO group-mean advantage (CODESCOUT-style tight clipping)
• Curriculum: difficulty 1-3, max villagers 4
• Opponents: self-hosted Qwen3-30B-A3B-Instruct-2507 (~75%) + hosted LLMs (25%)
• Hardware: 4× A100 on Modal (3 vLLM DP + 1 FSDP actor)
• Total steps: 100, checkpoints saved every 20 global steps

Checkpoints

Each folder is a full HF-format model checkpoint (safetensors + tokenizer).

Subfolder	Global step	Eval reward	Eval trainable_alive
step-19/	19	0.363	0.43
step-39/	39	0.466	0.57 (peak)
step-59/	59	0.402	0.47
step-79/	79	0.296	0.30
step-99/	99	—	—

step-39/ is the peak by eval trainable-wolf survival. Later checkpoints regress (likely overfitting on the training distribution + growing malformed-tool-call rate).

Load

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "leandermaben/werewolf-rl-gspo-v2-trial1_lower_diff"
subfolder = "step-39"  # peak
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=subfolder)
tok = AutoTokenizer.from_pretrained(repo, subfolder=subfolder)