siddharthmb/2026.RA.Fairness-GRPO-v2-Adapters

2026.RA.Fairness-GRPO-v2 — trained LoRA adapters (the λ-frontier)

The trained LoRA adapters from the fairness-GRPO v2 campaign: can outcome-aligned RL (GRPO on an engine-computed, text-blind log-Nash reward) make an LLM negotiator distribute surplus more fairly on held-out games — not merely close more deals — and generalize to settings where the numbers are not machine-readable?

The evaluation record for these adapters is the dataset siddharthmb/2026.RA.Fairness-GRPO-v2 (contrasts, telemetry, verdicts). This repo is the weights those numbers were computed from. Full methodology + the machine-derived final tables: research note experiments/rational_agents/research-notes/0028-fairness-grpo-v2.md; narrative hub: https://rational-agents-runs.sidmb.com/grpo-v2-lam1-two-attractors/.

What this is

Each folder <arm>_checkpoint-<STEP>/ is a complete PEFT adapter (r32/α64, LoRA on q/k/v/o/gate/up/down) over Qwen/Qwen3-8B, thinking off, saved at global training step STEP. STEP is continuous within an arm across its resume segments (see "Training provenance"). Three arms trace the λ-mixture, the swept variable — seat i trains on R_i(λ) = (1−λ)·g(z_i) + λ·R_table, where g is the smoothed clipped log-Nash utility (g_floor = −17.5) and R_table is the mean over seats:

arm	λ	reward	checkpoints (global step)
lam1	1.0	pure table welfare	5, 10, 15, 25, 30, 35, 40, 45, 50
lam05	0.5	mixture	5, 10, 15, 20, 25, 30, 35, 40, 45, 50
lam0	0.0	pure self-interest (selfish control, shaping off)	5, 10, 15

Headline result: the north-star answer is no

Every interpretable rung of every arm fails the preregistered POSITIVE gate. Held-out primary deal-rate degrades at every λ; prose-transfer fairness never resolves favorable. The verdict is a clean, fully-measured negative for this reward family — for negotiation fairness specifically, an algorithmic agent + a parser dominates the RL'd LLM.

Held-out primary deal-rate delta vs untrained Qwen3-8B (paired, game-cluster bootstrapped; the eval bank is instances_grpo_eval_v1):

step	5	10	15	20	25	30	35	40	45	50
λ=1.0	−0.052	−0.143	−0.147	—	−0.226	—	—	−0.453	−0.641	−0.746
λ=0.5	−0.058	−0.046	−0.058	−0.151	−0.267	−0.240	−0.445†	−0.173	−0.271	−0.369
λ=0	−0.028	−0.075	(partial‡)

(closure-conditional fairness metrics are stamped VOID at any rung whose deal rate falls >0.10 below base — a viability screen, Amendment 5(a). † rung 35 sits immediately after a fresh-Adam restart and is anomalously deep. ‡ λ=0 rung 15 stopped for cost, 346 eps, reported partial.)

What the frontier means: closure decay scales with how much a seat is paid on the table rather than on itself. λ=1.0 collapses monotonically to a policy that will not sign at all (prose closes 0.002, ultimatum closes nothing). λ=0.5 buys robustness of closure (half the depth, oscillation not monotone collapse) but never fairness. λ=0 (walk fully priced to self) shows no closure collapse through rung 10 — isolating table-welfare payment as the collapse mechanism, not GRPO-on-this-task.

The sharpest single finding — boundary extraction (from lam1_checkpoint-25): on the ultimatum family the policy learned a 100/0 split accepted every time, placing the responder exactly on its walk-away point (z<ε). Every closure-conditional guard reads this as perfect; only a share/dispersion term (max_share) sees it. Standing program rule: any below-threshold gate carries a share term beside the deal-rate floor.

How to use an adapter

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype=torch.bfloat16, device_map="cuda")
tok  = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "siddharthmb/2026.RA.Fairness-GRPO-v2-Adapters", subfolder="lam1_checkpoint-25")

Reproduce the training

Per arm (via experiments/rational_agents/grpo_v2/train.py), Qwen3-8B LoRA r32/α64, thinking off, g_floor=−17.5, GRPO groups=22 × k=8, frozen population-opponent zoo {bayes-rational, passive-gate, greedy-anchor}:

# λ=1.0 (arm lam1): steps 5-25 at groups=22; resumed from checkpoint-25 for the dense tail 30-50.
python -m grpo_v2.train --lam 1.0 --steps 50 --groups 22 --k 8 --bank instances_grpo_v2_train \
    --checkpoint-steps 5 10 15 25 30 35 40 45 50 --g-floor -17.5 \
    --wandb-run-name grpo_v2_lam1 --wandb-group fairness-grpo-v2
# λ=0.5 (arm lam05): --lam 0.5, same knobs; groups reduced 22→14→12 across resume segments (see provenance).
# λ=0  (arm lam0):  --lam 0.0 --shaping-weight 0 --steps 15 --checkpoint-steps 5 10 15  (selfish control, note 0050).

Training provenance (width changes across resume segments)

Both larger arms were trained across compute migrations (RunPod balance exhaustion → Lambda), so the rollout width (groups) changed mid-arm. This is recorded because the GRPO group-relative baseline normalizes per-instance, so width does not confound the contrasts, but consumers comparing rungs should know it (0028 Amendments 9–11):

• λ=1.0: steps 5–25 and the resume 30–50 both at groups=22 (B200). One fresh-Adam discontinuity at the step-25→30 resume boundary.
• λ=0.5: steps 1–15 at groups=22 (RunPod B200) → 16–31 at 14 → 32–50 at 12 (Lambda H100, 80 GB fit). Fresh Adam at each resume; a completion-token-alignment guard change (drop small-drift rows) landed mid-arm.
• λ=0: 15 steps single segment, shaping off.

Weights & Biases

Project wandb.ai/siddharth-stanford/rational_agents_fairness_grpo, group fairness-grpo-v2. Runs: grpo_v2_lam1 (npz11gav) → grpo_v2_lam1_resume25; grpo_v2_lam05 → grpo_v2_lam05_resume15 → grpo_v2_lam05_resume30b; grpo_v2_lam0.

Cluster paths

• Adapters (source): /nlp/scr/siddharth/ii_mats/rational_agents/grpo_v2/{lam1,lam1_resume25,lam05,lam05_resume15,lam05_resume30b,lam0}/checkpoint-*
• Eval verdicts (JSON): experiments/rational_agents/results/fairness_grpo_v2/eval_*.json
• Eval run dirs: /nlp/scr/siddharth/ii_mats/rational_agents/grpov2eval_*
• Rollout transcripts: .../grpo_v2/*/transcripts/; training telemetry .../*/steps.jsonl
• Log: experiments/rational_agents/self_benefit/LOG.md; note: research-notes/0028-fairness-grpo-v2.md; λ=0 control: research-notes/0050-lam0-selfish-control-prereg.md

Relation

Supersedes the v1 pilot adapters (2026.RA.Fairness-GRPO-lam1, -lam0; dataset 2026.RA.Fairness-GRPO), whose "discipline without distribution" negative (note 0023) is v2's design input. Base model: Qwen/Qwen3-8B.