README.md

---
base_model: Qwen/Qwen3.5-27B
library_name: peft
license: apache-2.0
pipeline_tag: text-generation
language:
- en
tags:
- lora
- sft
- qlora
- clembench
- game-playing
- process-reward-model
- transformers
- trl
---

# Qwen3.5-27B-sft-ep1

A QLoRA supervised fine-tune of [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) trained on high-scoring game trajectories from the [Playpen](https://github.com/lm-playpen/playpen) benchmark (clembench 2.0). This model serves as the policy in a Process Reward Model (PRM) guided inference pipeline, where a companion PRM scores candidate responses at each game turn to select the best action.

## Model Details

- **Developed by:** Diginyx
- **Base model:** Qwen/Qwen3.5-27B
- **Model type:** Causal LM — LoRA adapter (PEFT)
- **Language:** English
- **License:** Apache 2.0
- **Fine-tuning method:** QLoRA (4-bit NF4 base + LoRA adapters)
- **Training framework:** TRL + HuggingFace PEFT

## Training Methodology

The model is trained via supervised fine-tuning on game transcripts where the outcome was a win (positive clemscore contribution), filtered from rollouts of the base Qwen3.5-27B-Instruct model playing all clembench 2.0 games. The goal is to teach the policy the turn-level response patterns associated with successful multi-player game trajectories.

**Training pipeline:**
1. Run the base model on all clembench games to collect rollout transcripts
2. Filter to transcripts with a positive game outcome (win)
3. Fine-tune on the winning turns using QLoRA

**Design decisions:**
- **Positive-only filtering**: Using only winning trajectories (rather than all rollouts with reward labels) avoids the model learning from ambiguous partial-credit or losing sequences, keeping the training signal clean.
- **4-bit QLoRA**: Reduces VRAM from ~55 GB to ~14 GB, allowing the full 27B model to train on a single 48 GB A40 alongside optimizer states. This makes training accessible without multi-node tensor parallelism.
- **LoRA over full fine-tune**: Preserves the base model's general language capabilities while adapting the turn-level game response style. The small adapter (r=16) also prevents overfitting on the filtered game corpus.
- **Max length 1024**: Game turns are typically short; truncating at 1024 tokens keeps the full-vocabulary causal-LM logits tensor (batch × seq × 152k vocab) within GPU memory budget.

## Training Data

- **Dataset:** [colab-potsdam/playpen-data](https://huggingface.co/datasets/colab-potsdam/playpen-data) — clembench 2.0 game instances (training split)
- **Games:** All games present in the benchmark (wordle, taboo, reference, clean_up, and others)
- **Filtering:** Turn-level transcripts where the final game outcome was a win
- **Preprocessing:** Chat-templated using Qwen3.5 instruction template with `enable_thinking=False`

## Hyperparameters

| Parameter | Value |
|---|---|
| Learning rate | 2e-4 |
| LR scheduler | Cosine with warmup |
| Epochs trained | 1 (early stopping on val loss) |
| Per-device batch size | 4 |
| Effective batch size | 128 (auto grad-accum across GPUs) |
| Max sequence length | 1024 tokens |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Quantization | 4-bit NF4 (bitsandbytes) |
| Compute dtype | bfloat16 |
| Optimizer | paged_adamw_8bit |
| Val loss (best checkpoint) | 0.2511 |
| Val token accuracy | 94.0% |

## Compute

| Resource | Details |
|---|---|
| Hardware | 4× NVIDIA A40 (48 GB) |
| Cluster | University of Michigan HPC (SLURM) |
| Training time | ~1 epoch over the filtered positive corpus |
| Total FLOPs | ~2.64 × 10¹⁸ |

## Evaluation

Evaluated on the [Playpen](https://github.com/lm-playpen/playpen) benchmark (clembench 2.0) using **clemscore** (quality-weighted success rate across all games) and **statscore** (static benchmark aggregate).

When used with the companion PRM ([Diginyx/Qwen3.5-27B-prm-ep1](https://huggingface.co/Diginyx/Qwen3.5-27B-prm-ep1)) in a best-of-N or beam search guided inference setup, this model achieves higher clemscore than the greedy baseline.

## Usage

### Standalone (greedy inference)

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-27B",
    quantization_config=bnb_config,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "Diginyx/Qwen3.5-27B-sft-ep1")
tokenizer = AutoTokenizer.from_pretrained("Diginyx/Qwen3.5-27B-sft-ep1")
```

### With PRM-guided inference (best-of-N)

Install [Playpen](https://github.com/lm-playpen/playpen) and register the model in `model_registry.json`:

```json
{
  "model_name": "Qwen3.5-27B-sft-ep1",
  "backend": "huggingface_local",
  "huggingface_id": "Qwen/Qwen3.5-27B",
  "model_config": {
    "premade_chat_template": true,
    "load_in_4bit": true,
    "chat_template_kwargs": {"enable_thinking": false},
    "peft_model": "Diginyx/Qwen3.5-27B-sft-ep1"
  }
}
```

Then run:

```bash
python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --game-all \
    --n-candidates 4 \
    --temperature 0.7 \
    --max-tokens 2048
```

### With beam search

```bash
python examples/trl/prm_eval.py \
    --policy-model Qwen3.5-27B-sft-ep1 \
    --prm-path Diginyx/Qwen3.5-27B-prm-ep1 \
    --mode beam-search \
    --n-candidates 4 \
    --num-beam-iterations 20 \
    --game-all \
    --temperature 0.7 \
    --max-tokens 2048
```

## Companion Models

- **PRM:** [Diginyx/Qwen3.5-27B-prm-ep1](https://huggingface.co/Diginyx/Qwen3.5-27B-prm-ep1) — process reward model trained to score turn-level responses

## Framework Versions

- PEFT 0.19.1
- TRL
- Transformers
- bitsandbytes