myzleo/playpen-prm-9b

LM Playpen Submission: Qwen3.5-9B + 9B PRM Search

This repository packages the current LM Playpen submission artifacts for a Qwen3.5-9B policy evaluated with a trained Qwen3.5-9B process reward model (PRM) and four PRM-guided search methods.

What Is Included

• prm_checkpoint/: trained 9B PRM adapter checkpoint used for scoring candidate assistant actions.
• base_policy_qwen35_9b/: Qwen3.5-9B base policy weights used for evaluation.
• code/examples_trl/: PRM training, inference, evaluation, and score export scripts.
• code/playpen_search/: implementations of the four search methods: best-of-N, beam search, DVTS, and lookahead.
• scripts/: terminal-run shell entrypoints used for local reproduction.
• eval/: final Playpen .val.json files and score manifests.
• configs/: example model registry for Playpen reproduction.
• MODEL_CARD.md: training card and reproducibility details.
• SUBMISSION_MANIFEST.json: machine-readable summary of the package.
• UPLOAD_TO_HF.md: commands for creating and uploading a Hugging Face repo.

Base Policy Model

The policy model is Qwen3.5-9B. This package now includes a local copy of the full base policy weights in base_policy_qwen35_9b/.

The official shared-task README asks for full, merged model weights. Because this system is a composed inference-time method rather than a single merged policy checkpoint, the form submission should explicitly state that the submission consists of:

Qwen3.5-9B policy + trained Qwen3.5-9B PRM adapter + PRM-guided search code

Final Scores

Final standard Playpen score files are in eval/.

Search method	Clemscore	Statscore	File
best-of-N	40.85	71.88	eval/Qwen3.5-9B-9B-PRM-best_of_n.val.json
beam search	35.45	71.88	eval/Qwen3.5-9B-9B-PRM-beam.val.json
DVTS	35.48	71.88	eval/Qwen3.5-9B-9B-PRM-dvts.val.json
lookahead	28.21	71.88	eval/Qwen3.5-9B-9B-PRM-lookahead.val.json

The strongest current submission candidate is best-of-N.

Reproduction Sketch

Install and configure Playpen as described by the official repository. Then copy or adapt configs/model_registry.json into the Playpen checkout so that Qwen3.5-9B-local-4bit points to this repository's base_policy_qwen35_9b/ directory. Run the evaluation script directly from a terminal:

cd /path/to/playpen
PLAYPEN_ROOT=$PWD \
SUBMISSION_ROOT=/path/to/playpen-prm-9b \
CUDA_VISIBLE_DEVICES=0 \
bash /path/to/playpen-prm-9b/scripts/run_9b_prm_search_eval.sh

The script expects a Playpen checkout with the required validation data, clembench games, and model registry. It runs the four search methods sequentially and does not use cost-aware sharding or a worker pool. The data sharding used during the original experiment was only an acceleration mechanism.

Important Caveat

The base policy weights are fully packaged. The PRM checkpoint in prm_checkpoint/ is still an adapter checkpoint. If the organizers require a single monolithic PRM checkpoint, merge the adapter into a compatible Qwen3.5-9B sequence-classification PRM backbone before uploading, or ask the organizers whether PRM adapter + base model reference is acceptable for composed search submissions.