GenARM h0p01 c=0.175 Evaluation Code

This repository packages the runnable code for the h0p01 + c=0.175 GenARM evaluation on the 500-prompt non-overlap test set.

Experiment

Dataset: dataset/evaluation_prompts-full-728_without_train_overlap_500.json
Seed: 0
Base model: alpaca-7b-reproduced
Helpful ARM: arm_beta_0p5_masked_round4_extreme/final_checkpoint
Harmless ARM: arm_beta_0p01_masked_round4_extreme/final_checkpoint
Formula: alpha_help + 0.175 * alpha_harm = 1
Scheme A candidate set: A_t = {v | p_base(v) >= threshold}
Generation score inside A_t: S(v) = log p_base(v) + alpha_help * log p_help(v) + alpha_harm * log p_harm(v)
Sampling: softmax over S(v) restricted to A_t
Pairwise judge: local Meta-Llama-3-70B-Instruct
Human-like judge: single-model humanness judge

Required Paths

Set these environment variables on the target machine:

export BASE_MODEL=/path/to/alpaca-7b-reproduced
export HELPFUL_ADAPTER=/path/to/arm_beta_0p5_masked_round4_extreme/final_checkpoint
export HARMLESS_ADAPTER=/path/to/arm_beta_0p01_masked_round4_extreme/final_checkpoint
export JUDGE_MODEL=/path/to/Meta-Llama-3-70B-Instruct

The repo includes baseline/base_generation_seed0.json for pairwise evaluation. Override BASE_GENERATION=/path/to/base/generation.json only if you want to compare against a different base output.

Run One Point

bash scripts/run_h0p01_c0175_point.sh 0.7 scheme_a_threshold_0p0008

This computes alpha_harm = (1 - alpha_help) / 0.175, then runs generation, pairwise evaluation, humanness evaluation, and summary creation.

Submit the Full c=0.175 Set

The default full set is 11 points:

bash scripts/submit_pbs_h0p01_c0175.sh

The default methods are:

scheme_a_threshold_0p0008: alpha_help = 0, 0.1, ..., 1
scheme_a_threshold_0p0005: alpha_help = 0, 0.7, 0.8, 1
scheme_a_threshold_0p0003: alpha_help = 0, 0.7, 0.8, 1
scheme_a_threshold_0p001: alpha_help = 0, 0.7, 0.8, 1

Override with environment variables if needed:

METHODS="scheme_a_threshold_0p0008" ALPHA_HELPS="0 0.7 0.8 1" bash scripts/submit_pbs_h0p01_c0175.sh

Output

Results are written under:

outputs/h0p01_c0175_seed0/

Each point produces:

generation.json
pairwise judge JSON
humanness judge JSON
summary.json

The main summary metrics are:

pairwise.win_halfTie_helpfulness
pairwise.win_halfTie_harmlessness
humanness.avg_score_humanness

Notes

The packaged runner uses two GPUs for generation and sets the judge to auto device mapping across both GPUs by default. On a single machine without PBS, run scripts/run_h0p01_c0175_point.sh directly.