| # GenARM h0p01 c=0.175 Evaluation Code |
|
|
| This repository packages the runnable code for the `h0p01 + c=0.175` GenARM evaluation on the 500-prompt non-overlap test set. |
|
|
| ## Experiment |
|
|
| - Dataset: `dataset/evaluation_prompts-full-728_without_train_overlap_500.json` |
| - Seed: `0` |
| - Base model: `alpaca-7b-reproduced` |
| - Helpful ARM: `arm_beta_0p5_masked_round4_extreme/final_checkpoint` |
| - Harmless ARM: `arm_beta_0p01_masked_round4_extreme/final_checkpoint` |
| - Formula: `alpha_help + 0.175 * alpha_harm = 1` |
| - Scheme A candidate set: `A_t = {v | p_base(v) >= threshold}` |
| - Generation score inside `A_t`: |
| `S(v) = log p_base(v) + alpha_help * log p_help(v) + alpha_harm * log p_harm(v)` |
| - Sampling: softmax over `S(v)` restricted to `A_t` |
| - Pairwise judge: local `Meta-Llama-3-70B-Instruct` |
| - Human-like judge: single-model humanness judge |
|
|
| ## Required Paths |
|
|
| Set these environment variables on the target machine: |
|
|
| ```bash |
| export BASE_MODEL=/path/to/alpaca-7b-reproduced |
| export HELPFUL_ADAPTER=/path/to/arm_beta_0p5_masked_round4_extreme/final_checkpoint |
| export HARMLESS_ADAPTER=/path/to/arm_beta_0p01_masked_round4_extreme/final_checkpoint |
| export JUDGE_MODEL=/path/to/Meta-Llama-3-70B-Instruct |
| ``` |
|
|
| The repo includes `baseline/base_generation_seed0.json` for pairwise evaluation. Override `BASE_GENERATION=/path/to/base/generation.json` only if you want to compare against a different base output. |
|
|
| ## Run One Point |
|
|
| ```bash |
| bash scripts/run_h0p01_c0175_point.sh 0.7 scheme_a_threshold_0p0008 |
| ``` |
|
|
| This computes `alpha_harm = (1 - alpha_help) / 0.175`, then runs generation, pairwise evaluation, humanness evaluation, and summary creation. |
|
|
| ## Submit the Full c=0.175 Set |
|
|
| The default full set is 11 points: |
|
|
| ```bash |
| bash scripts/submit_pbs_h0p01_c0175.sh |
| ``` |
|
|
| The default methods are: |
|
|
| - `scheme_a_threshold_0p0008`: `alpha_help = 0, 0.1, ..., 1` |
| - `scheme_a_threshold_0p0005`: `alpha_help = 0, 0.7, 0.8, 1` |
| - `scheme_a_threshold_0p0003`: `alpha_help = 0, 0.7, 0.8, 1` |
| - `scheme_a_threshold_0p001`: `alpha_help = 0, 0.7, 0.8, 1` |
|
|
| Override with environment variables if needed: |
|
|
| ```bash |
| METHODS="scheme_a_threshold_0p0008" ALPHA_HELPS="0 0.7 0.8 1" bash scripts/submit_pbs_h0p01_c0175.sh |
| ``` |
|
|
| ## Output |
|
|
| Results are written under: |
|
|
| ```bash |
| outputs/h0p01_c0175_seed0/ |
| ``` |
|
|
| Each point produces: |
|
|
| - `generation.json` |
| - pairwise judge JSON |
| - humanness judge JSON |
| - `summary.json` |
|
|
| The main summary metrics are: |
|
|
| - `pairwise.win_halfTie_helpfulness` |
| - `pairwise.win_halfTie_harmlessness` |
| - `humanness.avg_score_humanness` |
|
|
| ## Notes |
|
|
| The packaged runner uses two GPUs for generation and sets the judge to auto device mapping across both GPUs by default. On a single machine without PBS, run `scripts/run_h0p01_c0175_point.sh` directly. |
|
|
