File size: 2,725 Bytes
33b38c9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | # GenARM h0p01 c=0.175 Evaluation Code
This repository packages the runnable code for the `h0p01 + c=0.175` GenARM evaluation on the 500-prompt non-overlap test set.
## Experiment
- Dataset: `dataset/evaluation_prompts-full-728_without_train_overlap_500.json`
- Seed: `0`
- Base model: `alpaca-7b-reproduced`
- Helpful ARM: `arm_beta_0p5_masked_round4_extreme/final_checkpoint`
- Harmless ARM: `arm_beta_0p01_masked_round4_extreme/final_checkpoint`
- Formula: `alpha_help + 0.175 * alpha_harm = 1`
- Scheme A candidate set: `A_t = {v | p_base(v) >= threshold}`
- Generation score inside `A_t`:
`S(v) = log p_base(v) + alpha_help * log p_help(v) + alpha_harm * log p_harm(v)`
- Sampling: softmax over `S(v)` restricted to `A_t`
- Pairwise judge: local `Meta-Llama-3-70B-Instruct`
- Human-like judge: single-model humanness judge
## Required Paths
Set these environment variables on the target machine:
```bash
export BASE_MODEL=/path/to/alpaca-7b-reproduced
export HELPFUL_ADAPTER=/path/to/arm_beta_0p5_masked_round4_extreme/final_checkpoint
export HARMLESS_ADAPTER=/path/to/arm_beta_0p01_masked_round4_extreme/final_checkpoint
export JUDGE_MODEL=/path/to/Meta-Llama-3-70B-Instruct
```
The repo includes `baseline/base_generation_seed0.json` for pairwise evaluation. Override `BASE_GENERATION=/path/to/base/generation.json` only if you want to compare against a different base output.
## Run One Point
```bash
bash scripts/run_h0p01_c0175_point.sh 0.7 scheme_a_threshold_0p0008
```
This computes `alpha_harm = (1 - alpha_help) / 0.175`, then runs generation, pairwise evaluation, humanness evaluation, and summary creation.
## Submit the Full c=0.175 Set
The default full set is 11 points:
```bash
bash scripts/submit_pbs_h0p01_c0175.sh
```
The default methods are:
- `scheme_a_threshold_0p0008`: `alpha_help = 0, 0.1, ..., 1`
- `scheme_a_threshold_0p0005`: `alpha_help = 0, 0.7, 0.8, 1`
- `scheme_a_threshold_0p0003`: `alpha_help = 0, 0.7, 0.8, 1`
- `scheme_a_threshold_0p001`: `alpha_help = 0, 0.7, 0.8, 1`
Override with environment variables if needed:
```bash
METHODS="scheme_a_threshold_0p0008" ALPHA_HELPS="0 0.7 0.8 1" bash scripts/submit_pbs_h0p01_c0175.sh
```
## Output
Results are written under:
```bash
outputs/h0p01_c0175_seed0/
```
Each point produces:
- `generation.json`
- pairwise judge JSON
- humanness judge JSON
- `summary.json`
The main summary metrics are:
- `pairwise.win_halfTie_helpfulness`
- `pairwise.win_halfTie_harmlessness`
- `humanness.avg_score_humanness`
## Notes
The packaged runner uses two GPUs for generation and sets the judge to auto device mapping across both GPUs by default. On a single machine without PBS, run `scripts/run_h0p01_c0175_point.sh` directly.
|