# GenARM h0p01 c=0.175 Evaluation Code

This repository packages the runnable code for the `h0p01 + c=0.175` GenARM evaluation on the 500-prompt non-overlap test set.

## Experiment

- Dataset: `dataset/evaluation_prompts-full-728_without_train_overlap_500.json`
- Seed: `0`
- Base model: `alpaca-7b-reproduced`
- Helpful ARM: `arm_beta_0p5_masked_round4_extreme/final_checkpoint`
- Harmless ARM: `arm_beta_0p01_masked_round4_extreme/final_checkpoint`
- Formula: `alpha_help + 0.175 * alpha_harm = 1`
- Scheme A candidate set: `A_t = {v | p_base(v) >= threshold}`
- Generation score inside `A_t`:
  `S(v) = log p_base(v) + alpha_help * log p_help(v) + alpha_harm * log p_harm(v)`
- Sampling: softmax over `S(v)` restricted to `A_t`
- Pairwise judge: local `Meta-Llama-3-70B-Instruct`
- Human-like judge: single-model humanness judge

## Required Paths

Set these environment variables on the target machine:

```bash
export BASE_MODEL=/path/to/alpaca-7b-reproduced
export HELPFUL_ADAPTER=/path/to/arm_beta_0p5_masked_round4_extreme/final_checkpoint
export HARMLESS_ADAPTER=/path/to/arm_beta_0p01_masked_round4_extreme/final_checkpoint
export JUDGE_MODEL=/path/to/Meta-Llama-3-70B-Instruct
```

The repo includes `baseline/base_generation_seed0.json` for pairwise evaluation. Override `BASE_GENERATION=/path/to/base/generation.json` only if you want to compare against a different base output.

## Run One Point

```bash
bash scripts/run_h0p01_c0175_point.sh 0.7 scheme_a_threshold_0p0008
```

This computes `alpha_harm = (1 - alpha_help) / 0.175`, then runs generation, pairwise evaluation, humanness evaluation, and summary creation.

## Submit the Full c=0.175 Set

The default full set is 11 points:

```bash
bash scripts/submit_pbs_h0p01_c0175.sh
```

The default methods are:

- `scheme_a_threshold_0p0008`: `alpha_help = 0, 0.1, ..., 1`
- `scheme_a_threshold_0p0005`: `alpha_help = 0, 0.7, 0.8, 1`
- `scheme_a_threshold_0p0003`: `alpha_help = 0, 0.7, 0.8, 1`
- `scheme_a_threshold_0p001`: `alpha_help = 0, 0.7, 0.8, 1`

Override with environment variables if needed:

```bash
METHODS="scheme_a_threshold_0p0008" ALPHA_HELPS="0 0.7 0.8 1" bash scripts/submit_pbs_h0p01_c0175.sh
```

## Output

Results are written under:

```bash
outputs/h0p01_c0175_seed0/
```

Each point produces:

- `generation.json`
- pairwise judge JSON
- humanness judge JSON
- `summary.json`

The main summary metrics are:

- `pairwise.win_halfTie_helpfulness`
- `pairwise.win_halfTie_harmlessness`
- `humanness.avg_score_humanness`

## Notes

The packaged runner uses two GPUs for generation and sets the judge to auto device mapping across both GPUs by default. On a single machine without PBS, run `scripts/run_h0p01_c0175_point.sh` directly.