File size: 2,653 Bytes
eaae571 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | # Phase 7 Experiment Guide
This guide explains the current comparison workflow for before-vs-after
mitigation experiments.
## Purpose
Phase 7 is where the team compares:
- a baseline run
- a defended run on the same slice
- the metric differences between them
The goal is to produce evidence for what improved, what remained weak, and what
still needs discussion in the final report.
## Main Comparison Pairs
### Larger active-corpus pairs
- baseline: `configs/hackaprompt_override_25.yaml`
- defended: `configs/hackaprompt_override_25_mitigated.yaml`
- baseline: `configs/tensortrust_hijacking_25.yaml`
- defended: `configs/tensortrust_hijacking_25_mitigated.yaml`
### Mock validation pairs
- `configs/hackaprompt_override_25_mock.yaml`
- `configs/hackaprompt_override_25_mock_mitigated.yaml`
- `configs/tensortrust_hijacking_25_mock.yaml`
- `configs/tensortrust_hijacking_25_mock_mitigated.yaml`
### Focused validation pairs
- `configs/hackaprompt_direct_10.yaml`
- `configs/hackaprompt_direct_10_mitigated.yaml`
- `configs/agentdojo_tool_10.yaml`
- `configs/agentdojo_tool_10_mitigated.yaml`
- `configs/hackaprompt_adaptive_1_mitigated.yaml`
## How To Run
1. Run the baseline config.
2. Run the matching defended config.
3. Compare the two `summary.json` files.
For exact hosted-run commands and a fill-in template, use:
- `docs/GROQ_RUNBOOK.md`
- `docs/RESULTS_TEMPLATE.md`
## How To Compare
Use:
```bash
python scripts/compare_runs.py --baseline results/<baseline_run_id>/summary.json --defended results/<defended_run_id>/summary.json
```
Optional saved outputs:
```bash
python scripts/compare_runs.py --baseline results/<baseline_run_id>/summary.json --defended results/<defended_run_id>/summary.json --output results/comparisons/run_comparison.json --markdown-output results/comparisons/run_comparison.md
```
## Output Coverage
The comparison script reports:
- overall metric deltas
- by-category deltas
- by-surface deltas
- by-source deltas
- mitigation activity counts
## What To Report
For each pair, record:
- attack success rate before and after mitigation
- successful attacks before and after mitigation
- prompt leakage changes
- refusal changes
- manual-review changes
- mitigation activity such as blocked, transformed, sanitized, and response-filtered cases
- category-level differences
## Current Position
The Phase 7 workflow is in strong shape:
- documented
- runnable with both mock and Groq
- able to generate report-ready comparison outputs
- already validated on real HackAPrompt and TensorTrust comparison pairs
Interpretation still matters, because not every benchmark slice improves equally.
|