File size: 3,902 Bytes
a2ffd07 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | # Evaluation Pipeline
Measures how well a knowledge-editing method suppresses toilet hallucination in LLaVA while preserving other capabilities.
## Quick Start
```bash
# LoRA adapter
python -m experiment.evaluation.validate \
--model_type lora \
--model_dir step3_lora_outputs/run_YYYYMMDD/lora_adapter
# Merged model
python -m experiment.evaluation.validate \
--model_type merged \
--model_dir step3_lora_outputs/run_YYYYMMDD/merged_model
# Delta-W checkpoint (transformers backend only)
python -m experiment.evaluation.validate \
--model_type delta_w \
--checkpoint step3_outputs/final_model.pt \
--inference_backend transformers
```
## Image Categories
| Category | bathroom | toilet | Role |
|----------|----------|--------|------|
| `bathroom_no_toilet` | 1 | 0 | **Edit target** — hallucination should be suppressed |
| `bathroom_with_toilet` | 1 | 1 | **Preserved** — should still mention toilet |
| `non_bathroom_with_toilet` | 0 | 1 | **Preserved** — toilet in other contexts must survive |
| `unrelated` | 0 | 0 | **Preserved** — general captioning must be unaffected |
## Metrics
### Per-Model
| Metric | Description |
|--------|-------------|
| `mention_rate_keyword` | Fraction of outputs containing an un-negated toilet synonym (fast regex). |
| `mention_rate_llm` | Fraction where VLM judge says toilet is present (more accurate, expensive). |
| `avg_clip_score` | Mean image–text cosine similarity (SigLIP). Only on "Describe this image." prompts. |
| `avg_caption_quality` | Mean caption quality score (1–5) from VLM judge. |
| `quality_gated_mention_rate` | Mention rate filtered to outputs with caption_quality >= 3. |
### KME Metrics
| Metric | Description | Category |
|--------|-------------|----------|
| **Efficacy** | `1 - mention_rate` on edit target. Did the edit work? | `bathroom_no_toilet` |
| **Generality** | Efficacy split by seen vs. unseen prompts. Does suppression generalize? | `bathroom_no_toilet` |
| **Locality** | Text similarity (exact match, ROUGE-L, BERTScore) between original and finetuned on preserved images. | all preserved categories |
| **Consistency** | Locality pooled across all preserved categories. | all preserved |
**Generality prompts:**
- Seen (train): `"Describe this image."`, `"Is there a toilet in this image?"`
- Unseen: `"What objects are in this picture?"`, `"Tell me what you see."`, `"What is in this bathroom?"`
## Files
| File | Purpose |
|------|---------|
| `validate.py` | Entry point, CLI, orchestration |
| `inference.py` | Generates outputs for both models (HuggingFace or vLLM) |
| `metrics.py` | `KeywordMentionDetector`, `CLIPScorer`, `LocalLLMJudge`, `TextSimilarityScorer` |
| `metric.py` | `evaluate_collected_outputs()`, `compute_kme_metrics()` |
| `summary.py` | Formats results to stdout |
## Output
Results saved to `--output_dir/<run_name>/`:
- `validation_results.json` — aggregate metrics + config
- `validation_details.json` — per-sample detail records
## Key CLI Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `--model_type` | *(required)* | `lora`, `merged`, or `delta_w` |
| `--base_model_name` | `llava-hf/llava-1.5-7b-hf` | Base model ID |
| `--model_dir` | — | LoRA adapter or merged model path |
| `--checkpoint` | — | `.pt` state dict (delta_w only) |
| `--inference_backend` | `vllm` | `transformers` or `vllm` |
| `--num_per_category` | `50` | Max images per category |
| `--mention_method` | `both` | `keyword`, `llm`, or `both` |
| `--output_dir` | `./step4_v2_outputs` | Root output directory |
## Optional Dependencies
| Package | Unlocks |
|---------|---------|
| `rouge-score` | ROUGE-L locality/consistency |
| `bert-score` | BERTScore locality/consistency |
| `peft` | LoRA model loading |
| `vllm` | Fast batched inference |
Missing packages degrade gracefully (metrics show `N/A`).
|