experiment/knowledge_editing/README.md

# Knowledge Editing Baselines

Compare our OVERTONE LoRA method against standard knowledge model editing
(KME) methods from EasyEdit on the same hallucination suppression task.

## Problem Formulation

### The Edit (Captioning)

Each edit instance is a concrete captioning correction:

```
Image_42 + "Describe this image."
  old:  "A bathroom with a toilet, sink, and mirror"    ← hallucinated
  new:  "A bathroom with a sink and mirror"              ← toilet removed
```

The target is the **model's own caption with toilet mentions surgically removed**.
This is a minimal edit — the model's style, vocabulary, and all correct content
are preserved.  Only the hallucinated part changes.

### Why Captioning (not VQA)

- Captioning is where the hallucination manifests in practice
- Each edit is a concrete (input, old_output → new_output) triple — exactly
  what KME methods are designed for
- The target preserves the model's own distribution (minimal intervention)
- All methods are evaluated on the same metric: toilet mention rate in captions

### Mapping to Standard KME

| KME Concept | Standard (text-only) | Our Setting |
|-------------|---------------------|-------------|
| Subject `s` | "Eiffel Tower" | bathroom-without-toilet image |
| Relation `r` | "located in" | "Describe this image." |
| Old object `o` | "Paris" | "...a toilet, sink, and mirror" |
| New object `o*` | "London" | "...a sink and mirror" |
| Rephrase (text) | "Where is the Eiffel Tower?" | "What objects are in this picture?" |
| Rephrase (visual) | — | Different bathroom-no-toilet image |
| Locality (text) | Unrelated facts | "What is the capital of France?" |
| Locality (vision) | — | Bathroom WITH toilet (should still mention toilet) |

### Key Differences from Standard KME

1. **Distributional subject** — not one entity but a distribution of images
2. **Suppression** — target is "same minus toilet", not a fixed replacement
3. **Visual conditioning** — same token must be suppressed or preserved
   depending on what the image shows
4. **Per-instance targets** — each image has a different correct caption

## Pipeline

```
Step 1: Build edit set (no GPU)    →  edit_set.json (structure, no targets)
Step 2: Generate targets (GPU)     →  edit_set.json (filled with captions)
Step 3: Run EasyEdit methods       →  edited models
Step 4: Evaluate all               →  comparison table
```

### Step 1+2: Build Edit Set

```bash
# Structure only (fill targets on server later)
python -m experiment.knowledge_editing.build_edit_set \
    --csv CC3M-Dataset/bathroom_filter/bathroom_toilet_labels.csv \
    --image_dir CC3M-Dataset/cc3m_images/train \
    --output experiment/knowledge_editing/edit_set.json \
    --max_locality_per_category 50

# Generate targets on server (needs GPU)
python -m experiment.knowledge_editing.build_edit_set \
    --fill_targets experiment/knowledge_editing/edit_set.json \
    --model llava-hf/llava-1.5-7b-hf

# Or do both in one go
python -m experiment.knowledge_editing.build_edit_set \
    --csv ... --image_dir ... --output ... \
    --generate_targets --model llava-hf/llava-1.5-7b-hf
```

The target generation step:
1. Runs original LLaVA on each bathroom-no-toilet image
2. Gets the hallucinated caption ("A bathroom with a toilet and sink")
3. Removes toilet mentions ("A bathroom with a sink")
4. Checks if the result is still a valid caption (min 4 words)

### Step 3: Run Baselines

```bash
PYTHONPATH="${PYTHONPATH}:./EasyEdit" \
python -m experiment.knowledge_editing.run_baselines \
    --edit_set experiment/knowledge_editing/edit_set.json \
    --methods wise grace lora \
    --n_edits 20 \
    --output_dir step4_ke_outputs
```

### Step 4: Compare with Our Method

```bash
# Run OVERTONE LoRA
python -m experiment.training.finetune_lora_v3 --config experiment/lora_v3_config.json

# Evaluate (same pipeline for all methods)
python -m experiment.evaluation.validate \
    --model_type lora \
    --model_dir <path> \
    --inference_backend transformers \
    --mention_method keyword
```

## File Reference

| File | Purpose |
|------|---------|
| `build_edit_set.py` | CSV → edit_set.json + target generation |
| `llava15_compat.py` | LLaVA-1.5 processor wrapper for EasyEdit |
| `run_baselines.py` | Run EasyEdit methods + save + evaluate |
| `hparams/*.yaml` | EasyEdit configs per method |

## Edit Set Format

```json
{
    "edit_instances": {
        "train": [
            {
                "image_id": "12345",
                "image_path": "...",
                "original_caption": "A bathroom with a toilet, sink, and mirror",
                "target": "A bathroom with a sink and mirror",
                "had_toilet": true,
                "is_usable": true
            }
        ]
    },
    "locality_instances": {
        "bathroom_with_toilet": [
            {
                "image_id": "67890",
                "original_caption": "A bathroom with a white toilet and bathtub"
            }
        ]
    }
}
```