eval/README.md

# TOSC Benchmark — Evaluation

Evaluation code for the **Triadic Object-State Consistency (TOSC)** benchmark.

TOSC probes whether a vision-language model keeps its object descriptions
consistent across three versions of the same scene:

| State | Folder | Filename pattern | Meaning |
|-------|--------|------------------|---------|
| **origin**   | `benchmark/original/images/` | `{N}.jpg`                              | the unedited image |
| **removed**  | `benchmark/repair/images/`   | `{N}_masked_{orig}_repair.png`         | the original object inpainted out |
| **replaced** | `benchmark/insert/images/`   | `{N}_masked_{orig}_insert_{repl}.png`  | the original object swapped for a different one |

Each index `N` forms one triplet (5,969 in total). The model is asked the same
question for every image — *"Describe this image in detail."* — and we check
which objects its caption mentions.

## Metrics

For a triplet with original object `o` and replacement object `r`, let:

- `m_orig_o` = `o` mentioned in the **origin** caption
- `m_rem_o`  = `o` mentioned in the **removed** caption
- `m_rep_o`  = `o` mentioned in the **replaced** caption
- `m_rep_r`  = `r` mentioned in the **replaced** caption

| Metric | Formula | Higher = |
|--------|---------|----------|
| **OPA** (object present) | `m_orig_o` | sees the object when it is there |
| **RCA** (removal consistency) | `1 - m_rem_o` | stops mentioning it once removed |
| **RUA** (replacement utility) | `m_rep_r · (1 - m_rep_o)` | sees the new object, not the old |
| **TOSC** (overall) | `m_orig_o · (1 - m_rem_o) · m_rep_r · (1 - m_rep_o)` | all of the above for one triplet |
| OldPersist_rem | `m_rem_o` | (lower better) hallucinated old object after removal |
| OldPersist_rep | `m_rep_o` | (lower better) hallucinated old object after replacement |
| RepFail | `1 - m_rep_r` | (lower better) missed the replacement object |
| CC | `(1 - m_rep_o) · (1 - m_rep_r)` | (lower better) saw neither object in the replaced image |
| MixConf | `m_rep_o · m_rep_r` | (lower better) saw both old and new objects at once |

Object mentions are detected with a local COCO synonym table plus
lemmatization (`eval_masked_obj_generative.py` supplies `OBJECT_SYNONYMS`); no
external API or LLM judge is used.

## Files

All evaluation code lives in `eval/`, except the vendored inference package
which sits at the repo root so it imports as `llava`:

| Path | Role |
|------|------|
| `eval/build_index.py` | Scans the image folders → writes `benchmark/TOSC_dataset.jsonl` + `benchmark/insertions.jsonl`. |
| `eval/run_eval.sh` | End-to-end runner: caption generation → merge → scoring. |
| `eval/eval_tosc.py` | Scoring engine (consumes captions + the two jsonl files). |
| `eval/eval_masked_obj_generative.py` | COCO object synonym table used by the scorer. |
| `llava/` | Vendored inference package (`llava.eval.model_vqa` + the LLaVA-1.5 model loader and Qwen helpers). |
| `requirements.txt` | Runtime dependencies. |

`TOSC_dataset.jsonl` / `insertions.jsonl` are committed for convenience but are
fully regenerable from the images with `build_index.py`.

## Requirements

This repo is **self-contained** — caption generation uses the vendored `llava`
package at the repo root (`llava/eval/model_vqa.py`); nothing outside the repo is
needed at runtime.

```bash
pip install -r requirements.txt
```

This covers both the LLaVA-1.5 path (custom loader for the `non_lora_trainables`
LoRA format) and the Qwen2-VL / Qwen2.5-VL path (`transformers` + `peft`). The
Qwen path requests `flash_attention_2`, so install `flash-attn` for those models
(or change `attn_implementation` to `"sdpa"` in
`llava/eval/utils/hf_utils.py`). `nltk` is optional for the scorer — it falls
back to a simple singularizer if WordNet is unavailable.

## Usage

```bash
# 1. Build the index files (run once; regenerable any time)
python eval/build_index.py

# 2. Run the full pipeline for one model.
#    Edit the model preset at the top of run_eval.sh, or override via env vars.
GPU_LIST=0,1,2,3 bash eval/run_eval.sh
```

Results are written to `${TOSC_RESULTS:-/home/yilin/tmp/tosc_results}` (outside
the repo by default): merged captions in `answers/<OUTPUT_NAME>.jsonl`, full
per-triplet results in `<OUTPUT_NAME>_eval.json`, and aggregate metrics in
`<OUTPUT_NAME>_eval_summary.json`.

### Shipped model presets

`run_eval.sh` carries presets for the four LoRA adapters in `lora/`:

| OUTPUT_NAME | Base model | LoRA |
|-------------|-----------|------|
| `LLaVA_v1_5_7b-TSA-DPO`  | `liuhaotian/llava-v1.5-7b`    | `lora/LLaVA_v1_5_7b-TSA-DPO` |
| `LLaVA_v1_5_13b-TSA-DPO` | `liuhaotian/llava-v1.5-13b`   | `lora/LLaVA_v1_5_13b-TSA-DPO` |
| `Qwen2_VL_7B-TSA-DPO`    | `Qwen/Qwen2-VL-7B-Instruct`   | `lora/Qwen2_VL_7B-TSA-DPO` |
| `Qwen2_5_VL_7B-TSA-DPO`  | `Qwen/Qwen2.5-VL-7B-Instruct` | `lora/Qwen2_5_VL_7B-TSA-DPO` |

`model_vqa.py` auto-selects the inference path: names containing `llava` + `1.5`
use the LLaVA loader, everything else uses the HuggingFace/Qwen path. Set
`LORA_NAME=""` to evaluate a base model without the adapter.