eval/README.md

TOSC Benchmark — Evaluation

Evaluation code for the Triadic Object-State Consistency (TOSC) benchmark.

TOSC probes whether a vision-language model keeps its object descriptions consistent across three versions of the same scene:

State	Folder	Filename pattern	Meaning
origin	benchmark/original/images/	{N}.jpg	the unedited image
removed	benchmark/repair/images/	{N}_masked_{orig}_repair.png	the original object inpainted out
replaced	benchmark/insert/images/	{N}_masked_{orig}_insert_{repl}.png	the original object swapped for a different one

Each index N forms one triplet (5,969 in total). The model is asked the same question for every image — "Describe this image in detail." — and we check which objects its caption mentions.

Metrics

For a triplet with original object o and replacement object r, let:

• m_orig_o = o mentioned in the origin caption
• m_rem_o = o mentioned in the removed caption
• m_rep_o = o mentioned in the replaced caption
• m_rep_r = r mentioned in the replaced caption

Metric	Formula	Higher =
OPA (object present)	m_orig_o	sees the object when it is there
RCA (removal consistency)	1 - m_rem_o	stops mentioning it once removed
RUA (replacement utility)	m_rep_r · (1 - m_rep_o)	sees the new object, not the old
TOSC (overall)	m_orig_o · (1 - m_rem_o) · m_rep_r · (1 - m_rep_o)	all of the above for one triplet
OldPersist_rem	m_rem_o	(lower better) hallucinated old object after removal
OldPersist_rep	m_rep_o	(lower better) hallucinated old object after replacement
RepFail	1 - m_rep_r	(lower better) missed the replacement object
CC	(1 - m_rep_o) · (1 - m_rep_r)	(lower better) saw neither object in the replaced image
MixConf	m_rep_o · m_rep_r	(lower better) saw both old and new objects at once

Object mentions are detected with a local COCO synonym table plus lemmatization (eval_masked_obj_generative.py supplies OBJECT_SYNONYMS); no external API or LLM judge is used.

Files

All evaluation code lives in eval/, except the vendored inference package which sits at the repo root so it imports as llava:

Path	Role
eval/build_index.py	Scans the image folders → writes benchmark/TOSC_dataset.jsonl + benchmark/insertions.jsonl.
eval/run_eval.sh	End-to-end runner: caption generation → merge → scoring.
eval/eval_tosc.py	Scoring engine (consumes captions + the two jsonl files).
eval/eval_masked_obj_generative.py	COCO object synonym table used by the scorer.
llava/	Vendored inference package (llava.eval.model_vqa + the LLaVA-1.5 model loader and Qwen helpers).
requirements.txt	Runtime dependencies.

TOSC_dataset.jsonl / insertions.jsonl are committed for convenience but are fully regenerable from the images with build_index.py.

Requirements

This repo is self-contained — caption generation uses the vendored llava package at the repo root (llava/eval/model_vqa.py); nothing outside the repo is needed at runtime.

pip install -r requirements.txt

This covers both the LLaVA-1.5 path (custom loader for the non_lora_trainables LoRA format) and the Qwen2-VL / Qwen2.5-VL path (transformers + peft). The Qwen path requests flash_attention_2, so install flash-attn for those models (or change attn_implementation to "sdpa" in llava/eval/utils/hf_utils.py). nltk is optional for the scorer — it falls back to a simple singularizer if WordNet is unavailable.

Usage

# 1. Build the index files (run once; regenerable any time)
python eval/build_index.py

# 2. Run the full pipeline for one model.
#    Edit the model preset at the top of run_eval.sh, or override via env vars.
GPU_LIST=0,1,2,3 bash eval/run_eval.sh

Results are written to ${TOSC_RESULTS:-/home/yilin/tmp/tosc_results} (outside the repo by default): merged captions in answers/<OUTPUT_NAME>.jsonl, full per-triplet results in <OUTPUT_NAME>_eval.json, and aggregate metrics in <OUTPUT_NAME>_eval_summary.json.

Shipped model presets

run_eval.sh carries presets for the four LoRA adapters in lora/:

OUTPUT_NAME	Base model	LoRA
LLaVA_v1_5_7b-TSA-DPO	liuhaotian/llava-v1.5-7b	lora/LLaVA_v1_5_7b-TSA-DPO
LLaVA_v1_5_13b-TSA-DPO	liuhaotian/llava-v1.5-13b	lora/LLaVA_v1_5_13b-TSA-DPO
Qwen2_VL_7B-TSA-DPO	Qwen/Qwen2-VL-7B-Instruct	lora/Qwen2_VL_7B-TSA-DPO
Qwen2_5_VL_7B-TSA-DPO	Qwen/Qwen2.5-VL-7B-Instruct	lora/Qwen2_5_VL_7B-TSA-DPO

model_vqa.py auto-selects the inference path: names containing llava + 1.5 use the LLaVA loader, everything else uses the HuggingFace/Qwen path. Set LORA_NAME="" to evaluate a base model without the adapter.