| # TOSC — Triadic Object-State Consistency |
|
|
| Benchmark, evaluation code, and trained LoRA models for the **Triadic |
| Object-State Consistency (TOSC)** benchmark. |
|
|
| TOSC tests whether a vision-language model keeps its object descriptions |
| consistent across three versions of the same scene: the **original** image, the |
| same image with one object **removed** (inpainted out), and the same image with |
| that object **replaced** by a different one. A consistent model should mention |
| the object when present, stop mentioning it once removed, and report the new |
| object after replacement. |
|
|
| This repository is **self-contained**: the benchmark images, the evaluation |
| code, and the model adapters are all here, and the evaluation pipeline runs with |
| no external code dependency. |
|
|
| ## Repository layout |
|
|
| ``` |
| tosc-model-weights/ |
| ├── benchmark/ # 5,969 triplets (origin / removed / replaced) |
| │ ├── original/images/ # {N}.jpg |
| │ ├── repair/images/ # {N}_masked_{obj}_repair.png (removed) |
| │ ├── insert/images/ # {N}_masked_{obj}_insert_{new}.png (replaced) |
| │ ├── TOSC_dataset.jsonl # generated index (regenerable) |
| │ └── insertions.jsonl # generated object metadata (regenerable) |
| ├── eval/ # evaluation code (see eval/README.md for details) |
| │ ├── build_index.py # image folders -> the two .jsonl files |
| │ ├── run_eval.sh # end-to-end runner: caption -> merge -> score |
| │ ├── eval_tosc.py # scoring engine |
| │ └── eval_masked_obj_generative.py |
| ├── llava/ # vendored inference package (imports as `llava`) |
| ├── lora/ # 4 trained LoRA adapters (TSA-DPO) |
| └── requirements.txt |
| ``` |
|
|
| ### Models |
|
|
| | Adapter (`lora/…`) | Base model | |
| |--------------------|-----------| |
| | `LLaVA_v1_5_7b-TSA-DPO` | `liuhaotian/llava-v1.5-7b` | |
| | `LLaVA_v1_5_13b-TSA-DPO` | `liuhaotian/llava-v1.5-13b` | |
| | `Qwen2_VL_7B-TSA-DPO` | `Qwen/Qwen2-VL-7B-Instruct` | |
| | `Qwen2_5_VL_7B-TSA-DPO` | `Qwen/Qwen2.5-VL-7B-Instruct` | |
|
|
| ## Installation |
|
|
| A CUDA GPU is required for caption generation. Python 3.10+ is recommended. |
|
|
| ```bash |
| # 1. Get the repo (images + adapters are stored with Git LFS) |
| git lfs install |
| git clone https://huggingface.co/julyanghar/tosc-model-weights |
| cd tosc-model-weights |
| |
| # 2. Create an environment and install dependencies |
| conda create -n tosc python=3.10 -y && conda activate tosc # or use a venv |
| pip install -r requirements.txt |
| |
| # 3. (Qwen models only) install flash-attn for the Qwen2-VL / Qwen2.5-VL path, |
| # or change attn_implementation to "sdpa" in llava/eval/utils/hf_utils.py |
| pip install flash-attn --no-build-isolation |
| ``` |
|
|
| Base models (`liuhaotian/llava-v1.5-*`, `Qwen/Qwen2*-VL-*-Instruct`) and the |
| CLIP vision tower are downloaded automatically from the Hugging Face Hub on first |
| use; the LoRA adapters are loaded from `lora/` in this repo. |
|
|
| ## How to run |
|
|
| ```bash |
| # 1. Build the index files once (regenerable from the images at any time) |
| python eval/build_index.py |
| |
| # 2. Evaluate a model. Pick one of the four presets in eval/run_eval.sh |
| # (uncomment it), or select it via environment variables. Example: Qwen2.5-VL |
| # on 4 GPUs. |
| GPU_LIST=0,1,2,3 \ |
| OUTPUT_NAME=Qwen2_5_VL_7B-TSA-DPO \ |
| MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct \ |
| LORA_NAME=$PWD/lora/Qwen2_5_VL_7B-TSA-DPO \ |
| bash eval/run_eval.sh |
| ``` |
|
|
| `run_eval.sh` runs three steps: (1) generate a caption for every benchmark image |
| (multi-GPU, sharded by `GPU_LIST`), (2) merge the shards, (3) score with |
| `eval/eval_tosc.py`. The inference path is selected automatically — model names |
| containing `llava` + `1.5` use the LLaVA-1.5 loader, everything else uses the |
| Qwen path. Set `LORA_NAME=""` to evaluate a base model without the adapter. |
|
|
| ### Single-GPU / quick smoke test |
|
|
| ```bash |
| GPU_LIST=0 bash eval/run_eval.sh # single GPU (slower) |
| ``` |
|
|
| To sanity-check the wiring on a few images, point `QUESTION_FILE` at a trimmed |
| copy of `benchmark/TOSC_dataset.jsonl`. |
|
|
| ### Outputs |
|
|
| Results go to `${TOSC_RESULTS:-/home/yilin/tmp/tosc_results}` (outside the repo |
| by default; override with `OUTPUT_DIR`): |
|
|
| - `answers/<OUTPUT_NAME>.jsonl` — merged captions |
| - `<OUTPUT_NAME>_eval.json` — full per-triplet results |
| - `<OUTPUT_NAME>_eval_summary.json` — aggregate metrics |
|
|
| ### Configuration (environment variables) |
|
|
| | Variable | Default | Meaning | |
| |----------|---------|---------| |
| | `GPU_LIST` | `0` | comma-separated GPU ids, e.g. `0,1,2,3` | |
| | `MODEL_NAME` / `LORA_NAME` / `OUTPUT_NAME` | Qwen2.5 preset | model selection | |
| | `MAX_NEW_TOKENS` | `512` | caption length | |
| | `TEMPERATURE` | `0` | `0` = greedy / deterministic | |
| | `SEED` | `42` | random seed | |
| | `OUTPUT_DIR` | `/home/yilin/tmp/tosc_results` | where results are written | |
|
|
| ## Metrics |
|
|
| The scorer reports the headline **TOSC** score plus **OPA** (sees the object), |
| **RCA** (drops it after removal), **RUA** (sees the replacement, not the old |
| object), and fine-grained failure rates (OldPersist / RepFail / CC / MixConf). |
| Object mentions are detected with a local COCO synonym table and lemmatization — |
| no external API or LLM judge. See [eval/README.md](eval/README.md) for the exact |
| formulas and per-state definitions. |
|
|
