Tri-PvP

Tri-PvP is a tri-modal conflict benchmark for measuring modality bias in omni-modal LLMs. Each sample gives the model an image, an audio clip, and a text passage that point to three different answers, then asks an open-ended question. The answer the model commits to shows which modality it trusts.

Image and audio each come in two evidence forms:

• Perceptual: direct sensory evidence, such as a natural photo or a real recording.
• Propositional: a declarative claim, such as a rendered word card or a TTS utterance saying "This is a ."

Text is always propositional.

The benchmark covers four domains, each released as a separate repository:

Dataset Structure

Every domain shares the same schema and has one test split. One row holds one label triple together with both forms of its image and audio (image_perceptual, image_propositional, audio_perceptual, audio_propositional). You pick the image and audio columns to build any of the four evidence conditions, so the same triple is reused across conditions as a matched counterfactual.

Across the four domains this gives 8,000 samples.

Fields

The three labels in a row are always mutually distinct. The perceptual and propositional versions of a modality always carry the same label, so switching evidence form never changes the conflict.

Evidence Conditions

Label-modality combinations are balanced within each domain, so no modality is more reliable than another by construction.

Usage

from datasets import load_dataset

domain = "animal"  # animal | emotion | environment | music
ds = load_dataset(f"Tri-PvP/{domain}", split="test")

CONDITIONS = {
    "perc_img__perc_aud": ("image_perceptual",    "audio_perceptual"),
    "perc_img__prop_aud": ("image_perceptual",    "audio_propositional"),
    "prop_img__perc_aud": ("image_propositional", "audio_perceptual"),
    "prop_img__prop_aud": ("image_propositional", "audio_propositional"),
}

img_col, aud_col = CONDITIONS["perc_img__prop_aud"]
row = ds[0]
image = row[img_col]                   # PIL.Image
audio = row[aud_col]                   # {"array": np.ndarray, "sampling_rate": int, ...}
text = row["text_propositional"]
question = row["question"]

Unless specified, the paper feeds inputs in the order image, audio, text, question. Score free-form responses against image_label, audio_label, and text_label with the LLM judge in the GitHub repo. It assigns one of eight labels: BIAS_IMAGE, BIAS_AUDIO, BIAS_TEXT, BIAS_IMAGE_AUDIO, BIAS_IMAGE_TEXT, BIAS_AUDIO_TEXT, HALLUCINATION, or NO_BIAS.

Source Licenses

License

The benchmark compilation and the generated data are released under CC BY-NC 4.0. That covers the text, the synthesized speech, the word-card images, and the metadata. Perceptual source files keep their original licenses listed above. ImageNet and ESC-50 restrict use to non-commercial purposes, so Tri-PvP is for non-commercial research and educational use only.

Citation

@misc{piao2026tripvpexposingmodalitybias,
      title={Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts}, 
      author={Yen-Ting Piao and Shu-Yun Chen and Chin-Hui Chu and Chun-Wei Chen and Shih-Yun Shan Kuan and Hung-yi Lee and Yun-Nung Chen},
      year={2026},
      eprint={2609.06011},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.06011}, 
}