data/benchmarks/VideoHallucer/README.md

---
license: mit
task_categories:
- question-answering
language:
- en
size_categories:
- 1K<n<10K
configs:
  - config_name: external_factual
    data_files:
      - split: test
        path: external_factual/external_factual.json
  - config_name: external_nonfactual
    data_files:
      - split: test
        path: external_nonfactual/external_nonfactual.json
  - config_name: fact_detect
    data_files:
      - split: test
        path:
          - fact_detect/fact_detect.json
          - fact_detect/fact_detect_yn.json
  - config_name: object_relation
    data_files:
      - split: test
        path: object_relation/object_relation.json
  - config_name: semantic_detail
    data_files:
      - split: test
        path: semantic_detail/semantic_detail.json
  - config_name: temporal
    data_files:
      - split: test
        path: temporal/temporal.json
---
# VideoHallucer

**Paper:** https://huggingface.co/papers/2406.16338

## Dataset Description

- **Repository:** [VideoHallucer](https://github.com/patrick-tssn/VideoHallucer)
- **Paper:** [2406.16338](https://arxiv.org/abs/2406.16338)
- **Point of Contact:** mailto:[Yuxuan Wang](wangyuxuan1@bigai.ai)

![images](./assets/videohallucer_teaser.png)

This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs). VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non-factual hallucinations. We adopt an adversarial binary VideoQA method for comprehensive evaluation, where pairs of basic and hallucinated questions are crafted strategically.

## Data Statistics
| | Object-Relation Hallucination | Temporal Hallucination | Semantic Detail Hallucination | External Factual Hallucination | External Nonfactual Hallucination |
| ---- | ---- | ---- | ---- | ---- | ---- |
|Questions | 400 | 400 | 400 | 400 | 400 |
|Videos | 183 | 165 | 400| 200 | 200 |


## Evaluation

We provide [VideoHallucerKit](https://github.com/patrick-tssn/VideoHallucer?tab=readme-ov-file#videohallucerkit) for evaluation


## Leaderboard

See our [page](https://videohallucer.github.io/)