README.md

---
language:
- en
pretty_name: CUHK-X — Large Model Track
tags:
- video
- multimodal
- human-activity
- depth
- infrared
- thermal
task_categories:
- visual-question-answering
---
# CUHK-X — Large Model Track

Multimodal **video question answering (VQA)** over human daily-activity clips recorded at home.
Given a short multimodal video, answer multiple-choice questions about it.

## Repository layout

```
.
├── Training/
│   ├── training_qa.csv      # questions + answers
│   ├── modality_list.csv    # which modalities each clip has
│   └── data/
│       ├── HARn.zip   →  HARn/<action>/<user>/<trial>/<modality>/<modality>.mp4
│       └── HAU.zip    →  HAU/<user>/<trial>/<modality>/<modality>.mp4
└── Testing/
    ├── test_qa.csv          # questions only (fill the `prediction` column)
    └── data/
        └── large_model_track_test.zip  →  large_model_track_test/<id>/<modality>/<modality>.mp4
```

The videos are packaged as the `*.zip` files under each `data/` folder — unzip them first.
**The `path` column in the CSVs is relative to the extracted root** (i.e. it starts with
`HARn/`, `HAU/`, or `large_model_track_test/`).

### Two sources (`source` column)

- **HARn** — single-action clips; the path contains the action, e.g. `HARn/0_Wash_face/user16/1-1-2`.
- **HAU** — complex / multi-action clips; the path is `HAU/<user>/<trial>`.

Test clips are anonymized as `LM_test_XXXX`.

### Modalities

Each clip is a directory with one sub-directory per modality, each holding a single `.mp4` (MPEG-4, 10 fps):

| Modality        | Description                                     | HARn | HAU |
| --------------- | ----------------------------------------------- | :--: | :-: |
| `Depth`       | depth map (16-bit, normalized to 8-bit gray)    |  ✓  | ✓ |
| `Depth_Color` | colorized depth                                 |  ✓  | ✓ |
| `IR`          | infrared                                        |  ✓  | ✓ |
| `Thermal`     | thermal (25 fps; frame count differs by design) |  —  | ✓ |

Not every clip has every modality — see `modality_list.csv`. (HARn never includes Thermal.)

## Files

**`Training/training_qa.csv`** — `qa_id, source, path, category, question, A, B, C, D, answer`

- `path` is the **clip directory**, e.g. `HARn/0_Wash_face/user16/1-1-2`.
- Load a modality from `<path>/<modality>/<modality>.mp4`.

**`Testing/test_qa.csv`** — same columns, but `path` is a **specific modality file**,
e.g. `large_model_track_test/LM_test_0066/Depth/Depth.mp4`. Swap the trailing `Depth/Depth.mp4` for
another modality if needed. There is no `answer`; fill the empty `prediction` column.

**`Training/modality_list.csv`** — `path, present_modalities, missing_modalities` (semicolon-separated).

> Note: in **training**, `path` points to a clip *directory*; in **testing**, it points to a modality *file*.

### Question categories

`single`, `multi`, `object_interaction`, `sequence`, `combination`, `emotion`.

## Submission

For each `qa_id` in `test_qa.csv`, fill `prediction` with the option letter(s): a single letter (e.g. `A`)
for single-answer questions, or several (e.g. `AC`) for multi-answer questions.

## Statistics

|          | Clips | Questions |
| -------- | ----: | --------: |
| Training | 3,912 |     4,087 |
| Testing  |   208 |       682 |

## Quick start

```python
import csv, os

# 1) unzip Training/data/HARn.zip and HAU.zip into ROOT (you get HARn/ and HAU/)
ROOT = "Training/data"   # wherever you extracted

for r in csv.DictReader(open("Training/training_qa.csv", encoding="utf-8-sig")):
    # training path is a clip directory -> pick a modality
    depth_mp4 = os.path.join(ROOT, r["path"], "Depth", "Depth.mp4")
    # use r["question"], r["A"]..r["D"], r["answer"]
```