LLaVA-Video-83K/README.md

# LLaVA-Video-83K

An 83,000-sample subset of [`lmms-lab/LLaVA-Video-178K`](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K),
selected by stratified density-flattened sampling rather than uniformly at random,
and shipped **with its media** so it is usable straight out of the bucket.

| | |
|---|---|
| Records | **83,000** (5.0% of the 1,650,547 upstream records) |
| QA pairs | 279,679 |
| Distinct media | 71,224 (59,428 mp4 + 11,796 llava_hound frame scenes) |
| Media size | 428.9 GiB |
| Tasks | Caption 10.97% · Open-ended VQA 75.84% · Multiple-choice VQA 13.19% |

Media for the `llava_hound` split comes from
[`ShareGPTVideo/train_video_and_instruction`](https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction)
(`train_300k`), which covers all 179,318 referenced scenes.

## Layout

```
LLaVA-Video-83K/
├── annotations/
│   ├── all_sampled.json          # 83,000 records, upstream schema unchanged
│   └── by_source/*.json          # 39 files, split as upstream
├── metadata/
│   ├── manifest.parquet          # record -> media, task, duration bucket, source
│   ├── shard_index.parquet       # member -> (shard, offset, nbytes)   ← random access
│   ├── scene_index.parquet       # llava_hound scene -> frame count
│   └── SAMPLING_REPORT.md        # how the subset was drawn, and its verification
├── embeddings/                   # MiniLM-L6 vector per record, for re-drawing subsets
├── shards/
│   ├── video/video-NNNNN.tar     # uncompressed, ~2 GiB each
│   └── frame/frame-NNNNN.tar     # uncompressed, ~2 GiB each
└── videos_large/                 # the 188 mp4s above 64 MiB, stored loose
```

A record's `video` field is unchanged from upstream and is exactly the member
name inside the shards, so nothing in the annotations needs rewriting.

## Why tar shards

Media is stored as **uncompressed** tar plus a byte-offset index, not as loose
files. A member therefore occupies a contiguous byte range and is one `pread`
away — `decord` decodes it from `BytesIO` byte-identically to reading the file
from disk, seeks included (verified on files up to 707 MiB).

The reason is I/O, not a Hub limit: 377k loose objects make transfer, listing
and Lustre metadata operations painfully slow, while ~220 shards of 2 GiB read
at full bandwidth. Videos above 64 MiB are the exception — they stay loose so a
dataloader never has to buffer a 700 MiB file in RAM.

## Usage

```python
from lvdu_reader import ShardReader

r = ShardReader("/path/to/LLaVA-Video-83K")

vr = r.video_reader("academic_source/youcook2/203/Qh_rtJaCpXs/split_6.mp4")   # decord VideoReader
frames = r.scene_frames("shareVideoGPTV/frames/all_frames/v_XXX-Scene-001")
```

`ShardReader` is safe under DataLoader workers: shard handles are opened lazily
per process and never shared across a fork.

## How the subset was drawn

Uniform random sampling inherits the upstream corpus's pathologies. This subset
instead:

- keeps the **task / duration / source mix** of the parent pool (task shares match to 0.01%);
- gives small sources (activitynetqa, nextqa, perceptiontest — 41k records between them) a floor so they survive the cut;
- drops records whose answers are entirely boilerplate, cutting ultra-short open-ended answers from 5.65% (random) to **2.39%**;
- **flattens density** inside each stratum: cluster the MiniLM vectors, then allocate per cluster ∝ size^0.5, so dense topic clusters stop dominating;
- constrains the multiple-choice **length shortcut**: a blind "pick the longest option" guesser scores 51.23% on the full set, and **49.79%** here.

Measured effect of the flattening, against an equally-sized random draw: cluster
*hit rate* is unchanged (the metric saturates at 5% sampling), but concentration
drops — max samples per cluster 287 → **161** globally and 52 → **45** inside the
largest stratum; Gini 0.712 → 0.697 and 0.394 → **0.341**.

Full numbers, including the verification of every claim above, are in
`metadata/SAMPLING_REPORT.md`.

## Provenance and licence

This is a derived subset. The videos and annotations remain under the terms of
the upstream releases — see `lmms-lab/LLaVA-Video-178K` and
`ShareGPTVideo/train_video_and_instruction`. Cite those datasets, not this
repackaging, for the underlying data.