test_set/README.md

# SpatialAV2AV — Test Set (held-out evaluation split)

Origin-disjoint **test split** for evaluating SpatialAV2AV spatial audio-video editing.
This folder ships the **manifests + split reports only** (no media). The media (mp4 with
embedded stereo audio) live in the dataset repo [`BingoG/LTX`](https://huggingface.co/datasets/BingoG/LTX).

## Files

| File | What |
|------|------|
| `test.list` | 11,616 metadata-JSON paths (absolute, on the source cluster) — the held-out test samples |
| `test.relative.list` | same 11,616 samples as **relative** paths (`final_json/<clip>.json`) — resolve against `BingoG/LTX` after `snapshot_download` |
| `split_summary.json` | full split statistics (origins, per-split sample & trajectory counts, seed, input SHA) |
| `split_overlap_report.json` | origin-overlap audit — **all zero** (train/val/test are origin-disjoint) |
| `MANIFEST.sha256` | SHA256 of train/val/test.list + input list, for downstream binding |

## How this split was built

- **Source set:** the 116,147-pair training pool (`all.list`, the authoritative
  SpatialAV2AV set — the same list the step_04000 weights trained on).
- **Ratios:** 80 / 10 / 10 (train / val / test), **seed 20260718** (frozen).
- **Grouping:** origin-disjoint by `video_id` — one source video (and all 10 of its
  camera-trajectory variants) lands entirely in one split. This is what makes the test
  score a **generalization** measure, not a memorization one: if `pan_left` were in train
  and `pan_right` in test, the model would already have seen the same person/room/voice.

### Numbers

| Split | origins | samples |
|-------|--------:|--------:|
| train | 11,616 | 92,915 |
| val   | 1,452  | 11,616 |
| **test**  | **1,452**  | **11,616** |

Test-set trajectory coverage (each of the 10 canonical trajectories present):
`pan_left/right` and `rotate_left/right` = 1,452 each, `push_in`/`pull_out` = 1,452 each,
`fixed_*` = 692–779 each. 1,452 origins ≫ the ≥100 needed for reliable origin-cluster
bootstrap CIs.

## Validation (all PASS)

1. **Origin overlap** — train∩val, train∩test, val∩test all **0** origins.
2. **Coverage** — 92,915 + 11,616 + 11,616 = 116,147 = input (no sample lost).
3. **Trajectory presence** — test contains all pan/rotate directions (I3 counterfactual
   pairs have data).

## Note on how the manifest was generated

The split's deterministic core (`random.Random(20260718)` shuffle + greedy origin
allocation) was reproduced from the filenames because the source media filesystem was
temporarily unreachable for bulk access. `origin_id == video_id == filename-stem-before-'+'`
was verified identical to the JSON contents on a 200-sample check, and `.resolve()` is
idempotent on these paths — so the emitted lists are byte-equivalent to the official
`SpatialAV2AV_build_splits.py` output. Counterfactual pair manifests
(`test_pairs.jsonl`, needed only for I3 eval) are **not** included here; they require
decoding each clip's media and will be added separately.

## Usage

```python
from huggingface_hub import hf_hub_download
p = hf_hub_download("BingoG/SpatialAV2AV", "test_set/test.relative.list")
# point the evaluator's TEST_LIST at the resolved paths under a BingoG/LTX snapshot
```