---
license: mit
task_categories:
  - video-classification
  - multiple-choice
size_categories: 1K<n<10K
tags:
  - physics
  - pendulum
  - causal-reasoning
  - clevrer
  - world-model
  - cjepa
  - vqa
  - synthetic
---

# Pendulum C-JEPA 3K + CLEVRER-style Questions

**3,000 silent H.264 MP4s of a nonlinear physical pendulum** (160 frames @ 12 fps = 13.33 s,
196×196 px), with frame-aligned CSV labels and **~24,000 CLEVRER-style questions**
(descriptive / explanatory / predictive / counterfactual) whose answers are all derived
from the ground-truth simulation.

This extends [`jimchen2/pendulum-cjepa-160f`](https://huggingface.co/datasets/jimchen2/pendulum-cjepa-160f)
(300 videos) two ways:

1. **10× more videos**, same renderer, same physics conventions, same CSV schema.
2. **The validation split is out-of-distribution physics, not a held-out seed.**
   Training videos all live in the original parameter box; every validation video
   breaks at least one physical assumption, so a model that merely memorizes the
   training distribution generically fails there. This matches the intent of a
   *causal* world-model dataset.

| split | videos | physics |
| --- | --- | --- |
| train | 2,400 | in-distribution (identical recipe to the original dataset) |
| val   | 600 (6 regimes × 100) | **OOD** — see regimes below |

## OOD validation regimes

| regime | what changes vs train | why it is a causal test |
| --- | --- | --- |
| `val_high_damp` | γ ∈ U(0.55, 0.90) s⁻¹ (train: 0.15–0.50) | envelope time-constant τ=2/γ as low as 2.2 s, shorter than anything seen |
| `val_low_damp` | γ ∈ U(0, 0.08) s⁻¹ | near-conservative dynamics; amplitude/energy almost constant |
| `val_large_angle` | θ₀ = ±(45–80)° (train: ±15–40°) | strong nonlinearity: period elongated up to ~30%, visibly non-sinusoidal |
| `val_gravity` | g ∈ {1.62 Moon, 3.71 Mars, 24.79 Jupiter} m/s² | period shifts by up to 2.5× both directions |
| `val_quad_damping` | drag = −β·ω·\|ω\|, β ∈ U(0.10, 0.35); γ = 0 | **new force law**: amplitude envelope decays ~linearly, not exponentially |
| `val_driven` | periodic torque A·cos(2πf t+φ), A ∈ U(0.35, 1.0) rad/s², f ∈ U(0.35, 0.65) Hz | **new causal mechanism**: beats / resonant growth around the ~0.5 Hz natural frequency; energy no longer decays monotonically |

Config rejection-sampling guarantees every simulated frame stays inside the scene and
|θ| ≤ 85° (the driven regime is the main source of rejections, near resonance).

## Question set (CLEVRER taxonomy)

~8 questions per video, ~24,000 total. Format mirrors CLEVRER:

```json
{
  "info": {...},
  "scene_annotation": [
    {"video_id": "video_00000",
     "video_filename": "videos/train/video_00000.mp4",
     "split": "train", "regime": "train",
     "questions": [
       {"question_id": 0, "question_type": "descriptive",
        "question": "...", "answer": "left"},
       {"question_id": 1, "question_type": "counterfactual",
        "question": "...", "choices": ["...", "..."], "answer": 0}
     ]}
  ]
}
```

- **descriptive** — observable facts: release side, bottom-crossing count, end-of-clip
  motion, damping envelope phrase, light drift, energy shape, fastest point, shadow direction.
  Mostly free-form string answers; three templates are convenience multiple-choice.
- **explanatory** *(4-way MC)* — what is responsible for what you see:
  amplitude decay (friction), stopping at the top (velocity through zero),
  fastest at the bottom (energy exchange), shadow sweeping (bob blocks light),
  never returning to release height (dissipated energy), beats in driven clips
  (rhythmic external push), large-angle period elongation (sin θ vs θ).
- **predictive** *(4-way MC)* — next peak higher/lower/same, motion right after the
  clip ends, bottom crossing within the next second, long-term fate
  (settle / keep swinging / driven pattern), future shadow motion.
  All computed from a **hidden 28-second ground-truth continuation** of the
  simulation (videos show only the first 13.33 s), so these have real answers,
  unlike CLEVRER's heuristic ones.
- **counterfactual** *(4-way MC)* — interventions: remove damping (returns to release
  height), add damping to an undamped clip, double gravity (faster), double rod
  length (slower), double bob mass (no change — mass independence), move/brighten the
  light (**no effect on motion** — the illumination is an independent nuisance factor
  by construction), steeper release (period slightly longer — amplitude coupling),
  double damping (dies out ~twice as fast), turn the drive off (settles), vacuum
  removes quadratic drag, Earth ↔ other-planet gravity.

Each question carries a `concept` tag (e.g. `damping envelope`,
`interventional invariance`, `g dependence of period`) so you can score accuracy
per causal concept and per OOD regime.

## Layout

```
.
├── videos/train/video_00000.mp4 ...        2400 clips
├── videos/val/video_10000.mp4 ...           600 clips (val ids offset by 10000)
├── timeseries/video_XXXXX.csv              3000 frame-aligned label files
├── questions/questions_train.json          ~19k questions
├── questions/questions_val.json            ~4.8k questions
├── index.csv                               per-video params, split, regime
├── metadata.json
├── sample_sheet.png
└── scripts/
    ├── pendulum_gen.py                     physics + renderer + regime samplers
    ├── questions.py                        CLEVRER-style question generator
    ├── run_generate.py                     parallel generator (resume-safe)
    ├── verify_dataset.py                   integrity checks
    ├── audit_answers.py                    re-derive & audit question answers
    ├── make_preview.py                     build preview.html
    ├── prepare_cjepa_root.sh               recreate cjepa_data_root/ for extract_four.py
    └── upload_to_hf.py                     push this folder to the Hub
```

Frame `n` ↔ CSV row `frame=n` ↔ `time_s = n/12`. Splits are whole-video.

## Physics

```
theta'' = -(g/L)·sin(theta) - gamma·theta' - beta·theta'·|theta'| + A·cos(2π f_d t + φ)
```

RK4 at fps × 8 = 96 Hz. Train: g=9.81, L=1.0, γ∈U(0.15,0.50), β=0, A=0,
θ₀=±U(15,40)°, ω₀=U(−0.2,0.2), light mean U(70,110)°, light amp U(0,15)°,
light freq U(0.04,0.12) Hz, random phase — identical to the original dataset.
Illumination is independent of the dynamics (interventional-invariance property
used by the counterfactual light questions).

`energy_j_per_kg`, `theta_rad`, `omega_rad_s`, `alpha_rad_s2`, bob/light/shadow
geometry are all in the CSVs, so numeric extrapolation error at frames 128→159
(+2.67 s ≈ 1.3 oscillations) can also be scored exactly, as in the original.

## Reproducing / extending

```bash
python scripts/run_generate.py --root dataset --n-train 2400 --val-per-regime 100 --seed 20261014
python scripts/verify_dataset.py dataset
```

Generation is seed-deterministic (per-video `SeedSequence([seed, index])`), runs at
~1 vid/s/core, and is resume-safe (re-running skips finished videos).

## License

MIT. Renderer/physics adapted from `jimchen2/pendulum-cjepa-160f` (whose own code
credits Huawei Technologies Co., Ltd., also MIT).

```
cat > /tmp/make_pendulum_shards.py <<'PY'
from pathlib import Path
import json
import webdataset as wds

data = Path.home() / "Downloads" / "pendulum-cjepa-3k-clevrer"
expected_counts = {"train": 2400, "val": 600}

for split, expected in expected_counts.items():
    source = data / "videos" / split
    files = sorted(source.glob("*.mp4"))
    if len(files) != expected:
        raise RuntimeError(f"{source}: expected {expected} MP4s, found {len(files)}")

    output = data / "wds" / split
    output.mkdir(parents=True, exist_ok=True)
    pattern = str(output / f"pendulum-{split}-%06d.tar")

    with wds.ShardWriter(pattern, maxcount=128) as sink:
        for path in files:
            sink.write({
                "__key__": path.stem,
                "video.mp4": path.read_bytes(),
                "meta.json": json.dumps({
                    "split": split,
                    "filename": path.name,
                }).encode("utf-8"),
            })

    print(f"{split}: wrote {len(files)} videos to {output}")
PY

python /tmp/make_pendulum_shards.py
```