| --- |
| license: mit |
| task_categories: |
| - video-classification |
| - multiple-choice |
| size_categories: 1K<n<10K |
| tags: |
| - physics |
| - pendulum |
| - causal-reasoning |
| - clevrer |
| - world-model |
| - cjepa |
| - vqa |
| - synthetic |
| --- |
| |
| # Pendulum C-JEPA 3K + CLEVRER-style Questions |
|
|
| **3,000 silent H.264 MP4s of a nonlinear physical pendulum** (160 frames @ 12 fps = 13.33 s, |
| 196×196 px), with frame-aligned CSV labels and **~24,000 CLEVRER-style questions** |
| (descriptive / explanatory / predictive / counterfactual) whose answers are all derived |
| from the ground-truth simulation. |
|
|
| This extends [`jimchen2/pendulum-cjepa-160f`](https://huggingface.co/datasets/jimchen2/pendulum-cjepa-160f) |
| (300 videos) two ways: |
|
|
| 1. **10× more videos**, same renderer, same physics conventions, same CSV schema. |
| 2. **The validation split is out-of-distribution physics, not a held-out seed.** |
| Training videos all live in the original parameter box; every validation video |
| breaks at least one physical assumption, so a model that merely memorizes the |
| training distribution generically fails there. This matches the intent of a |
| *causal* world-model dataset. |
|
|
| | split | videos | physics | |
| | --- | --- | --- | |
| | train | 2,400 | in-distribution (identical recipe to the original dataset) | |
| | val | 600 (6 regimes × 100) | **OOD** — see regimes below | |
|
|
| ## OOD validation regimes |
|
|
| | regime | what changes vs train | why it is a causal test | |
| | --- | --- | --- | |
| | `val_high_damp` | γ ∈ U(0.55, 0.90) s⁻¹ (train: 0.15–0.50) | envelope time-constant τ=2/γ as low as 2.2 s, shorter than anything seen | |
| | `val_low_damp` | γ ∈ U(0, 0.08) s⁻¹ | near-conservative dynamics; amplitude/energy almost constant | |
| | `val_large_angle` | θ₀ = ±(45–80)° (train: ±15–40°) | strong nonlinearity: period elongated up to ~30%, visibly non-sinusoidal | |
| | `val_gravity` | g ∈ {1.62 Moon, 3.71 Mars, 24.79 Jupiter} m/s² | period shifts by up to 2.5× both directions | |
| | `val_quad_damping` | drag = −β·ω·\|ω\|, β ∈ U(0.10, 0.35); γ = 0 | **new force law**: amplitude envelope decays ~linearly, not exponentially | |
| | `val_driven` | periodic torque A·cos(2πf t+φ), A ∈ U(0.35, 1.0) rad/s², f ∈ U(0.35, 0.65) Hz | **new causal mechanism**: beats / resonant growth around the ~0.5 Hz natural frequency; energy no longer decays monotonically | |
|
|
| Config rejection-sampling guarantees every simulated frame stays inside the scene and |
| |θ| ≤ 85° (the driven regime is the main source of rejections, near resonance). |
|
|
| ## Question set (CLEVRER taxonomy) |
|
|
| ~8 questions per video, ~24,000 total. Format mirrors CLEVRER: |
|
|
| ```json |
| { |
| "info": {...}, |
| "scene_annotation": [ |
| {"video_id": "video_00000", |
| "video_filename": "videos/train/video_00000.mp4", |
| "split": "train", "regime": "train", |
| "questions": [ |
| {"question_id": 0, "question_type": "descriptive", |
| "question": "...", "answer": "left"}, |
| {"question_id": 1, "question_type": "counterfactual", |
| "question": "...", "choices": ["...", "..."], "answer": 0} |
| ]} |
| ] |
| } |
| ``` |
|
|
| - **descriptive** — observable facts: release side, bottom-crossing count, end-of-clip |
| motion, damping envelope phrase, light drift, energy shape, fastest point, shadow direction. |
| Mostly free-form string answers; three templates are convenience multiple-choice. |
| - **explanatory** *(4-way MC)* — what is responsible for what you see: |
| amplitude decay (friction), stopping at the top (velocity through zero), |
| fastest at the bottom (energy exchange), shadow sweeping (bob blocks light), |
| never returning to release height (dissipated energy), beats in driven clips |
| (rhythmic external push), large-angle period elongation (sin θ vs θ). |
| - **predictive** *(4-way MC)* — next peak higher/lower/same, motion right after the |
| clip ends, bottom crossing within the next second, long-term fate |
| (settle / keep swinging / driven pattern), future shadow motion. |
| All computed from a **hidden 28-second ground-truth continuation** of the |
| simulation (videos show only the first 13.33 s), so these have real answers, |
| unlike CLEVRER's heuristic ones. |
| - **counterfactual** *(4-way MC)* — interventions: remove damping (returns to release |
| height), add damping to an undamped clip, double gravity (faster), double rod |
| length (slower), double bob mass (no change — mass independence), move/brighten the |
| light (**no effect on motion** — the illumination is an independent nuisance factor |
| by construction), steeper release (period slightly longer — amplitude coupling), |
| double damping (dies out ~twice as fast), turn the drive off (settles), vacuum |
| removes quadratic drag, Earth ↔ other-planet gravity. |
|
|
| Each question carries a `concept` tag (e.g. `damping envelope`, |
| `interventional invariance`, `g dependence of period`) so you can score accuracy |
| per causal concept and per OOD regime. |
|
|
| ## Layout |
|
|
| ``` |
| . |
| ├── videos/train/video_00000.mp4 ... 2400 clips |
| ├── videos/val/video_10000.mp4 ... 600 clips (val ids offset by 10000) |
| ├── timeseries/video_XXXXX.csv 3000 frame-aligned label files |
| ├── questions/questions_train.json ~19k questions |
| ├── questions/questions_val.json ~4.8k questions |
| ├── index.csv per-video params, split, regime |
| ├── metadata.json |
| ├── sample_sheet.png |
| └── scripts/ |
| ├── pendulum_gen.py physics + renderer + regime samplers |
| ├── questions.py CLEVRER-style question generator |
| ├── run_generate.py parallel generator (resume-safe) |
| ├── verify_dataset.py integrity checks |
| ├── audit_answers.py re-derive & audit question answers |
| ├── make_preview.py build preview.html |
| ├── prepare_cjepa_root.sh recreate cjepa_data_root/ for extract_four.py |
| └── upload_to_hf.py push this folder to the Hub |
| ``` |
|
|
| Frame `n` ↔ CSV row `frame=n` ↔ `time_s = n/12`. Splits are whole-video. |
|
|
| ## Physics |
|
|
| ``` |
| theta'' = -(g/L)·sin(theta) - gamma·theta' - beta·theta'·|theta'| + A·cos(2π f_d t + φ) |
| ``` |
|
|
| RK4 at fps × 8 = 96 Hz. Train: g=9.81, L=1.0, γ∈U(0.15,0.50), β=0, A=0, |
| θ₀=±U(15,40)°, ω₀=U(−0.2,0.2), light mean U(70,110)°, light amp U(0,15)°, |
| light freq U(0.04,0.12) Hz, random phase — identical to the original dataset. |
| Illumination is independent of the dynamics (interventional-invariance property |
| used by the counterfactual light questions). |
|
|
| `energy_j_per_kg`, `theta_rad`, `omega_rad_s`, `alpha_rad_s2`, bob/light/shadow |
| geometry are all in the CSVs, so numeric extrapolation error at frames 128→159 |
| (+2.67 s ≈ 1.3 oscillations) can also be scored exactly, as in the original. |
|
|
| ## Reproducing / extending |
|
|
| ```bash |
| python scripts/run_generate.py --root dataset --n-train 2400 --val-per-regime 100 --seed 20261014 |
| python scripts/verify_dataset.py dataset |
| ``` |
|
|
| Generation is seed-deterministic (per-video `SeedSequence([seed, index])`), runs at |
| ~1 vid/s/core, and is resume-safe (re-running skips finished videos). |
|
|
| ## License |
|
|
| MIT. Renderer/physics adapted from `jimchen2/pendulum-cjepa-160f` (whose own code |
| credits Huawei Technologies Co., Ltd., also MIT). |
|
|
| ``` |
| cat > /tmp/make_pendulum_shards.py <<'PY' |
| from pathlib import Path |
| import json |
| import webdataset as wds |
| |
| data = Path.home() / "Downloads" / "pendulum-cjepa-3k-clevrer" |
| expected_counts = {"train": 2400, "val": 600} |
| |
| for split, expected in expected_counts.items(): |
| source = data / "videos" / split |
| files = sorted(source.glob("*.mp4")) |
| if len(files) != expected: |
| raise RuntimeError(f"{source}: expected {expected} MP4s, found {len(files)}") |
| |
| output = data / "wds" / split |
| output.mkdir(parents=True, exist_ok=True) |
| pattern = str(output / f"pendulum-{split}-%06d.tar") |
| |
| with wds.ShardWriter(pattern, maxcount=128) as sink: |
| for path in files: |
| sink.write({ |
| "__key__": path.stem, |
| "video.mp4": path.read_bytes(), |
| "meta.json": json.dumps({ |
| "split": split, |
| "filename": path.name, |
| }).encode("utf-8"), |
| }) |
| |
| print(f"{split}: wrote {len(files)} videos to {output}") |
| PY |
| |
| python /tmp/make_pendulum_shards.py |
| ``` |
|
|
