docs/experiments.md

# Experiments

Experiments in DoVLA-CIL focus on whether same-state counterfactual interventions improve action
selection, effect prediction, language controllability, and robustness.

## CausalStress

CausalStress generates controlled toy-backend groups across:

- `minimal_language_change`
- `wrong_target_distractor`
- `near_miss_boundary`
- `physics_shift_placeholder`
- `effect_query`
- `counterfactual_ranking`
- `similar_distractors`
- `spatial_relation_minimal_pairs`
- `negation_and_avoidance`
- `sequential_tasks`
- `irreversible_failure`
- `physics_perturbation_placeholders`

Harder families include red mug vs red cup, blue bowl vs blue plate, same category/different color,
same color/different category, left/right, inside/next-to, behind/front, negation, sequential
skills, out-of-workspace failures, low friction, heavy objects, and sticky drawers.

Metrics:

- `task_success_rate`
- `instruction_switch_accuracy`
- `pairwise_ranking_accuracy`
- `top1_action_selection`
- `ndcg_at_k`
- `effect_prediction_mae`
- `success_prediction_accuracy`
- `regret_calibration_error`
- per-category success, instruction switch, and failure rate
- target-selection confusion matrices

Run:

```bash
python scripts/eval_causalstress.py \
  --checkpoint runs/dovla_toy/best.pt \
  --backend toy \
  --out runs/dovla_toy/causalstress.json \
  --num-tasks 20 \
  --k 16
```

## Scaling Over K

Scaling experiments keep total record budget fixed as `B = N * K`. For each `K`, the runner chooses
`N = total_records // K`, generates a toy CIL dataset, trains DoVLA, evaluates CausalStress, writes
per-run metrics, aggregates CSVs, creates plots, and fits:

```text
score = alpha + beta_log_k * log(K)
```

Run:

```bash
python scripts/run_scaling.py \
  --backend toy \
  --tasks builtins \
  --out runs/scaling_toy \
  --total-records 4096 \
  --k-values 1,2,4,8,16,32 \
  --epochs 3 \
  --seed 0
```

## Baselines

```bash
python scripts/run_baseline.py \
  --baseline expert_only_bc \
  --dataset data/cil_toy \
  --out runs/baselines/expert_only_bc
```

Modes:

- `expert_only_bc`: one best/expert action per group; no ranking/regret.
- `more_independent_demos`: K=1-style independent demonstration comparison.
- `random_negatives`: structured candidates replaced by random-negative labels.
- `cross_state_negatives`: matched-budget reward-ordered pairs from different states of the same
  task; this tests whether exact same-state cancellation matters.
- `label_only_counterfactual`: heuristic labels without measured outcomes.
- `world_model_auxiliary`: effect/progress/success auxiliary losses without ranking/regret.
- `no_effect_head`: effect loss removed.
- `no_rank_regret`: ranking and regret removed.

## Reports

Dataset report:

```bash
python scripts/report_dataset.py --dataset data/cil_toy --out reports/cil_toy
```

Evaluation report:

```bash
python scripts/report_eval.py \
  --inputs "runs/scaling_toy/*/metrics.json" \
  --out reports/scaling_toy
```

Paper artifacts:

```bash
python scripts/make_paper_artifacts.py --runs runs --out paper_artifacts
```

The paper artifact script writes scaling, baseline, ablation, and per-category tables, plus figures
for performance vs K, same-state vs cross-state ranking, physical-outcome vs label-only, success by
failure category, and regret calibration.

## Optional TransferCritic Studies

TransferCritic is a secondary data-curation module for selecting CIL records or groups under a
budget. It compares random, top-reward, task-balanced, and set-conditioned utility selections. See
`docs/transfercritic.md`.

## Optional Retrieval Studies

Critic-gated retrieval is an inference-time extension for retrieving successful, near-miss, and
partial-success CIL exemplars. It compares no retrieval, nearest-neighbor, success-only,
success/failure contrastive, and critic-gated retrieval. See `docs/retrieval.md`.

## Configs

Reproducible YAML configs live under:

- `configs/toy/`
- `configs/baselines/`
- `configs/large/`

The loader supports environment expansion, CLI overrides, and saving resolved configs into run
directories.

## Large-Scale Manifests

Large multi-stage experiment manifests live under `manifests/`:

- `cil_160m.yaml`
- `cil_1b_template.yaml`
- `scaling_k_sweep.yaml`
- `baselines_full.yaml`

Plan a manifest without executing jobs:

```bash
python scripts/run_manifest.py manifests/scaling_k_sweep.yaml --dry-run
```

Emit generic Slurm scripts and save a resolved manifest:

```bash
python scripts/run_manifest.py \
  manifests/cil_160m.yaml \
  --dry-run \
  --emit-slurm \
  --out runs/cil_160m_plan
```

The manifest runner redacts secret-looking fields and never emits API keys into planned commands.
Manifests are validated before any files are written: positive record counts, training duration,
loss weights, and unique positive K values are checked locally. Slurm resources use the optional
`scheduler` manifest section and may be overridden while emitting scripts with
`DOVLA_PARTITION`, `DOVLA_ACCOUNT`, `DOVLA_CPUS_PER_TASK`, `DOVLA_GPUS_PER_TASK`,
`DOVLA_MEM`, `DOVLA_TIME`, and `DOVLA_LOG_DIR`. These values are resolved into literal
`#SBATCH` directives because Slurm does not expand shell expressions in directive lines.

Backend planning is explicit. Toy manifests call `generate_cil.py` and may be run with
`--execute-local`. ManiSkill manifests call `generate_maniskill_lattice.py`, multiply
`num_tasks * num_states_per_task` into the physical state-group count, and require
`simulator_params.demo_path` (normally supplied through `MANISKILL_DEMO_PATH`). Genesis remains
a visible placeholder until a task-specific measured-intervention adapter exists. Training loss
weights are forwarded as repeated `--loss-weight NAME=VALUE` arguments and are saved again in the
trainer's resolved config.