Experiments

Experiments in DoVLA-CIL focus on whether same-state counterfactual interventions improve action selection, effect prediction, language controllability, and robustness.

CausalStress

CausalStress generates controlled toy-backend groups across:

minimal_language_change
wrong_target_distractor
near_miss_boundary
physics_shift_placeholder
effect_query
counterfactual_ranking
similar_distractors
spatial_relation_minimal_pairs
negation_and_avoidance
sequential_tasks
irreversible_failure
physics_perturbation_placeholders

Harder families include red mug vs red cup, blue bowl vs blue plate, same category/different color, same color/different category, left/right, inside/next-to, behind/front, negation, sequential skills, out-of-workspace failures, low friction, heavy objects, and sticky drawers.

Metrics:

task_success_rate
instruction_switch_accuracy
pairwise_ranking_accuracy
top1_action_selection
ndcg_at_k
effect_prediction_mae
success_prediction_accuracy
regret_calibration_error
per-category success, instruction switch, and failure rate
target-selection confusion matrices

Run:

python scripts/eval_causalstress.py \
  --checkpoint runs/dovla_toy/best.pt \
  --backend toy \
  --out runs/dovla_toy/causalstress.json \
  --num-tasks 20 \
  --k 16

Scaling Over K

Scaling experiments keep total record budget fixed as B = N * K. For each K, the runner chooses N = total_records // K, generates a toy CIL dataset, trains DoVLA, evaluates CausalStress, writes per-run metrics, aggregates CSVs, creates plots, and fits:

score = alpha + beta_log_k * log(K)

Run:

python scripts/run_scaling.py \
  --backend toy \
  --tasks builtins \
  --out runs/scaling_toy \
  --total-records 4096 \
  --k-values 1,2,4,8,16,32 \
  --epochs 3 \
  --seed 0

Baselines

python scripts/run_baseline.py \
  --baseline expert_only_bc \
  --dataset data/cil_toy \
  --out runs/baselines/expert_only_bc

Modes:

expert_only_bc: one best/expert action per group; no ranking/regret.
more_independent_demos: K=1-style independent demonstration comparison.
random_negatives: structured candidates replaced by random-negative labels.
cross_state_negatives: matched-budget reward-ordered pairs from different states of the same task; this tests whether exact same-state cancellation matters.
label_only_counterfactual: heuristic labels without measured outcomes.
world_model_auxiliary: effect/progress/success auxiliary losses without ranking/regret.
no_effect_head: effect loss removed.
no_rank_regret: ranking and regret removed.

Reports

Dataset report:

python scripts/report_dataset.py --dataset data/cil_toy --out reports/cil_toy

Evaluation report:

python scripts/report_eval.py \
  --inputs "runs/scaling_toy/*/metrics.json" \
  --out reports/scaling_toy

Paper artifacts:

python scripts/make_paper_artifacts.py --runs runs --out paper_artifacts

The paper artifact script writes scaling, baseline, ablation, and per-category tables, plus figures for performance vs K, same-state vs cross-state ranking, physical-outcome vs label-only, success by failure category, and regret calibration.

Optional TransferCritic Studies

TransferCritic is a secondary data-curation module for selecting CIL records or groups under a budget. It compares random, top-reward, task-balanced, and set-conditioned utility selections. See docs/transfercritic.md.

Optional Retrieval Studies

Critic-gated retrieval is an inference-time extension for retrieving successful, near-miss, and partial-success CIL exemplars. It compares no retrieval, nearest-neighbor, success-only, success/failure contrastive, and critic-gated retrieval. See docs/retrieval.md.

Configs

Reproducible YAML configs live under:

configs/toy/
configs/baselines/
configs/large/

The loader supports environment expansion, CLI overrides, and saving resolved configs into run directories.

Large-Scale Manifests

Large multi-stage experiment manifests live under manifests/:

cil_160m.yaml
cil_1b_template.yaml
scaling_k_sweep.yaml
baselines_full.yaml

Plan a manifest without executing jobs:

python scripts/run_manifest.py manifests/scaling_k_sweep.yaml --dry-run

Emit generic Slurm scripts and save a resolved manifest:

python scripts/run_manifest.py \
  manifests/cil_160m.yaml \
  --dry-run \
  --emit-slurm \
  --out runs/cil_160m_plan

The manifest runner redacts secret-looking fields and never emits API keys into planned commands. Manifests are validated before any files are written: positive record counts, training duration, loss weights, and unique positive K values are checked locally. Slurm resources use the optional scheduler manifest section and may be overridden while emitting scripts with DOVLA_PARTITION, DOVLA_ACCOUNT, DOVLA_CPUS_PER_TASK, DOVLA_GPUS_PER_TASK, DOVLA_MEM, DOVLA_TIME, and DOVLA_LOG_DIR. These values are resolved into literal #SBATCH directives because Slurm does not expand shell expressions in directive lines.

Backend planning is explicit. Toy manifests call generate_cil.py and may be run with --execute-local. ManiSkill manifests call generate_maniskill_lattice.py, multiply num_tasks * num_states_per_task into the physical state-group count, and require simulator_params.demo_path (normally supplied through MANISKILL_DEMO_PATH). Genesis remains a visible placeholder until a task-specific measured-intervention adapter exists. Training loss weights are forwarded as repeated --loss-weight NAME=VALUE arguments and are saved again in the trainer's resolved config.