horizon-shift — checkpoints, demonstrations and results

Artifacts for "When Should Robot Policies Replan? A Commitment–Feedback View of Action Chunk Execution".

The paper studies the execution horizon L of action-chunk policies: how many predicted actions are executed open loop before the policy is queried again. The trained policy is held fixed and only L varies. This bucket contains every trained policy behind the paper's numbers, the demonstration datasets we generated or edited, and the frozen result tables.

Code (training, evaluation, analysis, paper build) is released separately. Loading a checkpoint requires that code (tba.models.policies.load_agent).

Layout

checkpoints/
  maniskill_state/<Task>/<policy>/<demonstrations>/<condition>/seed<k>/{model.pt, config.json}
  maniskill_rgb/PickCube-v1/<policy>/low-ambiguity/N<n>/seed<k>/{model.pt, config.json}
  robomimic_lowdim/<task>/act/<demonstrations>/<condition>/seed<k>/{model.pt, config.json}
demonstrations/
  maniskill/<Task>/<demonstrations>/trajectory.<obs>.<control>.physx_cpu.{h5,json}  (+ edits.json / holds.json)
  maniskill/StackCube-v1/training_subsets/*.json      (episode indices of every N-subset)
  robomimic/<task>/high-ambiguity/ph_low_dim_hold0-16.{hdf5,json}
results/
  core_stackcube/     frozen results registry (one row per model x condition x horizon), analysis JSONs
  generalization/     external-setting summaries (ManiSkill tasks, robomimic), audits, calibrations
  vla_libero/         OpenVLA-OFT execution-semantics equivalence test
MANIFEST.jsonl        every file: path, bytes, sha256, source run, paper section

Naming

Every config.json lists the exact training episode indices, the seed, the model hyper-parameters, the internal run name used in the paper's analysis code, and the paper section it backs.

Evaluating a checkpoint

• Execution horizon. The policy predicts a chunk of H actions; execute the first L and re-query. L=1 is "Fresh".

• Action gain (ACT only). State-based ACT under-predicts translation magnitude. Every ACT result scales the executed xyz by one frozen per-setting gain, fixed by a pre-declared criterion on held-out demonstration states (config.json: evaluation_gain_xyz):

  Setting	Gain
  StackCube	1.20
  PickCube	1.10
  PushCube	1.05
  robomimic Lift	1.00
  robomimic Square	1.10

  DP and DirectChunk use no correction.

• Seeds.

  • Test environment seeds are 100000–100299 (ManiSkill) and 100000–100199 (robomimic).
  • Validation seeds start at 200000 and are used only to select L*.
  • Disturbance experiments use the first 200 test seeds.

• ManiSkill. ManiSkill 3.0.1, obs_mode=state, physx_cpu, pd_ee_delta_pos (pd_ee_delta_pose for PegInsertionSide).

• robomimic. robosuite 1.4 with the low_dim_v141 observation keys listed in config.json.

Known pitfalls (found while building these experiments)

1. RGB baselines: state key order.
   • The official ManiSkill RGB training scripts build the state vector in the observation-space key order: sorted, e.g. extra = [goal_pos, is_grasped, tcp_pose].
   • FlattenRGBDObservationWrapper at evaluation keeps the raw insertion order ([is_grasped, tcp_pose, goal_pos]).
   • Evaluating with the wrapper silently permutes the state and yields ~0% success. Build evaluation observations exactly as training does.
2. OpenVLA-OFT execution horizon. The official LIBERO evaluator stores the chunk in a deque(maxlen=L), which executes the last L actions for L < 8. Execute chunk[:L] instead.
3. LIBERO environment reuse. Re-using one LIBERO env across episodes carries simulator internals over, so identical initial states diverge, even between two runs of the official evaluator. Create a fresh env per episode for paired comparisons. See results/vla_libero/.

Provenance and licences

• ManiSkill demonstrations.
  • natural ManiSkill demonstrations are ManiSkill's official motion-planning demos (Apache-2.0), converted to end-effector delta control with ManiSkill's replay tool.
  • The unconverted originals are not re-hosted, except for PushCube-v1, which we generated with ManiSkill's motion planner.
  • low-ambiguity and high-ambiguity are our edits; edits.json and holds.json record exactly which steps were removed or inserted.
• robomimic. The proficient-human datasets are by Mandlekar et al. (MIT); download the originals from robomimic. We host only our edited high-ambiguity variants.
• Not included.
  • Per-episode raw rollouts (~3.6 GB; available on request).
  • The OpenVLA-OFT checkpoint (moojink/openvla-7b-oft-finetuned-libero-spatial-object-goal-10 on the Hub).
  • Exploratory models that do not back a reported number.

Integrity

import hashlib, json
for row in map(json.loads, open("MANIFEST.jsonl")):
    h = hashlib.sha256(open(row["path"], "rb").read()).hexdigest()
    assert h == row["sha256"], row["path"]