| # Dataset Schema |
|
|
| The primary dataset unit is a `CILRecord`: one action intervention executed from one shared |
| serialized simulator state. Records with the same `group_id` form a `CILGroup`. |
|
|
| ## CILRecord Fields |
|
|
| Core fields: |
|
|
| - `version`: schema version string. |
| - `record_id`: deterministic record identifier. |
| - `group_id`: shared intervention lattice ID. |
| - `state_hash`: hash of the serialized initial simulator state. |
| - `task_id`: task identifier. |
| - `scene_id`: optional scene identifier. |
| - `instruction`: language instruction used for the group. |
| - `instruction_family`: task family/templates/minimal-pair metadata. |
| - `observation_ref` / `observation_inline`: initial observation by reference or inline payload. |
| - `action_chunk`: action intervention. |
| - `next_observation_ref` / `next_observation_inline`: post-action observation. |
| - `structured_effect`: extracted physical and symbolic effect. |
| - `reward`: progress, success, terminal success, and dense components. |
| - `regret`: best group reward minus this record reward. |
| - `rank_within_group`: reward rank among same-state candidates. |
| - `candidate_type`: expert, near miss, wrong target, wrong relation, random negative, no-op, etc. |
| - `failure`: deterministic failure classification plus optional language explanation. |
| - `metadata`: backend, benchmark, annotation, and experiment metadata. |
|
|
| ## ActionChunk |
|
|
| `ActionChunk` stores: |
|
|
| - `action_id` |
| - `representation` |
| - `horizon` |
| - `values` |
| - `skill_type` |
| - `metadata` |
|
|
| For the toy backend, semantic actions use dictionaries such as `move_to`, `grasp`, `push`, |
| `place_at`, `open`, and `close`. Numeric/vector actions are supported for model training. |
|
|
| ## StructuredEffect |
|
|
| `StructuredEffect` stores: |
|
|
| - object pose deltas |
| - contact events |
| - relation truth values before and after |
| - grasp success |
| - moved objects |
| - articulation deltas |
| - symbolic before/after states |
| - metadata |
|
|
| Reward and failure classification should be reproducible from this object plus the task. |
|
|
| ## Directory Layout |
|
|
| ```text |
| data/cil_toy/ |
| metadata.json |
| manifest.json |
| shards/ |
| shard_000000.jsonl |
| shard_000001.jsonl |
| group_index.jsonl |
| record_index.jsonl |
| states/ |
| <group_id>.pkl |
| ``` |
|
|
| `metadata.json` summarizes dataset name, schema version, backend, group count, record count, K, |
| task count, seed, and creation time. `group_index.jsonl` stores group-to-shard mapping, record IDs, |
| reward summaries, success counts, and candidate-type counts. |
|
|
| ## Group Invariants |
|
|
| Within a valid group: |
|
|
| - all records share `group_id` |
| - all records share `state_hash` |
| - all records share `task_id` |
| - ranks and regret are computed only against records from the same group |
|
|
| These invariants are required for same-state ranking and causal contrastive losses. |
|
|
