# Dataset Schema

The primary dataset unit is a `CILRecord`: one action intervention executed from one shared
serialized simulator state. Records with the same `group_id` form a `CILGroup`.

## CILRecord Fields

Core fields:

- `version`: schema version string.
- `record_id`: deterministic record identifier.
- `group_id`: shared intervention lattice ID.
- `state_hash`: hash of the serialized initial simulator state.
- `task_id`: task identifier.
- `scene_id`: optional scene identifier.
- `instruction`: language instruction used for the group.
- `instruction_family`: task family/templates/minimal-pair metadata.
- `observation_ref` / `observation_inline`: initial observation by reference or inline payload.
- `action_chunk`: action intervention.
- `next_observation_ref` / `next_observation_inline`: post-action observation.
- `structured_effect`: extracted physical and symbolic effect.
- `reward`: progress, success, terminal success, and dense components.
- `regret`: best group reward minus this record reward.
- `rank_within_group`: reward rank among same-state candidates.
- `candidate_type`: expert, near miss, wrong target, wrong relation, random negative, no-op, etc.
- `failure`: deterministic failure classification plus optional language explanation.
- `metadata`: backend, benchmark, annotation, and experiment metadata.

## ActionChunk

`ActionChunk` stores:

- `action_id`
- `representation`
- `horizon`
- `values`
- `skill_type`
- `metadata`

For the toy backend, semantic actions use dictionaries such as `move_to`, `grasp`, `push`,
`place_at`, `open`, and `close`. Numeric/vector actions are supported for model training.

## StructuredEffect

`StructuredEffect` stores:

- object pose deltas
- contact events
- relation truth values before and after
- grasp success
- moved objects
- articulation deltas
- symbolic before/after states
- metadata

Reward and failure classification should be reproducible from this object plus the task.

## Directory Layout

```text
data/cil_toy/
  metadata.json
  manifest.json
  shards/
    shard_000000.jsonl
    shard_000001.jsonl
  group_index.jsonl
  record_index.jsonl
  states/
    <group_id>.pkl
```

`metadata.json` summarizes dataset name, schema version, backend, group count, record count, K,
task count, seed, and creation time. `group_index.jsonl` stores group-to-shard mapping, record IDs,
reward summaries, success counts, and candidate-type counts.

## Group Invariants

Within a valid group:

- all records share `group_id`
- all records share `state_hash`
- all records share `task_id`
- ranks and regret are computed only against records from the same group

These invariants are required for same-state ranking and causal contrastive losses.