Buckets:

EmergentTwilight
/

MMCoT

Files
xet
EmergentTwilight/MMCoT / docs /zebra_cot.md
EmergentTwilight
20 days ago
preview code
download
raw
57.7 kB
# Zebra-CoT for UniAR interleaved SFT

## Status and scope

- Local link: `data/Zebra-CoT -> /mnt/ultracube/datasets/Zebra-CoT`
- Dataset: 182,384 train-only examples, 18 configurations, 163 Parquet shards.
- On-disk Parquet payload: 66.10 GiB; images are embedded in Parquet rows.
- Dataset-card license: CC-BY-NC-4.0. Use is limited to non-commercial research unless separately licensed.
- Official sources: [dataset](https://huggingface.co/datasets/multimodal-reasoning-lab/Zebra-CoT), [paper](https://arxiv.org/abs/2507.16746), [training code](https://github.com/multimodal-reasoning-lab/Bagel-Zebra-CoT), and the complete standalone [Graph-Algorithms source](https://huggingface.co/datasets/multimodal-reasoning-lab/Graph-Algorithms).

Zebra-CoT is suitable for teaching UniAR an interleaved visual-reasoning protocol and for studying explicit visual-state rollout. It is not a substitute for Stage 1 posed multi-view supervision: most reasoning images are sketches, crops, edited boards, diagrams, or demonstrated trajectory frames rather than camera/action-conditioned novel views with calibrated geometry.

### Raw 1% development subset

`data/Zebra-CoT-small/` is a directly materialized 633 MiB development subset,
not a symlink and not a preprocessed UniAR cache. It contains 1,825 of 182,384
rows: each of the 18 configurations contributes
`max(1, floor(config_rows * 0.01 + 0.5))` rows selected with seed `20260825`.
Every configuration retains its original Arrow schema and embedded image bytes
in `train-00000-of-00001.parquet`. Exact source shard/row coordinates live in
`SPATIAL_MMCOT_SUBSET.json`; all 1,825 copied rows were verified field-by-field
against their source rows. Reproduce the immutable output at a new path with:

```bash
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" mamba run -n UniAR python -m \
  zebra_sft.sample_raw_subset --source-root data/Zebra-CoT \
  --output-root data/Zebra-CoT-small-rebuild --fraction 0.01 --seed 20260825
```

### Materialized task-specific full subsets

Three full configurations are physically materialized for controlled
task-specific training; none is a symlink or hardlink. Each root preserves the
original config directory, Parquet shard names, Arrow schema, row order, and
embedded image bytes. A checksum-mode rsync comparison found zero differences
against every source shard. `SPATIAL_MMCOT_TASK_SUBSET.json` records provenance
and the answer audit in each root.

| Root | Config | Shards | Rows | Answer contract |
|---|---|---:|---:|---|
| `data/Zebra-CoT-Visual-Jigsaw` | 2D Visual Reasoning - Visual Jigsaw | 26 | 21,899 | 100% A--D visual multiple choice |
| `data/Zebra-CoT-Chess` | Visual Logic & Strategic Games - Chess | 8 | 20,483 | 100% explicit A--D; 20,480 raw answers unambiguous |
| `data/Zebra-CoT-Maze` | Visual Logic & Strategic Games - Maze | 11 | 20,000 | 9,000 explicit MC + 2,500 image-labeled MC + 8,500 numeric |

Each task uses a deterministic whole-group 90/5/5 split. Because every group
in these three configurations contains one row, the row and group counts are
identical and the requested ratios are exact after rounding:

| Task | Train | Validation | Test | Group leakage |
|---|---:|---:|---:|---:|
| Visual Jigsaw | 19,709 | 1,095 | 1,095 | 0 |
| Chess | 18,435 | 1,024 | 1,024 | 0 |
| Maze | 18,000 | 1,000 | 1,000 | 0 |

`SPATIAL_MMCOT_SPLIT.json` records the split method and counts. The canonical
membership lives in each task's `UniAR-960-cache/manifest.jsonl`; it reuses the
immutable packed tensors in `data/Zebra-CoT-UniAR-960` through absolute shard
paths instead of duplicating roughly 40 GiB of BSQ tensors. The raw Parquet
payload above remains physically materialized. Rebuild a task view with
`experiments/src/zebra_sft/task_split.py` at a fresh destination.

The current controlled Visual-Jigsaw BitFlip comparison uses the five
`OURS/config/sft/zebra_visual_jigsaw_control_*_bitflip_2gpu_1epoch.yaml`
configs. All use seed 17, the same split/order, rsLoRA, optimizer, one epoch,
50% stable noisy samples with 2% per-bit corruption, two GH200 ranks, an
eight-sample microbatch under a 32,768 padded-token budget, global batch up to
16, and per-step TensorBoard/JSONL logging. Only the IG enable flag and weights
vary: Vanilla; T->V/V->T `.005/.1`; `.005/1`; `.05/10`; and `.5/100`.
The 2-GPU task-2 smoke completed 10/10 finite updates, reproduced a 1.996%
noisy-group flip rate, peaked at 36.26 GiB allocated per GPU, and recorded
about 97% average utilization per GH200. The resulting full schedule is 1,312
updates over the same 19,709 train rows. Older four-A6000 configs and runs are
historical comparisons, not members of this new controlled matrix.

Within this project, unqualified "Vanilla SFT" denotes the IG-disabled member
with the stable 50% noisy-sample partition and 2% per-bit BSQ input corruption;
the no-corruption variant must be named explicitly.
The controlled `gradnorm5` follow-ups change the clipping threshold from 1.0
to 5.0 while preserving global batch 16 with single-GPU gradient accumulation.
The two-epoch half-sample variant changes only training duration; the
`all_samples` variant changes only the noisy-sample fraction to 1.0.

On DeltaAI, the controlled Vanilla and `.005/.1` IG runs completed with the
two-rank configs before the sequential bundle was stopped. The remaining three
weight settings and the four `event_selection: all` variants use explicit
`*_bitflip_1gpu_1epoch.yaml` configs: microbatch and padded-token budget remain
8/32,768, while two sequential microbatches are accumulated per optimizer
update. This preserves effective global batch 16 and the same 1,312-update
scheduler without DDP synchronization. All-events IG uses trajectory-balanced
reduction: events are averaged inside each trajectory before averaging the
contributing trajectories. The single-GH200 all-events smoke completed 10/10
finite updates, used 20 microbatches, and peaked at 36.28 GiB.

Native multi-rank SFT exposes `trainer.ddp_find_unused_parameters` and
`trainer.ddp_static_graph`; historical configs retain `true/false`. The v1
Tetris and Visual-Jigsaw base family uses `false/false`. Changing these flags
alone did not resolve historical failures. A confirmed Torch-2.7 interaction
was the custom dataclass returned by the SFT objective: DDP's output pytree
did not expose its loss to the static-graph backward sink, silently bypassing
gradient synchronization. `ObjectiveOutput` now uses a NamedTuple. Four-process
CPU regressions exercise the actual output type under normal, unused-parameter
discovery, and static-graph DDP, plus checkpoint/RNG/optimizer resume parity.
Pre-fix static-graph checkpoints must not be treated as valid distributed-SFT
results; recovery should return to the pre-static parent checkpoint.

SFT multi-rank control traffic is isolated from DDP gradients: NCCL carries
only reducer buckets, while a CPU Gloo group carries statistics, finite flags,
objects, barriers, validation aggregation, and checkpoint coordination. This
preserves wall-clock recovery checkpoints. Mixed NCCL gradient/control traffic
alone passed a 2,000-iteration four-GPU communication test; it is not by itself
proof of a defect. First-backward gradient and pre-checkpoint parameter hashes
now enforce replica equality using the Gloo group.

The default is `trainer.gradient_sync: ddp`: native backward bucket reduction
with `gradient_as_bucket_view=True`, no custom communication hook, and no
per-bucket CPU rendezvous. The v1 family explicitly uses DDP with
`find_unused_parameters=false`, `static_graph=false`, microbatch 4 per GPU,
and accumulation 1 on four GPUs (global batch 16). Existing sample membership,
loss normalization, BitFlip seeds, and optimizer schedules are preserved.
The explicit `post_backward` SUM path remains a diagnostic comparison only.

The supported stack is Torch 2.7.1+cu128 and torchvision 0.22.1. Torch 2.7.0
implicitly enabled nonblocking NCCL host APIs with eager `device_id`, causing
a main/watchdog race in `ncclCommGetAsyncError`/`pthread_join` on both clusters.
Torch 2.7.1 disables that implicit mode upstream
([PyTorch #153960](https://github.com/pytorch/pytorch/issues/153960)), so SFT now
uses ordinary process-group options. Keep `TORCH_NCCL_USE_COMM_NONBLOCKING`
and `NCCL_COMM_BLOCKING` unset on this stack. Tests use
`TORCH_DISTRIBUTED_DEBUG=OFF`; DETAIL changes synchronization and can mask
timing-sensitive defects. CUDA stream asynchrony and watchdog monitoring stay
enabled. Neither extra barriers nor wall-checkpoint removal repairs the race.

The v1 duration extension uses
`zebra_tetris_vanilla_{3,4,5,6}epoch.yaml`; these configs differ from the
Tetris vanilla base only in run identity and epoch count. The first
Visual-Jigsaw relative-IG weight extension uses
`zebra_visual_jigsaw_ig_lambda{2,4}_3epoch.yaml`, changing both directional
weights together while retaining three epochs and temperature 0.25.

The v2 Tetris base pair lives at
`OURS/config/sft/v2/zebra_tetris_{vanilla,ig}_base.yaml`. It preserves the v1
pair except for `run.version/name` and a constant LR schedule with 20 fixed
warmup updates. V2 long runs may request exact epoch-boundary checkpoints
without changing legacy checkpoint behavior.

The RPM v1 duration family is
`OURS/config/sft/v1/zebra_rpm_vanilla_{2,4,8}epoch.yaml`. It preserves the
Tetris v1 vanilla recipe (rank 32, half-sample uniform BitFlip up to 0.1,
IG/EMA off, clip norm 5, four-GPU DDP/global batch 16), using the existing
RPM cache's 2,700 train and all 150 validation rows. Only run name and epoch
count vary across that duration family. The controlled 4-epoch relative-IG
weight sweep is `zebra_rpm_ig_lambda{2,4,8}_4epoch.yaml`: it changes the run
identity/category, enables all-event `relative_per_unit` IG with equal T→V and
V→T weights, and enables validation IG statistics. Both temperatures remain
0.25. Data, uniform half-sample BitFlip, model, optimizer, schedule, clipping,
DDP, global batch 16, and all 150 validation rows match the 4-epoch vanilla
control. Each λ run starts independently from UniAR-SFT.
Each duration starts independently from UniAR-SFT with its own cosine schedule.
Internal validation runs at startup, every 100 updates and completion; it is
teacher-forced, not rollout accuracy.
Use validation for duration/IG selection and keep the 150-row test split for
the final comparison. These files do not by themselves submit Slurm jobs.

The first EMA-distillation control is
`zebra_visual_jigsaw_ema_nec_all_events_bitflip_full_gradnorm5_1gpu_1epoch.yaml`.
It keeps one epoch, full-sample 2% BitFlip, global batch 16, all events, and
`max_grad_norm=5`, while disabling IG so the only new objective is EMA
Necessity. The initial EMA settings are decay `.99`, ratio clip `5`, contrast
alpha `.5` in both directions, and T->V/V->T weights `.05/2.0`. Microbatch 2
with eight-step accumulation preserves global batch 16 on a 40-GiB A100. A
two-update GH200 smoke completed with 26.73 GiB peak allocation and roughly
1,000--1,070 sequence tokens/s; EMA update count exactly matched the two
successful optimizer updates. The formal task pairs one Delta training job with
an after-success full 1,095-row autonomous test job.

The controlled IG Temperature center is
`zebra_visual_jigsaw_ig_temperature_center_tv0p5_vt0p05_all_events_bitflip_full_gradnorm5_1gpu_1epoch.yaml`.
It changes the strong-weight all-events anchor only to full-sample BitFlip,
T->V/V->T temperatures `.5/.05`, `max_grad_norm=5`, and 200-update
validation/checkpoint boundaries; weights remain `.05/10`, EMA distillation is
explicitly disabled, and the original microbatch 8 / accumulation 2 schedule
retains all 19,709 rows and 1,312 updates. The matching historical run reached
50.29 GiB after its short smoke window, so Phase A uses a Delta `gpuA100x8`
device rather than an A100-40GB and does not change batch packing. Each
teacher-forced validation artifact includes trajectory-balanced gain and gate
distributions. `evaluate_task_subset --selection validation` owns the complete
1,095-row autonomous validation rollout and reports one-/four-image accuracy,
grid completion, image totals, and assistant text/visual token counts. The
`--sample-offset N --max-samples M` pair selects a deterministic manifest-order
window for short calibration runs while preserving each row's split-global
`evaluation_index`; shard merging validates those nonzero indices directly.
The
train-512 diagnostic accepts an exact prior subset file and a complete
`step-000200` checkpoint adapter, recording both subset and adapter fingerprints.
Temperature calibration also supports exact `--resume`: an atomic
`progress.json` commits complete batch byte boundaries, cumulative event hash,
sample IDs and counts; a torn JSONL tail is truncated to the last committed
batch, stable request/subset metadata is revalidated, and only pending cache rows
are loaded. Event output is buffered and flushed once per batch rather than once
per event. Because the fixed panel is globally length-sorted, calibration keeps
all of its mmap-backed shard mappings open instead of repeatedly thrashing a
two-shard LRU; tensor storage remains demand-paged.

DeltaAI training reads the independently rebuilt
`data/Zebra-CoT-Visual-Jigsaw/UniAR-960-cache-deltaai`. Its 21,899 sample IDs,
source coordinates, group IDs, split membership, sequence lengths, and visual
statistics exactly match the migrated task manifest; all 344 local shards are
present. The old task view is retained as provenance but its absolute
`/mnt/ultracube` shard paths are unavailable on this cluster.

The preliminary autonomous evaluation uses a fixed 100-row Visual-Jigsaw test
subset: 25 examples for each target choice A--D, selected independently by a
seeded SHA256 order. `evaluation_subset_index=0..99` in the task cache manifest
is the canonical cross-model join key; all non-members store `null`.
`SPATIAL_MMCOT_EVALUATION_SUBSET.json` records exact membership and provenance.
Rebuild the index with `select_evaluation_subset.py`; run two-rank native
understanding or unbounded interleaved evaluation with
`evaluate_task_subset.py`. The latter preserves exact token/event sequences,
generated BSQ codes, deterministic RGB decodes, strict-final scoring, and
per-sample artifacts for the rollout viewer.

Controlled Visual-Jigsaw comparisons use `--selection test`, which evaluates
the complete 1,095-row test split rather than the indexed 20/100-row qualitative
subsets. On one GH200, launch two evaluator processes with
`torchrun --nproc-per-node=2` plus `--workers-per-gpu 2 --scheduling dynamic`.
Both processes share the assigned GPU and atomically claim the next unfinished
sample, so variable rollout lengths do not create fixed-rank tail imbalance.
Autonomous evaluation also stops an individual text or scaffold phase when its
token suffix contains an exact period of at most 128 tokens repeated three
times and spanning at least 64 repeated tokens. The minimum span avoids
misclassifying short, legitimate enumerations as degeneration. This token-level
guard is separate from the existing complete-image-phase cycle detector,
records `cycle_period` with `cycle_unit=text_tokens`, and does not alter logits
or otherwise constrain non-cyclic generation.
The evaluator holds an advisory lock for the run directory for its full
lifetime; a second job targeting the same output must fail before touching the
shared queue or shards. Do not bypass this single-writer invariant.
The final merge rejects duplicate, missing, or provenance-mismatched rows and
supports exact resume from completed per-rank shards. If an interrupted legacy
run contains byte-identical duplicate commits, repair it with
`reshard_task_subset_eval.py --deduplicate-identical`; the command validates
all sample artifacts and preserves the original shards under `resume-backups/`
before atomically installing a clean shard set. A fixed six-trajectory
smoke matched single-worker token/event trajectories and generated BSQ tensors
exactly; its inference-phase GPU utilization rose from 36.9% for one worker to
81.7% for two, with 40.3 GiB peak memory.

For SIQ TIMEOUT retries, the identical evaluation command reuses the same run
directory without an explicit `--resume`: an existing run claim enables resume,
committed sample bundles are validated, an incomplete journal suffix is rolled
back, and all workers atomically rebuild one queue containing only unfinished
samples. Sample text/JSON/packed-code files are atomic and the JSONL commit is
fsynced after them; `summary.json` remains the single final accuracy result.

Task evaluation loads selected cache items through mmap-backed shard grouping
and a small dynamic-worker LRU instead of retaining nearly the entire mixed-split
cache in every process; the measured loader RSS fell from about 4.11 GB to
0.57 GB. Resume validates completed shards before model loading, loads only
pending items, and can publish an already-complete run without loading the 7B
model. Generated 0/1 BSQ artifacts use the versioned packed-bit schema
`zebra_generated_codes_packed_v1`; all readers remain compatible with historical
raw tensor lists. The largest measured four-image artifact shrank by 62.9x.
Standalone paired RGB/BSQ quality scoring is owned by `docs/visual_quality.md`;
it consumes completed eval artifacts but is not part of task evaluation.

Cross-phase language-KV reuse is intentionally **off by default**. Real GH200
FlashAttention2 diagnostic `3057393` kept the first eight greedy text tokens
exact but changed a subsequent visual bit because incremental-decode hidden
states were not bit-identical to a full-prefix language prefill. Production
rollout therefore retains phase-local language prefills; the experimental
single-cache path requires explicit `reuse_language_cache=True` and must not be
used for benchmark comparisons. Retroactive `mask_prior_thought` can never use
that cache because later K/V already encode the masked thought indirectly.
Likewise, GH200 two-token diagnostic `3057464` found that incremental caching
inside the four-layer visual decoder changed a generated bit relative to its
historical full-prefix call. `reuse_visual_decoder_cache` therefore also defaults
to false; its implementation remains an explicit experimental path only.

The Qwen3.8-27B direct-answer baseline uses the same 1,095 manifest rows through
`qwen3_5.eval_visual_jigsaw`. It reads only the verbatim `Question` events and
`problem_image_1`; `Text Reasoning Trace`, `reasoning_image_*`, and `Final
Answer` never enter the prompt. The source-positioned raw 2400x1800 image is
processed by vLLM with checkpoint defaults, followed only by an output-format
request for `Final answer: X`. Inference uses Qwen revision
`1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`, vLLM 0.28.0, and the checkpoint
chat template. The completed default-template run passed no Thinking-control
arguments, so this revision emitted its implicit xhigh instruction and
`<think>` prefix. The current comparison runner explicitly passes the official
template argument `reasoning_effort="medium"`; it does not add a custom system
prompt. `prompt_audit.json` stores
the rendered template plus a lossless run-length encoding and SHA256 of the
actual post-image-expansion prompt token IDs. Primary accuracy parses only the
post-`</think>` answer region, while explicit-`Final answer:` accuracy is also
reported. Launch through `run_visual_jigsaw_vllm.sh`; partial JSONL output is
resume-safe. Output generation has no per-request token cap and the engine uses
the checkpoint-native 262,144-token context; Delta runs therefore request four
40GB A100s so memory is not traded against the model context.

On this exact test100, native UniAR understanding scores **23/100**, the
one-epoch Vanilla interleaved SFT scores **50/100**, and the controlled
Information-Gain SFT scores **53/100**; every interleaved rollout terminates
with `im_end`. These are task-specific exact-choice results, not full Zebra-CoT
scores. The public rollout viewer exposes the first five indexed rows within
each answer choice (20 total), including exact assistant token sequences, all
generated states, and the GT trajectory. The full 100-row summaries remain the
accuracy source of truth; the balanced 20-row display is qualitative only.
The shared choice scorer treats a bare numeric answer such as `5` as equivalent
to `option 5`, just as bare `C` is equivalent to `option C`; this is required
for RPM, whose valid choices are numbered 1--8 and whose target wording varies.

Chess has three source-label anomalies: shard 0 row 1882 says `A. Qf5`
although Qf5 is option C; shard 2 row 2404 says `C. Rc8` although Rc8 is
option B; shard 1 row 1642 copies the complete option list instead of choosing
one. In all three cases the reasoning trace identifies the intended move. Keep
raw data immutable and either filter these rows from validation or apply an
explicit scorer-side repair.

Maze is therefore not entirely multiple choice. Its 8,500 free-response rows
remain deterministic and parser-friendly: connected-component count (2,000),
largest component size (2,000), total hole count (1,000), minimum steps (2,500),
and shortest-path count plus length (1,000). All 20,000 answers match one of the
audited discrete/templates, so task-specific exact scoring is practical.

The most relevant first-stage subset is all 39,610 3D examples:

| Configuration | Examples | Role |
|---|---:|---|
| 3D Multi-Hop Objects Counting | 10,000 | Synthetic action-conditioned object-state changes; best first pilot |
| 3D Embodied CoT | 22,666 | Initial/goal observations and intermediate embodied states |
| 3D Robot Planning | 6,944 | Initial/goal observations and intermediate manipulation states |

The remaining domains are useful for protocol diversity and general visual reasoning, but should be added only after the spatial pilot is understood.

## Local data audit

Audit performed against every row and Parquet metadata on 2026-08-19.

| Check | Result |
|---|---:|
| Rows | 182,384 |
| Problem-image references / physical images | 210,830 / 210,830 |
| Reasoning-image references / physical images after local repair | 718,866 / 718,869 |
| Blank question, trace, or answer | 0 |
| Trace without a reasoning image | 0 |
| Duplicate image reference | 0 |
| Duplicate/non-monotonic `THOUGHT n` ID | 0 |

Known packaging defects:

- The upstream consolidated Zebra-CoT `Scientific Reasoning - Graph Algorithms` release contains only `reasoning_image_1..9`; 1,802 rows reference 8,660 unavailable `reasoning_image_10..19`. This was not an incomplete clone: commit `f6d1defc169d19180a69f517925ed6cbb06a1c97` matched Hugging Face `main`. The local checkout has since been repaired as documented below.
- The authors' standalone `multimodal-reasoning-lab/Graph-Algorithms` repository at commit `020bc9d2a442d9989ba49344dd23c0bb4cfac90f` contains `reasoning_image_1..19`. Its seven shards have the same row counts, and all `Question`, `Text Reasoning Trace`, and `Final Answer` values are exactly equal in the same row order. Its columns 10–19 contain exactly 8,660 non-null images, so they can be joined unambiguously by shard name and row index without replacing the consolidated text or images 1–9.
- Geometry has 10 rows and Physics has 1 row that place `problem_image_1` inside the assistant reasoning trace.
- ARC-AGI has three physical reasoning images not referenced by its traces; ignore these extras.

### Local Graph repair

The shared Zebra checkout is now repaired in place. All seven consolidated
Graph Parquet files were replaced by the corresponding complete official
Graph-Algorithms shards after proving every pre-existing Arrow column was
value-identical row-by-row. The repaired shards contain
`reasoning_image_1..19`, retain the same 10,000 row order/source coordinates,
and expose 8,660 non-null images in the newly available columns 10--19.

Post-repair audit: 84,072 reasoning-image references, zero missing referenced
images, and target file SHA256 values exactly equal to the standalone official
repository at revision `020bc9d2a442d9989ba49344dd23c0bb4cfac90f`.

- Repair manifest: `/mnt/ultracube/datasets/Zebra-CoT/SPATIAL_MMCOT_REPAIR.json`
- Original-file backup: `/mnt/ultracube/datasets/Zebra-CoT-repair-backups/20260819-133848_graph-algorithms-original/`
- Reproducible repair: `experiments/src/zebra_sft/repair_graph_dataset.py`

The checkout is intentionally dirty; do not run a destructive Git reset. The
streaming loader detects the repaired schema and no longer requires the
standalone overlay, while remaining backward-compatible with an unrepaired
checkout.

The unpatched upstream release accepts 180,582 rows; the repaired local checkout accepts all 182,384. The 11 Geometry/Physics assistant images stored in `problem_image_1` columns are retained as generated states with explicit provenance rather than excluded. Do not silently truncate trajectories or reinterpret a missing physical image as text-only reasoning.

Image sizes and aspect ratios are heterogeneous: sampled images range from small crops such as 198x116 through 2400x1800 and 2085x2999. Training must use an explicit resolution/bucketing policy rather than the original dimensions blindly.

## Canonical trajectory contract

Apart from the documented Graph repair, do not edit the Parquet files. Convert each row to a model-independent typed trajectory:

```json
{
  "sample_id": "sha256:...",
  "source": {"dataset": "Zebra-CoT", "config": "..."},
  "user": [
    {"type": "text", "text": "..."},
    {"type": "image", "role": "observation", "key": "problem_image_1"}
  ],
  "assistant": [
    {"type": "text", "role": "thought", "text": "..."},
    {"type": "image", "role": "generated_state", "key": "reasoning_image_1"},
    {"type": "text", "role": "answer", "text": "..."}
  ]
}
```

The generic record stores semantics, not checkpoint-specific token strings. The UniAR collator renders each `generated_state` into the exact checkpoint protocol. This guarantees that observation images never accidentally receive a generation marker.

### UniAR rendering

Problem images are user observations and use the normal understanding path:

```text
<|im_start|>user
question text <|vision_start|><|image_pad|>...<|vision_end|>
<|im_end|>
```

A problem image is optional. Text-only questions that introduce one or more
assistant reasoning images are valid text-to-image/interleaved SFT examples;
only the assistant reasoning-image target is required. Do not discard these
Chemistry, Competitive Programming, Geometry, ARC-AGI, or Physics rows merely
because the user turn has no observation image.

Semantic role follows turn position, not the source column prefix. Eleven
released rows use `problem_image_1` for a diagram first referenced inside the
assistant trace; retain these as `generated_state`, insert `<image_gen>`, and
record the source-column mismatch in provenance.

Each assistant reasoning image is a generation target:

```text
<|im_start|>assistant
<think>thought before image 1</think>
<image_gen> generate image H W<|vision_start|>
GT_BSQ_CODES_1
<|vision_end|>
<think>thought before image 2</think>
<image_gen> generate image H W<|vision_start|>
GT_BSQ_CODES_2
<|vision_end|>
Final answer: ...
<|im_end|>
```

`H W` are visual-token grid dimensions, not pixel dimensions. With the current `patch_size=16` and `spatial_merge_size=2`, a 512x512 target is `16 16` and contains 256 autoregressive visual positions.

The `<image_gen>` marker is therefore required before every `reasoning_image_*`, not before any `problem_image_*`. Use the fast tokenizer loaded by `AutoProcessor` and assert that the literal marker encodes to exactly one ID. The checkpoint currently maps it to ID 151669 in that runtime; slow-tokenizer metadata instead reports 151675, so no numeric ID may be hard-coded in prepared data.

Do not add a new `<answer>` token in the first experiment. Use ordinary `Final answer:` text plus ChatML end-of-turn; this isolates the experiment to the already-required image boundary.

## Loss and model contract

[modeling.md](modeling.md) owns matched context views, bidirectional IG,
OPD divergences and RL rewards. Legacy auxiliaries and current execution
contracts belong to [uniar.md](uniar.md#training-runtime-contracts). In particular,
one UniAR visual prediction step is one composite token containing
`4 levels x 4 merge cells x 64 bits`; the bits factorize its likelihood but do
not count as 1,024 autoregressive time steps.

One teacher-forced mixed sequence should provide:

| Region | Supervision |
|---|---|
| System, user text, problem images | masked |
| Assistant thoughts and final answer | language CE |
| `<image_gen> generate image H W<|vision_start|>` | language CE |
| Reasoning-image BSQ codes | per-bit two-class CE through the visual prediction head |
| Text after a reasoning image | language CE conditioned on ground-truth BSQ features |

In canonical autonomous rollout, the student samples the complete
`<image_gen> generate image H W<|vision_start|>` scaffold and owns
`<|vision_start|>` as a language action/target. The controller inserts only
`<|vision_end|>` after exactly $H\times W$ visual positions. Grid sides are
validated in 1–30 only after the delimiter is sampled; default text, phase,
image, and visual-token horizons are context-bound and use shared short-cycle
detection rather than fixed caps.

Normalize text and visual losses separately before combining them; otherwise the many BSQ bit decisions dominate the objective. Log both losses and their gradient norms.

Freeze the BSQ encoder and SD3 visual decoder initially. Train the AR backbone with LoRA, the visual decoder/prediction head, and the `<image_gen>` input/output rows. Ordinary LoRA alone is insufficient because the current fast-token `<image_gen>` embedding is nearly untrained and the base checkpoint does not emit it reliably.

The released UniAR repository does not provide this SFT path. Its default forward computes language loss, while `is_vistok_pred=True` returns visual logits with `loss=None`; only image-generation GRPO training is public. The canonical reusable cache reader, objective, and trainer now live under `OURS/src/sft/` without importing or modifying `projects/UniAR/`; older Zebra-specific preparation and experiment runners remain under `experiments/`.

Canonical on-policy distillation is implemented separately under
`OURS/src/opd/`. It rolls out the current student from the cached $B_0$,
keeps sampled text and hard BSQ states in their exact serialized prefix, and
calls the binary answer verifier once. Event selection supports the first
complete visual-to-text window or all complete bidirectional transitions.
The eight supported routing/replay combinations are configured under
`OURS/config/opd/`; shared EMA, distribution, model, replay-SFT, scoring, and
artifact logic lives under `OURS/src/shared/` rather than being imported across
the SFT and OPD packages. Objective semantics remain owned by
`docs/modeling.md`.
## Cleaning and preparation plan
1. Stream Parquet rather than extracting all RGB files.
2. Overlay only `reasoning_image_10..19` for Graph Algorithms from the standalone repository, keyed by shard name and row index. Assert exact equality of the three text columns before accepting every shard, and record both repository revisions in provenance.
3. Validate non-empty fields, exact placeholder-to-column mapping, non-null image bytes, unique ordered references, and decodability.
4. Remove `THOUGHT n:` labels while preserving the text/image order; keep the original fields in provenance metadata.
5. Build stable IDs and group splits from configuration, normalized question, and problem-image content hashes. Split before turning one trajectory into windows so alternate annotations of the same problem cannot leak across splits.
6. Preserve aspect ratio and cap the long edge at 960 through the exact released qwen-vl-utils + UniAR processor path. Derive and assert each rendered `H W` from the resulting `image_grid_thw`; only lower the per-sample cap if the full sequence would exceed 32K.
7. Precompute frozen BSQ codes into a content-addressed shared cache. The prepared manifest refers to source Parquet coordinates and cache keys rather than duplicating JPEG bytes.
8. Keep full trajectories when they fit 32K. For an overflow, lower that trajectory's aspect-preserving cap through audited buckets; reject only if text plus minimum-resolution visuals still cannot fit.
9. Emit deterministic audit metrics and a small human-readable sample gallery before training.
## Historical pilot framework
The original pilot code is preserved under `experiments/src/zebra_sft/`:

- `prepare_samples.py`: exact raw Parquet fixture plus typed/normalized samples;
- `trajectory.py`: placeholder parsing and UniAR rendering;
- `dataset.py`: processor inputs, assistant-only language labels, and generated-image masks;
- `objective.py`: frozen BSQ encoding and joint language/visual teacher-forcing loss;
- `train.py`: LoRA overfit runner, BSQ cache, metrics, and adapter saving;
- `evaluate.py`: autonomous `<image_gen>` emission and teacher-forced BSQ decoding;
- `continue_after_visual.py`: generated-BSQ replay and text-head resumption;
- `audit_bsq_roundtrip.py`: RGB decode/re-encode and light-perturbation BSQ stability audit;
- `rollout.py`: complete repeated text/visual state machine;
- `evaluate_full.py`: all-step image decoding, protocol audit, and answer scoring;
- `evaluate_baseline.py`: constrained and native direct-answer baselines;
- `summarize_comparison.py`: paired metrics and per-sample comparison;
- `run_overfit16.sh`: canonical queue payload.

The inspectable fixture is `data/Zebra-CoT-samples/`:

- `raw/samples.parquet` preserves 16 exact source-format rows with embedded images;
- `raw/records.jsonl` exposes original text and image hashes;
- `prepared/trajectories.jsonl` uses `spatial_mmcot_trajectory_v1`;
- `prepared/images/` contains 96 aspect-preserving 256x256 PNGs;
- `prepared/rendered/` shows the exact semantic placement of every generation marker.

The current pilot freezes the BSQ tokenizer, caches its ground-truth grouped bits,
and applies LoRA to the language/visual decoder projections plus `embed_tokens`
and `lm_head`. There are 53,489,664 trainable parameters out of 9,680,563,696
(0.5525%). `<image_gen>` targets receive higher language-loss weight because
they are sparse but protocol-critical. The generation-intent switch is only
`<image_gen>`; `<|vision_start|>` remains a shared structural boundary for both
input observations and generated images.

### 16-trajectory overfit result

Run: `experiments/output/uniar-sft/zebra-cot-samples/unversioned/smoke/sft/20260818-222131_uniar_sft_zebra_overfit16/`.
Queue tasks: smoke `6da5630c`, train `eb5e82d4`, autonomous/decode eval
`6dfbb48c`, and post-image continuation `464ac353`. The 320-step run used one
queue-assigned RTX A6000 (GPU 5), took 6m40s, and peaked at 20.26 GiB.

| Metric | Base | Final |
|---|---:|---:|
| Total joint loss | 4.8585 | 0.1439 |
| Weighted text loss | 4.5908 | 0.00918 |
| Marker loss | 18.1007 | 0.00000685 |
| Text token accuracy | 58.70% | 99.75% |
| `<image_gen>` accuracy | 0/80 | 80/80 |
| Visual bit loss | 0.2677 | 0.1348 |
| Visual bit accuracy | 88.04% | 94.28% |

After adapter reload, all 16 prompts autonomously emitted `<image_gen>`. The
controller then generated and decoded a first visual state. Generated-BSQ replay
successfully resumed the next textual thought and emitted the next `<image_gen>`.
Teacher-forced decoded layout closely matches the target reconstruction; the
autonomous image preserves the main colors/layout but contains one extra brown
object. Therefore P0 confirms the SFT/control contract but not exact visual-state
memorization. See
`docs/research/20260818-223457_zebra_cot_uniar_sft_overfit16_report.md`.

### Complete-rollout result

Paired run:
`experiments/output/uniar-sft/zebra-cot-samples/unversioned/smoke/eval/20260818-231640_zebra_overfit16_full_vs_base/`.
The fine-tuned path repeatedly generates text until the model emits
`<image_gen>`, lets the controller add only `H W + vision_start`, greedily
generates 64 BSQ positions, directly replays those generated codes without RGB
re-encoding, and continues until ChatML end or a safety failure.

| Mode | Final answers |
|---|---:|
| Base UniAR-SFT, constrained direct answer | 4/16 (25%) |
| Base UniAR-SFT, native free-form answer | 4/16 (25%) |
| Fine-tuned complete interleaved rollout | 8/16 (50%) |

Only 11/16 fine-tuned trajectories complete the exact annotated interleave;
75/80 expected images are emitted. Mean autonomous visual-bit agreement is
78.06% and mean pixel PSNR is 17.47 dB. Agreement declines with rollout depth:
81.47%, 80.59%, 79.33%, 75.27%, 73.87%, 72.71%, 68.88%, and 68.21% for
steps 1–8 respectively (later steps have fewer examples).

Contact sheets show that generated scenes often preserve coarse color/layout
but fail the requested state change: removed objects persist, required objects
are absent, or empty scenes remain populated. Some answers remain correct despite
poor visual states, so this training-set gain cannot be attributed to useful
visual world modeling. The canonical paired table is `summary.md` in the run;
per-sample sequences and contact sheets are under
`finetuned_interleaved/<sample_id>/`. Full analysis:
`docs/research/20260818-234151_zebra_cot_full_rollout_vs_baseline_report.md`.

### Sequence-protocol audit

The canonical v2 Zebra serialization uses this system prompt for both SFT and
autonomous inference:

```text
Solve the problem using interleaved textual and visual reasoning. Wrap text reasoning in <think>...</think>. To generate a visual state, emit <image_gen> generate image H W<|vision_start|> and continue after <|vision_end|>. Conclude with "Final answer:" followed by the answer.
```

The assistant turn starts with the input-owned
`<|im_start|>assistant\n`. Each assistant event is separated by exactly one
newline. Text scaffolds through the student-emitted `<|vision_start|>`, visual
BSQ codes, the newline following the controller-owned `<|vision_end|>`, final
answer, and `<|im_end|>` are student targets; only `<|vision_end|>` is inserted
by the rollout controller. IG source masks remain semantic thought payloads or
visual payload positions and exclude their wrappers. The protocol identifier,
prompt, and source hashes are recorded in each cache `config.json`. Full-corpus
v2 artifacts live at `data/Zebra-CoT-UniAR-960-cache-v2`; older caches remain
immutable because their token sequences use the v1 prompt and whitespace.

Three independent audits of the UniAR paper, released code, Zebra raw data, and
project implementation agree on the following boundary:

- `generate image 8 8` is correct for the 256x256 pilot: `8 8` is the merged
  visual-token grid (`256 / (patch16 * merge2)`), not pixel dimensions;
- `<image_gen>` is the only generation-intent special token; `generate image`,
  spaces, and digits are ordinary text tokens, while `vision_start/end` are
  structural boundaries;
- Qwen image-pad expansion, image/grid order, M-RoPE, pre-norm visual head input,
  and causal visual-target shift match the released UniAR implementation;
- the complete topology does not match released UniAR SFT/inference: native T2I
  places marker/dimensions in the user turn and starts assistant visual output at
  `vision_start`; this project teaches assistant
  `<think> → <image_gen> → image → <think> → ...` as a new capability;
- current SFT also differs through marker-weighted assistant CE, 1:1 text/visual
  loss rather than the paper's 1:10, no random visual-bit flipping, generated-only
  visual loss, LoRA rather than full AR training, and 256-square letterboxing.

Before scaling, pin the fast tokenizer/ID, derive and assert H/W from actual
processor grids, decide whether controller-forced scaffold tokens should receive
language CE, add visual-noise/scheduled-sampling ablations, and repair data edges
(`THOUGHT N:`, adjacent-image spans, answer wrappers). Full audit:
`docs/research/20260819-001450_zebra_cot_uniar_sequence_protocol_audit.md`.

For token inspection use `experiments/src/zebra_sft/render_token_alignment.py`. It emits
fixed-width `RAW/TOK/ID` rows; blank RAW cells mark inserted protocol tokens.
Current raw mapping is event-level provenance for inspection, not yet a universal
character-offset provenance system for all 182K rows.

## Fine-tuning plan

### Historical full-corpus run implementation

The completed 2026-08-20 production run used `experiments/src/zebra_sft/`:

- `full_data.py`: strict row validation, standalone Graph overlay, grouped split, exact dynamic image grids, packed BSQ cache, and batching;
- `prepare_full.py`: four-GPU cache orchestration and corpus audit;
- `train_full.py`: BF16 rank-32 rsLoRA + full visual-head synchronous DDP, validation, checkpoints, and TensorBoard;
- `cached_rollout.py`: cache-native dynamic H/W modality-switching controller;
- `evaluate_corpus.py`: held-out baseline/interleaved comparison, trajectories, metrics, and image galleries.

Long cache builds submitted through SIQ use four workers plus
`--auto-resume --balanced-resume`. Each retry automatically starts fresh when
the output is absent, repartitions only unfinished source shards when partial,
or exits successfully when the atomic `READY` marker already exists.

The public Hub release has no validation or test split. The production cache
uses deterministic grouped train/validation/test assignment. Within each
configuration, hash-sorted groups reserve 1% test (32--256 groups) and 0.25%
validation (8--64 groups); the remainder trains. The final audit requires every
configuration in both held-out splits and zero group leakage. Full design:
`docs/research/20260819-012200_zebra_cot_full_sft_design.md`.

Current production queue chain: completed cache `04155db3`, guarded continuation
train `698f61d8`, held-out test `8c683c2f`. The continuation writes TensorBoard,
adapters, checkpoints, optimizer state, and validation metrics to
`experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/sft/20260820-054100_zebra_full960_resume2000_guarded/`.

The original train task `7b95ef57` is valid only through step 2,000. At step
2,001 its gradient norm became NaN; the unconditional optimizer update corrupted
the model, while the logger converted NaN metrics to zero. Recovery uses the
clean step-2,000 adapter, a fresh lower-LR optimizer, FP32 visual-head parameters
and moments, non-finite update skipping/recording, and a hard abort threshold.
Four-step DDP smoke task `537479fd` exactly reproduced step-2,000 validation and
completed steps 2,001--2,004 with zero non-finite updates.

### Final full-SFT and held-out rollout result

Guarded continuation task `698f61d8` completed global step 4,110. One inf-gradient
update at step 3,179 was safely skipped; no loss/parameter corruption recurred.
Fixed 36-sample validation changed from base to final as follows:

| Metric | Base | Final |
|---|---:|---:|
| Joint loss | 5.6757 | 2.7391 |
| Text accuracy | 58.99% | 84.88% |
| `<image_gen>` accuracy | 0% | 100% |
| Visual-bit accuracy | 87.73% | 89.79% |

Held-out task `8c683c2f` evaluated one low-compute complete trajectory from each
of the 18 configs. Base direct answers scored 1/18; fine-tuned autonomous
interleaving scored 6/18. Strict annotated protocol completion was 7/18, with
38/40 expected images generated. All 38 emitted images used syntactically valid
`<image_gen> generate image H W<|vision_start|>` scaffolds, but only 12/18
trajectories matched every annotated grid and only 10/18 reached ChatML end with
a final answer. Failures comprise five extra-image markers, three text-token
limits, and four emitted-grid mismatches (plus two no-image token-limit cases).

Across 33 generated steps whose grid matched the target, autonomous visual-bit
agreement averaged 80.27% (median 85.10%, range 55.42--96.52%). Contact sheets
show recognizable coarse appearance for natural images, but weak state-change
semantics: removed objects persist, manipulation states differ, and chemistry
diagrams can depict the wrong molecule. Therefore the answer gain is real on
this small diagnostic, but does not establish useful visual world modeling.

A relaxed retry raised the text-segment limit from 512 to 4,096 and allowed four
additional images for all eight initially incomplete samples. Five reached a
final answer: Physics and ARC-AGI resolved their text limits, while Robot
Planning, Connect Four, and Tetris completed with extra images. Geometry moved
from text truncation into an image loop; Competitive Programming and Graph
Algorithms also continued looping past the enlarged image budget. Relaxed
completion is therefore 15/18 overall. Image count/HW differences remain
reported diagnostics, not primary failure criteria; valid image scaffolds plus
a final answer define the relaxed format.

A final native-context rerun removed both artificial budgets: text and scaffold
generation may use the remaining 32,768-token model context, and generated-image
count is independent of the annotation. It produced 64 images, 17/18 trajectories
ended with a final answer and `im_end`, all emitted scaffolds were valid, and no
sequence ended with a dangling `<image_gen>`. Graph Algorithms continued from 6
to 7 images and Geometry from 5 to 10 before ending normally. Competitive
Programming instead entered an exact period-2 image/thought loop; after the full
period repeated three times it was recorded as `cycle_detected` after image 8 at
a complete `vision_end` boundary. This is semantic degeneration detection, not a
token or image-count budget. Automatic answer accuracy remains 6/18 and strict
annotated-grid protocol rises to 9/18. Canonical artifacts are under
`experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/eval/20260821-111254_zebra_native_context_test18/`.

A latent-copy audit compares each generated BSQ state with every same-grid
observation and annotated state. Of 49 comparable generated images, 29 (59.2%)
are nearest an observation; among 23 cases where both an observation and the
index-aligned GT are comparable, the generated state is closer to the
observation in all 23. Directly replaying the closest observation reaches
87.66% bit agreement with the GT, versus 85.90% for autonomous generation; the
generator beats this copy baseline in only 30.4% of cases. Consecutive generated
states average 96.28% agreement and 71.4% exceed 99%, exposing visual fixed-point
collapse. Robot Planning specifically emits `obs1 → obs1 → obs2`; its prompt
already contains both initial and goal images, enabling keyframe retrieval
instead of action-conditioned transition. Treat index alignment cautiously when
the model emits extra actions/images. Audit artifact:
`experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/eval/20260821-111254_zebra_native_context_test18/visual_copy_analysis.json`.

### Generated-visual-context ablation

The same 18 held-out test samples were rerun under six causal interventions at
the native 32,768-token limit, with no per-segment or image-count budget:
deterministic RGB noise encoded to BSQ, gray-128 encoded to BSQ, image-pad word
embeddings without BSQ/DeepStack, zero main/DeepStack visual features, empty
vision spans, and masking all prior completed text phases during later text
generation. The control and text-mask conditions use native autoregressive BSQ.

Strict automatic accuracy, gated by an explicit `Final answer:`, is respectively
`6/18 control, 7/18 noise, 9/18 gray, 3/18 image-pad, 4/18 zero-feature,
5/18 empty-span, 1/18 text-mask`. Noise/gray change T2 in 14/18 despite containing
no task evidence, and the model repeatedly narrates specific evidence from gray,
zero, or empty visual states. Their apparent score gains are perturbation/chance,
not improved visual reasoning. Masking old text while retaining native images
reduces normal completion from 17/18 to 1/18; 15/18 traces cycle and adjacent
generated states are nearly frozen (median same-size BSQ agreement 99.898%). The
supported conclusion is that visual state affects decoding and modality control,
but the text chain carries most rollout progress and generated images are not a
robust standalone cross-step memory.

`cached_rollout.py`, `ablate_rollouts.py`, and `analyze_ablations.py` own the
implementation, artifact validation, and paired analysis. Canonical artifacts:
`experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/ablation/20260823-170404_zebra_test18_visual_ablation_suite/`.
Full protocol, examples, scoring audit, and limitations:
`docs/research/20260823-173015_zebra_visual_context_ablation_report.md`.

### BSQ RGB round-trip pilot

A four-image Zebra fixture pilot at long-edge 384 and eight decoder steps finds
that `RGB -> BSQ -> decoded RGB -> BSQ` preserves only 71.68% of bits on
average; just 0.20% of 64-bit groups remain exactly equal. A second cycle
preserves 74.84% of the preceding codes. Applied to the first decoded RGB,
Gaussian noise at sigma `2/255`, JPEG quality 95, and 0.95x bicubic down/up
change respectively 10.85%, 9.88%, and 11.31% of re-encoded bits while keeping
the input RGB at 41.76, 42.84, and 45.03 dB PSNR. Contact sheets retain coarse
object/layout semantics but also expose texture and fine-geometry drift, so this
pilot establishes a lossy BSQ bottleneck, not semantic invariance or evidence of
a private channel. Any future grounding reward must still compare direct-BSQ
and rebuilt RGB paths behaviorally. Script, complete metrics, and limitations
are recorded in `docs/research/20260825-014710_bsq_rgb_roundtrip_pilot.md`.

### Official model membership audit

The public Zebra-CoT release exposes only `train` shards. The released Bagel
configuration feeds one `zebra_cot.jsonl` into its iterable dataset with
`num_used_data: None`, and the Anole card says it is fine-tuned on the corpus.
Our grouped 18-example split is consequently held out from our UniAR SFT only;
for released Bagel/Anole it is likely author-training membership, not an
author-held-out test. The paper does not release its claimed Anole
in-distribution test IDs/split or the IDs of its 512-question frontier-model
evaluation. We therefore run a labelled six-domain task-coverage proxy rather
than misrepresenting it as the hidden test set.

With native public serializations, Bagel/Anole score 8/20 and 13/20 on the fixed
external CV-Bench probe; on our grouped 18 their strict-final scores are 5/18
and 4/18, and on the six-domain proxy both are 0/6. The public viewer contains
both Zebra settings, all trajectories, native token representations, and
generated images. Protocol, revisions, and limitations:
`docs/research/20260824-122930_official_zebra_models_membership_and_cvbench_probe.md`.

The public rollout viewer exposes all 18 native-context runs alongside the old
512 and 4,096 variants, the seven-condition ablation, exact token IDs and masked
attention spans, input/GT images, and all decoded visual states:
`https://zebra-cot-uniar-rollouts.leckyelt.chatgpt.site`.

The cleaned unified TensorBoard excludes corrupted old steps 2,001--2,284,
merges valid steps 1--2,000 with guarded steps 2,001--4,110, and includes all
validation points, 18 output trajectories, eight relaxed retries, six contact
sheets, and final test scalars:

`experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/sft/20260820-054100_zebra_full960_resume2000_guarded/tensorboard_merged/`

The first full cache pass stopped after 166,706 committed rows because it
incorrectly rejected 7,718 legal text-only-question rows and a faster rank hit
the default 10-minute NCCL barrier timeout. The canonical preparation path now
accepts zero observation images, supports `--resume` from committed manifest
coordinates, uses collision-free appended cache shards, and gives preprocessing
ranks a 24-hour coordination timeout.

### Final full-corpus preparation result

Balanced resume task `04155db3` completed successfully. The final cache is
`data/Zebra-CoT-UniAR-960 -> /mnt/ultracube/datasets/Zebra-CoT-UniAR-960-v1`.

| Audit | Result |
|---|---:|
| Source / retained rows | 182,384 / 182,384 (100%) |
| Train / validation / test rows | 180,080 / 461 / 1,843 |
| Unique grouped problems | 166,713 |
| Group leakage | 0 |
| Text-only questions retained | 7,718 |
| Misnamed assistant-image rows retained with provenance | 11 |
| Discarded rows | 0 |
| Resolution cap | 960 for all rows |
| Mean / maximum sequence length | 2,116.06 / 29,415 |
| Mean / maximum visual positions | 1,608.35 / 18,000 |
| Cache size | 36 GiB |

The final manifest has 182,384 unique sample IDs and source coordinates. All
2,853 referenced cache shards exist; there are no missing/orphan shards or bad
per-shard index ranges. `stats.json` owns the complete per-config split audit.

The full autonomous-interleaved CV-Bench audit scores **77.37** on all 2,638
questions, versus **83.86** for the base checkpoint's released direct-answer
path. The loss is concentrated in 3D (88.50 -> 76.58; Depth 81.00, Distance
72.17). The adapter generates 2,759 images on 2,635 questions, with three
invalid H/W scaffolds and seven unparsed answers. This comparison still
confounds continued-training regression with generated-visual interference;
the same-adapter visual-disabled/direct-answer ablation remains required.
Artifacts and the four-task 5-sample visual/token gallery are owned by
`docs/cv_bench.md`.

## Current next experiments

### P0 — Causal coupling audit

- Build 200--500 machine-checkable one-step windows from Multi-Hop Objects,
  Robot Planning, games, and ARC.
- Run identical-prefix GT/generated/counterfactual/shuffled/null BSQ interventions,
  next-token/state probes, and the GT/generated text × visual-history 2×2.
- Compare greedy CFG=1 with temperature 0.1 at CFG 1, 2.0, and 2.5 using a
  correct unconditional history. Do not train another full-corpus model first.

### P1 — Targeted remediation

- If GT-prefix next-state generation fails, train action-conditioned one-step
  transitions with change-region and semantic-state supervision. Compare the
  current rsLoRA against full AR-backbone/visual-decoder/head tuning as a capacity
  ablation while keeping the BSQ tokenizer and SD3 decoder frozen.
- If counterfactual BSQ does not control subsequent text, add post-image state QA
  and paired counterfactual continuations before more long-trajectory SFT. A
  full-versus-visual-blind dependency term must be anchored by correct,
  state-consistent targets so it cannot create arbitrary visual dependence.
- If both one-step directions work but autonomous histories fail, train on
  detached hard-BSQ self-replay with clean-teacher-forced batches retained. Do not
  attach the original GT continuation to a semantically different sampled state:
  relabel it with a simulator/state parser, train explicit critique-and-repair, or
  mask the inconsistent downstream target.
- Treat the existing all-position/all-level bit CE as the baseline. Test a soft
  BSQ-feature/delta-aware auxiliary loss with changed-region weighting and a
  source-copy contrastive margin; plain full-frame reconstruction is not targeted
  to the observed fixed-point shortcut.
- Retain native understanding/QA replay data in every continued-training branch.

### P2 — World-model and transfer gate

- Prefer Stage 1 posed multi-view/action-conditioned data over more heterogeneous
  Zebra traces for establishing world-model competence.
- Re-enable longer multimodal rollouts only after transition, recognition, and
  causal-use metrics pass on held-out one-step windows.
- Test generated/GT/disabled visual inference and cross-task transfer before RL;
  RL should optimize when to imagine only after visual states are reliable.

## Required comparisons

1. Base UniAR-SFT.
2. Zebra text-only CoT SFT with identical questions/answers.
3. Zebra interleaved SFT with visual loss but forced boundary at inference.
4. Zebra interleaved SFT with autonomous `<image_gen>` boundary.
5. Spatial-only 3D Zebra versus full-domain Zebra.

Evaluate exact protocol compliance, visual bit loss, decoded state correctness, final-answer accuracy with generated images, ground-truth BSQ replay, and visual generation disabled. Retain CV-Bench as the native-understanding regression test. The Zebra-CoT paper reports that its Bagel fine-tune learned visual CoT but did not improve standard benchmarks and sometimes regressed because hallucinated images corrupted later reasoning; this makes the generated/ground-truth/disabled three-way comparison mandatory rather than optional.

Before further full-corpus training, run the causal coupling audit in
`docs/research/20260823-144502_zebra_cross_modal_coupling_diagnosis.md` on
machine-checkable one-step windows. The image-to-text comparison must additionally
inject valid counterfactual, shuffled, and null/copy BSQ states under an identical
text prefix and report next-token NLL plus answer/action sensitivity. GT replay is
an oracle bound, not evidence that the model causally uses visual state. Evaluate
the 2x2 of GT/generated prior text and GT/generated prior visual state to separate
textual drift, visual drift, and exposure bias. Visual evaluation must include
changed-region/action correctness and unchanged-region preservation; global BSQ
agreement alone is invalid because observation copying already reaches 87.66%.
The native-context `strict_protocol=9/18` is only an autonomous structural check
(normal end plus exact annotated grid sequence); it is not GT BSQ replay.
Xet Storage Details

Size:: 57.7 kB
Xet hash:: 1784bcd0caeff6a1ac5a2e5b67d15f96bd9fcd17c18fb2b4b9c7386f7f905b16
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.