docs/artifacts.md

Experiment artifacts

Output resolution

Resolve the output root in this order:

1. An explicit command argument/configuration.
2. SPATIAL_MMCOT_OUTPUT_ROOT.
3. <repository>/experiments/output.

Canonical layout

Every run has one owner, a dataset or operational topic, an explicit version, a lifecycle category, a task, and its original timestamped basename:

experiments/output/<owner>/<dataset-or-topic>/<version>/<category>/<task>/YYYYMMDD-HHMMSS_<name>/

For model-specific work, the owner is the actual initialization-model slug: uniar-sft, bagel-7b-mot, thinkmorph-7b, latentum-base, or the corresponding Qwen/official-baseline model. Framework names do not add a redundant directory. UniAR trained on ThinkMorph belongs to uniar-sft/thinkmorph, not thinkmorph-7b. Determine identity from model/config/submission provenance, not the dataset name. Historical uniar-rl evidence remains distinct from the canonical uniar-sft runtime.

shared owns indivisible cross-model comparisons and presentation derivatives (such as site-rollouts and figures). ops owns cluster, environment and storage operations. A model-unknown legacy record stays under shared/unclassified with its provenance in the relocation manifest; do not infer its model from neighboring folders.

Preserve recorded versions such as v1, v2, v3, and RLv3. Missing versions use unversioned; do not invent an experimental version retrospectively. Keep each run, its _history and _continuations, together. Separate runs are not merged merely because their configurations look similar.

Example:

experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/sft/
└── 20260827-033935_zebra_visual_jigsaw_vanilla_bitflip_p0p02_half_1epoch/
    ├── adapter/
    ├── config.yaml
    ├── summary.json
    └── tensorboard/

The lifecycle categories have distinct meanings:

• runs: result-producing experiments and evaluations; preparation, running, and completion status live in metadata rather than changing the directory.
• sweeps: hyperparameter-search bundles and members, including prefix studies.
• smoke: runs declared as short probes, diagnostics, or launch checks.
• archive: failed, interrupted, duplicate, or superseded artifacts moved after the run ends. New jobs must never target this category.

Zebra dataset slugs include zebra-cot-full, zebra-cot-small, zebra-cot-visual-jigsaw, the first-1/20 prefix zebra-cot-visual-jigsaw-1of20, and the 16-row fixture zebra-cot-samples. Tasks include sft, opd, rl, eval, probe, diagnostics, and cache.

Use artifacts.timestamped_run_name() from experiments/src/artifacts.py. Names use UTC-7 by default and remain descriptive after their timestamp. Keep generated media, decoded responses, effective configuration, metrics, scheduler identifiers, and logs from the same run together. Stable content-addressed caches are exempt; they are shared infrastructure.

New Slurm runs are created at submission time with an immutable _submission/ source/launch snapshot. Claim only a valid prepared run; see docs/submission_snapshots.md. Existing artifacts require an explicit user cleanup/relocation request; no implicit overwrite, truncation, or relocation.

Relocation and historical evidence

experiments/output/layout-map.json records reviewed old-to-new run paths. The detailed migration audit is below ops/storage/unversioned/runs/migration/. Relocation preserves checkpoint bytes, metrics, raw logs, saved configs and frozen _submission files. Paths inside those historical records remain provenance, not executable instructions for the current layout. Current source/config/documentation references use the new paths. Resolve an old archival path through the map instead of recreating old directory aliases.

A historical frozen launcher must not be resubmitted unchanged after its run moves. Prepare an explicit parent-linked relocated continuation, preserving mathematical configuration and checkpoint contents, and verify the new source/ input paths using docs/submission_snapshots.md. Do not rewrite a frozen snapshot, serialized optimizer state, or integrity hashes to hide a move.

Bucket archive and local working set

The current cluster root is /work/nvme/bgjw/tcheng2/MMCoT. Applications and default caches are its siblings ../apps/ and ../.cache/; none should point back to the retired BGOE NVMe workspace. The private sibling ../.env is loaded by zshrc/bashrc on each cluster and exports credentials without placing an API_KEY file in a source checkout.

The canonical bucket is hf://buckets/EmergentTwilight/MMCoT. A sparse local working set is not the inventory of everything retained remotely. Daily uploads are add/update-only, scoped to real files; never apply a whole-workspace hf sync --delete from a sparse checkout. Remote deletion requires a separate, explicit user request and a reviewed exact-target deletion plan.

Upload only committed checkpoint bundles (their complete inventory and commit marker), or a quiesced run. Verify remote contents against the upload manifest before releasing any local checkpoint. Keep pinned/active recovery state and enough free space for the next checkpoint plus staging. A missing local file is not a request to delete its cloud copy.

Remote references are explicit metadata, never text files masquerading as .pt, .bin, or .safetensors. Hydrate and verify the entire required bundle before checkpoint discovery/resume; publish its local COMPLETE marker last. Credentials belong to the ignored local workspace .env, exported by the shell; exclude .env, API_KEY, tokens and private keys from public archives.

For daily maintenance, first stop or quiesce writers to the selected bundle, upload its complete file inventory, and compare remote logical sizes and content identities. Evict only that verified, unpinned bundle after checking that no running or queued job needs it. Keep space for one new checkpoint and one hydration staging directory. Directory existence, a metrics file, or a partially downloaded adapter is not proof that weights are locally available.

Hydrate into a separate staging directory on the same filesystem, using the remote archive inventory as the authority. Verify every required file before an atomic promotion, with COMPLETE made visible last. Historical launchers still require a parent-linked relocated continuation as described above. Do not create placeholder tensor files or commit markers for cloud-only data.

The one-time BGOE-to-BGJW archive replaces the old Bucket inventory under the user's explicit migration instruction. This is not permission to repeat that deletion during daily synchronization. The regular .hf-sync-filter excludes Sites and Git metadata. The one-time full archive includes Sites, but Git metadata stays local: exclude every .git directory or .git file, including nested upstream repositories. Credentials and generated caches remain excluded.