social-data-attribution-w2/docs/COMPUTE_LEDGER.md

# Project Compute Ledger

This file is the project-level index for GPU and large compute accounting.
Keep detailed experiment notes in `notes/` or artifact bundles, but record the
rollup here so cost information is not lost during branch cleanup.

## Policy

- Do not delete compute ledgers only because an experiment was superseded.
- If a result is superseded, mark the note as historical and point to the
  current interpretation.
- Record raw GPU-hours and job counts separately from estimates.
- Mark CPU-only stages as CPU-only so they are not counted as GPU-hours.
- Do not sum rows unless their scopes are confirmed independent. Some rows
  below are subsets or alternate accounting views of the same run family.

## Required Fields For New Entries

| Field | Meaning |
| --- | --- |
| Date or period | When the compute ran or was measured |
| Scope | Experiment, branch, ticket, or artifact family |
| Source | `sacct`, cloud bill, sidecar stats, run logs, or estimate |
| Platform | PACE ICE, PACE Phoenix, RunPod, Modal, local GPU, or CPU |
| Raw GPU-hours | Wall clock x allocated GPUs |
| Job count | Slurm jobs, array tasks, cloud jobs, or local runs |
| Productive vs retry | Separate useful work from retry/debug overhead when known |
| Artifact pointer | Result directory, bucket prefix, or note path |

## Recorded GPU Usage

| Period | Scope | Platform | Source | Raw GPU-hours | Job count | Notes |
| --- | --- | --- | --- | ---: | ---: | --- |
| 2026-03-21 | SOC-119 500-doc/bin exploratory TrackStar run | PACE ICE | `docs/ATTRIBUTION_RUNBOOK.md` | 156.1 | not recorded here | Reduce: 0.51 GPU-h; score: 155.6 GPU-h. |
| 2026-05-23 to 2026-05-24 | N10 + N10b early ToM benchmark, attribution, unlearning, and eval attempts | PACE ICE | `notes/n10_gpu_hours.md` | 88.99 | 574 | Historical snapshot. Includes ~34 GPU-h rework/exploration overhead and ~55 GPU-h steady-state N10b attribution + unlearning estimate. |
| 2026-05-24 to 2026-05-25 | N10 ToM 24-probe x 24-topic attribution + unlearning gamma grid | PACE ICE | `notes/n10_gpu_ledger.md` | 590.9 | 10,479 | Historical snapshot. Includes ~330 GPU-h retry overhead; final grid coverage is 576/576 gamma cells. |
| 2026-06-09 to 2026-06-10 | ARC 5.68M unlearning on broken `+forget_ids_file` injection (challenge re-run + 3-variation easy/combined) | PACE ICE | run logs / `LOGBOOK.md` | 468.8 | 240 | SUPERSEDED by PR #117. Trained on random forget docs (unread Hydra key); ~123 GPU-h lost to a scratch-quota failure. Replaced by the corrected redo below; kept for honest accounting. |
| 2026-06-10 to 2026-06-11 | ARC 5.68M re-score: faithful per-doc + bin-level 3-variation redo + OLMES eval (PR #117, current) | PACE ICE | `sacct` | 492.5 | 561 | Productive ~417 GPU-h; ~75 GPU-h retry/lost (own-account mass-cancel, 2TB shared-volume fill, HX00 node faults, slow-node timeouts). Training (faith + blr) is exact at 454.8 GPU-h; eval (~37 GPU-h, `oevarr` arrays) shares the job-name with concurrent experiments so that portion is an upper bound. Artifacts: `staff/TDA/arc_rescore/`, HF bucket `base_rescore_20260608`. |
| 2026-06-12 to 2026-06-13 | W2 consensus document GPU smoke: two-row `forget_texts_file` training plus minimal RC eval | PACE ICE | `sacct` jobs 5390877, 5390880, 5390883, 5390891, 5390898, 5390900, 5390902, 5390912 | 0.74 | 16 | Productive 0.37 GPU-h for two train smoke tasks plus two minimal eval tasks; retry/debug 0.36 GPU-h from queue retargets, invalid null-forget smoke, unregistered SocialTDA aliases, oversized 79-task cloze retry, and stale-output retry. Artifacts: `findings/W2/out/gpu_smoke_inputs`, `gpu_smoke_runs/forgettexts_retry_*`, `gpu_smoke_eval/minimal_rc_retry`. |
| 2026-06-13 | W2 consensus document controlled 200-step GPU pilot plus SocialTDA MC eval retries | PACE ICE | `sacct` jobs 5390990, 5391021, 5391022, 5392244, 5392278, 5392324, 5392368 | 11.12 | 28 | Productive 9.38 GPU-h for four 200-step adapters and metric-producing MC evals; retry/debug 1.74 GPU-h from canceled infeasible 5000-step attempt and eval wrapper/data retries. Final eval coverage 112/112 MC metrics. Artifacts: `findings/W2/out/gpu_eval_pilot200_mc_*`, `gpu_wave_runs/pilot200_*`, `gpu_wave_eval/socialtda_required_primary_pilot200_*`. |
| 2026-06-13 | W2 consensus document 1000-step target/control confirmation pilot | PACE ICE | `sacct` jobs 5392538, 5392539 | 39.47 | 8 | Productive bounded confirmation pass: four 1000-step adapters plus four MC eval arrays. Artifacts: `findings/W2/out/gpu_eval_pilot1000_mc_*`, `gpu_wave_runs/pilot1000_*`, `gpu_wave_eval/socialtda_required_primary_pilot1000_mc`. |

## Linked Detail Notes

- `notes/n10_gpu_hours.md`: early N10/N10b GPU-hour ledger.
- `notes/n10_gpu_ledger.md`: later 24-probe ToM grid GPU-hour ledger.
- `notes/n10b_unlearn_results.md`: historical four-probe N10b result note with
  pipeline cost estimate and caveats.
- `notes/n10_tom_unlearning_findings.md`: current N10 ToM interpretation and
  24-probe, 3-seed analysis summary.
- `docs/slurm_gpu_catalog.md`: broad Slurm GPU/walltime catalog and estimates.
- `docs/ATTRIBUTION_RUNBOOK.md`: SOC-119 completed-run details.

## Open Accounting Items

- Reconcile overlapping N10/N10b accounting rows before using them in a paper or
  grant-level total.
- Add RunPod, Modal, and any local GPU billing/export data when available.
- Keep future Slurm `sacct` snapshots with the note they support and add a
  rollup row here.