social-data-attribution-w2/docs/COMPUTE_LEDGER.md

Project Compute Ledger

This file is the project-level index for GPU and large compute accounting. Keep detailed experiment notes in notes/ or artifact bundles, but record the rollup here so cost information is not lost during branch cleanup.

Policy

• Do not delete compute ledgers only because an experiment was superseded.
• If a result is superseded, mark the note as historical and point to the current interpretation.
• Record raw GPU-hours and job counts separately from estimates.
• Mark CPU-only stages as CPU-only so they are not counted as GPU-hours.
• Do not sum rows unless their scopes are confirmed independent. Some rows below are subsets or alternate accounting views of the same run family.

Required Fields For New Entries

Field	Meaning
Date or period	When the compute ran or was measured
Scope	Experiment, branch, ticket, or artifact family
Source	sacct, cloud bill, sidecar stats, run logs, or estimate
Platform	PACE ICE, PACE Phoenix, RunPod, Modal, local GPU, or CPU
Raw GPU-hours	Wall clock x allocated GPUs
Job count	Slurm jobs, array tasks, cloud jobs, or local runs
Productive vs retry	Separate useful work from retry/debug overhead when known
Artifact pointer	Result directory, bucket prefix, or note path

Recorded GPU Usage

Period	Scope	Platform	Source	Raw GPU-hours	Job count	Notes
2026-03-21	SOC-119 500-doc/bin exploratory TrackStar run	PACE ICE	docs/ATTRIBUTION_RUNBOOK.md	156.1	not recorded here	Reduce: 0.51 GPU-h; score: 155.6 GPU-h.
2026-05-23 to 2026-05-24	N10 + N10b early ToM benchmark, attribution, unlearning, and eval attempts	PACE ICE	notes/n10_gpu_hours.md	88.99	574	Historical snapshot. Includes ~34 GPU-h rework/exploration overhead and ~55 GPU-h steady-state N10b attribution + unlearning estimate.
2026-05-24 to 2026-05-25	N10 ToM 24-probe x 24-topic attribution + unlearning gamma grid	PACE ICE	notes/n10_gpu_ledger.md	590.9	10,479	Historical snapshot. Includes ~330 GPU-h retry overhead; final grid coverage is 576/576 gamma cells.
2026-06-09 to 2026-06-10	ARC 5.68M unlearning on broken +forget_ids_file injection (challenge re-run + 3-variation easy/combined)	PACE ICE	run logs / LOGBOOK.md	468.8	240	SUPERSEDED by PR #117. Trained on random forget docs (unread Hydra key); ~123 GPU-h lost to a scratch-quota failure. Replaced by the corrected redo below; kept for honest accounting.
2026-06-10 to 2026-06-11	ARC 5.68M re-score: faithful per-doc + bin-level 3-variation redo + OLMES eval (PR #117, current)	PACE ICE	sacct	492.5	561	Productive 417 GPU-h; ~75 GPU-h retry/lost (own-account mass-cancel, 2TB shared-volume fill, HX00 node faults, slow-node timeouts). Training (faith + blr) is exact at 454.8 GPU-h; eval (37 GPU-h, oevarr arrays) shares the job-name with concurrent experiments so that portion is an upper bound. Artifacts: staff/TDA/arc_rescore/, HF bucket base_rescore_20260608.
2026-06-12 to 2026-06-13	W2 consensus document GPU smoke: two-row forget_texts_file training plus minimal RC eval	PACE ICE	sacct jobs 5390877, 5390880, 5390883, 5390891, 5390898, 5390900, 5390902, 5390912	0.74	16	Productive 0.37 GPU-h for two train smoke tasks plus two minimal eval tasks; retry/debug 0.36 GPU-h from queue retargets, invalid null-forget smoke, unregistered SocialTDA aliases, oversized 79-task cloze retry, and stale-output retry. Artifacts: findings/W2/out/gpu_smoke_inputs, gpu_smoke_runs/forgettexts_retry_*, gpu_smoke_eval/minimal_rc_retry.
2026-06-13	W2 consensus document controlled 200-step GPU pilot plus SocialTDA MC eval retries	PACE ICE	sacct jobs 5390990, 5391021, 5391022, 5392244, 5392278, 5392324, 5392368	11.12	28	Productive 9.38 GPU-h for four 200-step adapters and metric-producing MC evals; retry/debug 1.74 GPU-h from canceled infeasible 5000-step attempt and eval wrapper/data retries. Final eval coverage 112/112 MC metrics. Artifacts: findings/W2/out/gpu_eval_pilot200_mc_*, gpu_wave_runs/pilot200_*, gpu_wave_eval/socialtda_required_primary_pilot200_*.
2026-06-13	W2 consensus document 1000-step target/control confirmation pilot	PACE ICE	sacct jobs 5392538, 5392539	39.47	8	Productive bounded confirmation pass: four 1000-step adapters plus four MC eval arrays. Artifacts: findings/W2/out/gpu_eval_pilot1000_mc_*, gpu_wave_runs/pilot1000_*, gpu_wave_eval/socialtda_required_primary_pilot1000_mc.

Linked Detail Notes

• notes/n10_gpu_hours.md: early N10/N10b GPU-hour ledger.
• notes/n10_gpu_ledger.md: later 24-probe ToM grid GPU-hour ledger.
• notes/n10b_unlearn_results.md: historical four-probe N10b result note with pipeline cost estimate and caveats.
• notes/n10_tom_unlearning_findings.md: current N10 ToM interpretation and 24-probe, 3-seed analysis summary.
• docs/slurm_gpu_catalog.md: broad Slurm GPU/walltime catalog and estimates.
• docs/ATTRIBUTION_RUNBOOK.md: SOC-119 completed-run details.

Open Accounting Items

• Reconcile overlapping N10/N10b accounting rows before using them in a paper or grant-level total.
• Add RunPod, Modal, and any local GPU billing/export data when available.
• Keep future Slurm sacct snapshots with the note they support and add a rollup row here.