| --- |
| name: Manage disk proactively — prefer move-to-/shm over delete; record manifest |
| description: Disk-fill-up incidents are costly. Mitigate by uploading important to HF + MOVING (not deleting) intermediate files to /shm with a manifest of what went where. |
| type: feedback |
| originSessionId: 4037f43b-2133-46c6-84bd-02f7d454ec8b |
| --- |
| When a long-running session generates many GB of artifacts: |
|
|
| 1. **Upload important checkpoints to HF immediately** when achieved, don't accumulate on local disk: |
| - Final-quality oracles → `explcre/dnathinker-checkpoints/runs/<run_dir>/` |
| - Major arch-ablation checkpoints (e.g., midway-decision-point ones) |
| - Ground-truth motif targets / extracted labels (if costly to regenerate) |
| - The bundle scripts + README so others can use them |
|
|
| 2. **MOVE intermediate files to `/shm` instead of deleting them.** `/shm` is a symlink to `/dev/shm` (157 GB tmpfs). Always log every move so the user can restore. |
| - Quarantine area: `/shm/dnathinker_quarantine/<original-relative-path>` |
| - Manifest log: `/shm/dnathinker_quarantine/MANIFEST.tsv` — append `<ISO-timestamp>\t<source-abs-path>\t<dest-abs-path>\t<size-bytes>\t<reason>` per move. |
| - Order priority for moving (most-unneeded first): corrupt/partial ckpts, then obsolete intermediate eval_loss > final, then HF Trainer's `checkpoint-<step>/` dirs that duplicate human-named `ckpt_step<step>.pt`, then eval-output caches (FIMO TSVs). |
| - Only resort to actual `rm` when the file is truly known-recoverable AND the user explicitly approves (e.g., regenerable in minutes from versioned source). |
|
|
| **Why:** previous H100 disk fill killed an in-flight ablation (loss 5.45→4.57 was achieved but ckpt-2000 corrupted) and broke the Bash runtime itself. The user does not want a repeat — and prefers reversibility ("prefer not deleting things, and record what's moved, to where"). |
|
|
| **Volatility caveat:** `/dev/shm` is RAM-backed tmpfs — files there DO NOT survive container/host restart. Mention this to the user when moving any file that is not regenerable from HF or source. If they want durable moves, ask first; possible alternates: another large block-mounted volume, or just push to HF and delete. |
|
|
| **How to apply:** at every meaningful milestone (oracle done, ablation step landed, comparison done): (a) push final artifacts to HF, (b) MOVE previous intermediate copies to `/shm/dnathinker_quarantine/...` and append to the manifest. Never silently `rm` to free space. |
|
|
| **Move helper boilerplate:** |
| ```bash |
| QUAR=/shm/dnathinker_quarantine |
| MANIFEST=$QUAR/MANIFEST.tsv |
| mkdir -p "$QUAR" |
| move_to_shm() { |
| local src="$1" reason="$2" |
| local rel="${src#/}"; local dst="$QUAR/$rel" |
| mkdir -p "$(dirname "$dst")" |
| local sz; sz=$(stat -c %s "$src" 2>/dev/null || du -sb "$src" | cut -f1) |
| mv "$src" "$dst" && \ |
| printf "%s\t%s\t%s\t%s\t%s\n" "$(date -Iseconds)" "$src" "$dst" "$sz" "$reason" >> "$MANIFEST" |
| } |
| ``` |
|
|
| **HF upload boilerplate:** |
| ```python |
| from huggingface_hub import HfApi |
| api = HfApi() |
| api.upload_file(path_or_fileobj=local, path_in_repo=repo_path, |
| repo_id="explcre/dnathinker-checkpoints", repo_type="model") |
| ``` |
|
|
