fast local checkpoints, fewer bucket writes
Browse files
README.md
CHANGED
|
@@ -72,11 +72,16 @@ hf jobs stats ProjectScugnizz/<job_id>
|
|
| 72 |
|
| 73 |
## Checkpoints
|
| 74 |
|
| 75 |
-
**Keep** (optimizer resume): `checkpoint_resume.pt`, `checkpoint_last.pt`
|
| 76 |
|
| 77 |
-
**Safe to delete**: `checkpoint_weights_last.pt`
|
|
|
|
|
|
|
| 78 |
|
| 79 |
```bash
|
|
|
|
|
|
|
|
|
|
| 80 |
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run
|
| 81 |
```
|
| 82 |
|
|
|
|
| 72 |
|
| 73 |
## Checkpoints
|
| 74 |
|
| 75 |
+
**Keep** (optimizer resume): `checkpoint_resume.pt`, `checkpoint_last.pt`
|
| 76 |
|
| 77 |
+
**Safe to delete**: `checkpoint_weights_last.pt`
|
| 78 |
+
|
| 79 |
+
**I/O performance:** checkpoints write to `/tmp/scugnizz-ckpts` first; only resume/last copied to bucket. Defaults: `WEIGHTS_SAVE_INTERVAL=0`, `SAVE_INTERVAL=100`. Logs show `(N inst)` instantaneous tok/s.
|
| 80 |
|
| 81 |
```bash
|
| 82 |
+
export CKPT_LOCAL_DIR=/tmp/scugnizz-ckpts
|
| 83 |
+
export WEIGHTS_SAVE_INTERVAL=0
|
| 84 |
+
export SAVE_INTERVAL=100
|
| 85 |
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run
|
| 86 |
```
|
| 87 |
|