File size: 2,740 Bytes
99d635b 467382e 99d635b 467382e 99d635b 467382e 99d635b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | # Scugnizz Llama-PCS Training (HF Jobs + DDP)
Parallel pretraining for **ScugnizzDecoder** (Llama-adjacent + PCS + RoPE) on Hugging Face Jobs.
## Quick start
```bash
export HF_TOKEN=hf_...
export OUT_DIR=/mnt/rope-v2-training-results/runs/my-run
export TARGET_TOKENS=1000000000
export FLAVOR=a100x4
export TOKENS_PER_STEP=262144
export MODEL_SIZE=1.7b
export HUB_REPO_ID=ProjectScugnizz/scugnizz-llama-pcs
export HUB_PATH=training-runs/my-run
# scripts from Hub (recommended for collaborators)
export SCRIPT_SOURCE=hub:ProjectScugnizz/scugnizz-llama-training
bash run-hf-job.sh
```
## Model sizes
| `MODEL_SIZE` | ~params |
|--------------|---------|
| `1b` | 1.0B |
| `1.7b` | 1.7B |
| `3b` | 3.0B |
Architecture must match the starting weights.
## Starting weights
Priority:
1. **`OUT_DIR/checkpoint_resume.pt`** or **`checkpoint_last.pt`** — full resume with optimizer
2. **`INIT_FROM`** — bucket/local path to `model_final.pt` or full checkpoint
3. **`INIT_HUB_REPO` + `INIT_HUB_FILE`** — download from Hub
4. **`OUT_DIR/model_final.pt`** — weights-only fallback
Examples:
```bash
# Continue an existing bucket run (automatic if checkpoints exist)
export OUT_DIR=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b
# Start from Hub weights
export INIT_HUB_REPO=ProjectScugnizz/scugnizz-llama-pcs
export INIT_HUB_FILE=training-runs/pretrain-fineweb-llama-pcs-550m/model_final.pt
export INIT_STEP=2100
# Start from bucket file
export INIT_FROM=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b/model_final.pt
export INIT_STEP=0
export NO_RESUME=1
```
## Multi-GPU (DDP)
Uses `torchrun` automatically when the job has >1 GPU. Set global throughput with:
```bash
export TOKENS_PER_STEP=262144 # auto-computes GRAD_ACCUM per GPU count
# or set GRAD_ACCUM manually per GPU
```
Monitor GPUs:
```bash
hf jobs stats ProjectScugnizz/<job_id>
```
## Checkpoints
**Keep** (optimizer resume): `checkpoint_resume.pt`, `checkpoint_last.pt`
**Safe to delete**: `checkpoint_weights_last.pt`
**I/O performance:** checkpoints write to `/tmp/scugnizz-ckpts` first; only resume/last copied to bucket. Defaults: `WEIGHTS_SAVE_INTERVAL=0`, `SAVE_INTERVAL=100`. Logs show `(N inst)` instantaneous tok/s.
```bash
export CKPT_LOCAL_DIR=/tmp/scugnizz-ckpts
export WEIGHTS_SAVE_INTERVAL=0
export SAVE_INTERVAL=100
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run
```
## Files
| File | Purpose |
|------|---------|
| `scugnizz-llama.py` | Model + training loop (DDP) |
| `job.sh` | In-job entrypoint (pip install + torchrun) |
| `run-hf-job.sh` | Parametrized HF Jobs submitter |
| `examples/` | Example launch configs |
|