# Scugnizz Llama-PCS Training (HF Jobs + DDP)

Parallel pretraining for **ScugnizzDecoder** (Llama-adjacent + PCS + RoPE) on Hugging Face Jobs.

## Quick start

```bash
export HF_TOKEN=hf_...
export OUT_DIR=/mnt/rope-v2-training-results/runs/my-run
export TARGET_TOKENS=1000000000
export FLAVOR=a100x4
export TOKENS_PER_STEP=262144
export MODEL_SIZE=1.7b
export HUB_REPO_ID=ProjectScugnizz/scugnizz-llama-pcs
export HUB_PATH=training-runs/my-run

# scripts from Hub (recommended for collaborators)
export SCRIPT_SOURCE=hub:ProjectScugnizz/scugnizz-llama-training
bash run-hf-job.sh
```

## Model sizes

| `MODEL_SIZE` | ~params |
|--------------|---------|
| `1b`         | 1.0B    |
| `1.7b`       | 1.7B    |
| `3b`         | 3.0B    |

Architecture must match the starting weights.

## Starting weights

Priority:

1. **`OUT_DIR/checkpoint_resume.pt`** or **`checkpoint_last.pt`** — full resume with optimizer
2. **`INIT_FROM`** — bucket/local path to `model_final.pt` or full checkpoint
3. **`INIT_HUB_REPO` + `INIT_HUB_FILE`** — download from Hub
4. **`OUT_DIR/model_final.pt`** — weights-only fallback

Examples:

```bash
# Continue an existing bucket run (automatic if checkpoints exist)
export OUT_DIR=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b

# Start from Hub weights
export INIT_HUB_REPO=ProjectScugnizz/scugnizz-llama-pcs
export INIT_HUB_FILE=training-runs/pretrain-fineweb-llama-pcs-550m/model_final.pt
export INIT_STEP=2100

# Start from bucket file
export INIT_FROM=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b/model_final.pt
export INIT_STEP=0
export NO_RESUME=1
```

## Multi-GPU (DDP)

Uses `torchrun` automatically when the job has >1 GPU. Set global throughput with:

```bash
export TOKENS_PER_STEP=262144   # auto-computes GRAD_ACCUM per GPU count
# or set GRAD_ACCUM manually per GPU
```

Monitor GPUs:

```bash
hf jobs stats ProjectScugnizz/<job_id>
```

## Checkpoints

**Keep** (optimizer resume): `checkpoint_resume.pt`, `checkpoint_last.pt`

**Safe to delete**: `checkpoint_weights_last.pt`

**I/O performance:** checkpoints write to `/tmp/scugnizz-ckpts` first; only resume/last copied to bucket. Defaults: `WEIGHTS_SAVE_INTERVAL=0`, `SAVE_INTERVAL=100`. Logs show `(N inst)` instantaneous tok/s.

```bash
export CKPT_LOCAL_DIR=/tmp/scugnizz-ckpts
export WEIGHTS_SAVE_INTERVAL=0
export SAVE_INTERVAL=100
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run
```

## Files

| File | Purpose |
|------|---------|
| `scugnizz-llama.py` | Model + training loop (DDP) |
| `job.sh` | In-job entrypoint (pip install + torchrun) |
| `run-hf-job.sh` | Parametrized HF Jobs submitter |
| `examples/` | Example launch configs |