# scripts/slurm/

These are cluster-specific SLURM `sbatch` templates, carried over from the five source repos
that were consolidated into this one (file names are prefixed by origin: `pretrain_*` from the
core pretraining repo, `tagger_*`, `syntax_*`, `meter_*`, `insc_*` from the four downstream
repos). They were written for and run on one specific SLURM cluster (GH200
nodes, apptainer containers, a particular SLURM account) and have been genericized just enough
to not contain that cluster's hardcoded account or absolute paths — they are **not**
drop-in-runnable on another cluster without editing.

Before submitting any of these, you MUST:

1. **Edit the SLURM account.** Every file has:
   ```
   #SBATCH -A YOUR_ACCOUNT   # EDIT: your SLURM account
   ```
   Replace `YOUR_ACCOUNT` with your own allocation/project account.

2. **Check the partition name.** Every file has:
   ```
   #SBATCH -p gpu   # EDIT: your GPU partition name
   ```
   `gpu` was this cluster's GPU partition name — yours may differ (`gpu-a100`, `mig`, etc).

3. **Check node/GPU counts and wall-time limits** (`--nodes`, `--gpus-per-node`, `-t`) against
   your own cluster's node shapes and queue limits — these were tuned for 4x GH200/node.

4. **Set `STOICHEIA_SIF`** (or otherwise adapt the `apptainer exec --nv $SIF ...` calls) to your
   own container image — the original NGC-based training container is not included in this
   repo. If you're not using a container at all, replace the `apptainer exec ...` wrapper with
   a plain shell invocation of the same inner command.

5. **Set `STOICHEIA_DATA`** in your login-shell environment (or export it via `--export` on
   `sbatch`) before submitting — every script assumes `$STOICHEIA_ROOT/env.sh` (sourced inside
   the job) can resolve `STOICHEIA_DATA` for shard/checkpoint/run paths.

6. **Submit from the repo root, with a `logs/` directory already there** (`mkdir -p logs`).
   Every script's `#SBATCH -o`/`-e` is a relative path (`logs/<name>-%j.out`) because
   `#SBATCH` directives are parsed statically by `sbatch` *before* the script body runs —
   a script-computed variable like `$STOICHEIA_ROOT` is never expanded there. `sbatch`
   resolves a relative `-o`/`-e` against the directory you ran `sbatch` from, which is
   reliable as long as that's the repo root and `logs/` exists first (`sbatch` does not
   create it, and job launch fails outright if it doesn't exist).

7. **`STOICHEIA_DATA` must be exported in the *submitting* shell, or passed via `--export`**
   — `env.sh` has no default for it and sourcing fails outright without one. Note that a
   job whose apptainer step runs with an empty `$SIF` will interpret the next token on the
   command line (e.g. the literal word `bash`) as the image path and fail near-instantly
   with a cryptic "could not open image .../bash" error — check `STOICHEIA_SIF` first if
   you see that.

These templates were genericized from cluster-specific originals; wall-time budgets, node
counts, and topology-specific tuning (rendezvous ports, node-local staging in
`stage_shards.sh`, etc.) were deliberately left as used and should be reviewed before
running on a different cluster.