Time Series Forecasting
PyTorch
English
pluvium
weather
weather-forecasting
climate
meteorology
earth-science
precipitation
rainfall
extreme-rainfall
nowcasting
probabilistic-forecasting
tropical-cyclones
hurricane
multi-model-ensemble
post-processing
online-learning
geospatial
caribbean
naturev1
ihelix
graphcast
aifs
Pluvium: code, model card, papers and benchmarks
Browse files- CHANGELOG.md +15 -0
- docs/MODULES.md +4 -4
- docs/TRAINING.md +7 -1
- notebooks/pluvium_blend_cell.py +1 -1
- notebooks/pluvium_everything.py +1 -1
- pluvium/__init__.py +1 -1
- pluvium/program.py +138 -11
- pluvium/training/ema.py +7 -1
- pyproject.toml +1 -1
- tests/test_fit.py +34 -2
- tests/test_program.py +51 -0
CHANGELOG.md
CHANGED
|
@@ -1,5 +1,20 @@
|
|
| 1 |
# Changelog
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
## 0.5.5 (November 2026)
|
| 4 |
|
| 5 |
The third GPU run trained, then stopped on a missing file. Its log was on the Hub this time, so the cause is known.
|
|
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
+
## 0.5.6 (November 2026)
|
| 4 |
+
|
| 5 |
+
The GPU run trained on molab for 11.6 hours, to step 8,625, and its best held-out loss was 0.734. Then two things
|
| 6 |
+
stopped it.
|
| 7 |
+
|
| 8 |
+
- molab shuts a notebook down 12 hours after it starts and the run was planned for 24, so it was stopped while it
|
| 9 |
+
was still training, before it was scored and published. On molab the run now ends half an hour before the 12
|
| 10 |
+
hours are up, counted from when the notebook started, with 1.5 hours kept for scoring and publishing.
|
| 11 |
+
`PLUVIUM_SESSION_HOURS` sets the limit on other machines.
|
| 12 |
+
- Run again, it went on from step 8,625 and stopped at the next step with "Expected all tensors to be on the same
|
| 13 |
+
device". The checkpoint is read onto the CPU and the averaged weights stayed there while the model was on the
|
| 14 |
+
GPU. They are put on the model's device now.
|
| 15 |
+
- The three restarts after that came back to the same error, each with less of the card. Only a stop that has to
|
| 16 |
+
do with memory makes the next try smaller now. Any other error is tried again at the same size.
|
| 17 |
+
|
| 18 |
## 0.5.5 (November 2026)
|
| 19 |
|
| 20 |
The third GPU run trained, then stopped on a missing file. Its log was on the Hub this time, so the cause is known.
|
docs/MODULES.md
CHANGED
|
@@ -114,7 +114,7 @@ Every module of the package, what it is for, and its size. Generated from the mo
|
|
| 114 |
| `pluvium/prob/extremes.py` | 91 | The tail: cloudbursts and their return periods. |
|
| 115 |
| `pluvium/prob/scores.py` | 88 | Scores that a single mean error hides. |
|
| 116 |
| `pluvium/products.py` | 192 | From the heads' raw outputs to what a person asks: will it rain here, when will it start, how hard, will there be thunder, how long will it last, how much will fall, and why. |
|
| 117 |
-
| `pluvium/program.py` |
|
| 118 |
| `pluvium/quick.py` | 617 | From nothing to a forecast in one call. |
|
| 119 |
| `pluvium/render/__init__.py` | 16 | Forecasts in the forms people and agencies use: |
|
| 120 |
| `pluvium/render/alerts.py` | 77 | Alerts in the Common Alerting Protocol (OASIS CAP 1.2). |
|
|
@@ -156,12 +156,12 @@ Every module of the package, what it is for, and its size. Generated from the mo
|
|
| 156 |
| `pluvium/tails/twcrps.py` | 45 | Scoring the tail on its own: the threshold-weighted CRPS. |
|
| 157 |
| `pluvium/tails/withheld.py` | 189 | What happens when the storm is bigger than anything in training? |
|
| 158 |
| `pluvium/timing.py` | 234 | When in the day it rains: the first three days of the forecast, split into six-hour periods. |
|
| 159 |
-
| `pluvium/train.py` |
|
| 160 |
| `pluvium/training/__init__.py` | 23 | How the network is taught, beyond the loss: |
|
| 161 |
| `pluvium/training/augment.py` | 53 | Perturbations with a physical meaning, applied to training examples. |
|
| 162 |
| `pluvium/training/curriculum.py` | 58 | Showing the network the rare events. |
|
| 163 |
| `pluvium/training/distill.py` | 38 | Distillation: a small Pluvium that forecasts like the large one. |
|
| 164 |
-
| `pluvium/training/ema.py` |
|
| 165 |
| `pluvium/training/fit.py` | 228 | Sizing a training step to the GPU's memory. |
|
| 166 |
| `pluvium/upstream/__init__.py` | 22 | The published weights of the AI weather models whose forecasts Pluvium reads, and how to keep a copy of them. |
|
| 167 |
| `pluvium/upstream/fetch.py` | 215 | Copy a published set of weights to a local folder, unchanged, with its licence and a record of where it came from. |
|
|
@@ -172,4 +172,4 @@ Every module of the package, what it is for, and its size. Generated from the mo
|
|
| 172 |
| `pluvium/weather.py` | 283 | What the weather *is*, in the words people use: intensity classes, drizzle versus rain versus showers, thunder. |
|
| 173 |
| `pluvium/zarrlite.py` | 166 | A minimal reader for consolidated Zarr v2 stores over HTTPS (the public ERA5 stores on Google Cloud). |
|
| 174 |
|
| 175 |
-
167 modules, 25,
|
|
|
|
| 114 |
| `pluvium/prob/extremes.py` | 91 | The tail: cloudbursts and their return periods. |
|
| 115 |
| `pluvium/prob/scores.py` | 88 | Scores that a single mean error hides. |
|
| 116 |
| `pluvium/products.py` | 192 | From the heads' raw outputs to what a person asks: will it rain here, when will it start, how hard, will there be thunder, how long will it last, how much will fall, and why. |
|
| 117 |
+
| `pluvium/program.py` | 1062 | Pluvium-500 from nothing on one GPU, in stages that each finish or resume: |
|
| 118 |
| `pluvium/quick.py` | 617 | From nothing to a forecast in one call. |
|
| 119 |
| `pluvium/render/__init__.py` | 16 | Forecasts in the forms people and agencies use: |
|
| 120 |
| `pluvium/render/alerts.py` | 77 | Alerts in the Common Alerting Protocol (OASIS CAP 1.2). |
|
|
|
|
| 156 |
| `pluvium/tails/twcrps.py` | 45 | Scoring the tail on its own: the threshold-weighted CRPS. |
|
| 157 |
| `pluvium/tails/withheld.py` | 189 | What happens when the storm is bigger than anything in training? |
|
| 158 |
| `pluvium/timing.py` | 234 | When in the day it rains: the first three days of the forecast, split into six-hour periods. |
|
| 159 |
+
| `pluvium/train.py` | 987 | Training on one GPU: batches built on every CPU core, bf16 on the GPU, a step sized by measuring to fill the GPU's memory (:mod:`pluvium.training.fit`), a learning-rate schedule that ends when the time budget does, checkpoints that resume exactly, and (optionally) a mirror of every checkpoint on the Hugging Face Hub so a rented machine that stops loses nothing. |
|
| 160 |
| `pluvium/training/__init__.py` | 23 | How the network is taught, beyond the loss: |
|
| 161 |
| `pluvium/training/augment.py` | 53 | Perturbations with a physical meaning, applied to training examples. |
|
| 162 |
| `pluvium/training/curriculum.py` | 58 | Showing the network the rare events. |
|
| 163 |
| `pluvium/training/distill.py` | 38 | Distillation: a small Pluvium that forecasts like the large one. |
|
| 164 |
+
| `pluvium/training/ema.py` | 56 | Polyak averaging of the weights. |
|
| 165 |
| `pluvium/training/fit.py` | 228 | Sizing a training step to the GPU's memory. |
|
| 166 |
| `pluvium/upstream/__init__.py` | 22 | The published weights of the AI weather models whose forecasts Pluvium reads, and how to keep a copy of them. |
|
| 167 |
| `pluvium/upstream/fetch.py` | 215 | Copy a published set of weights to a local folder, unchanged, with its licence and a record of where it came from. |
|
|
|
|
| 172 |
| `pluvium/weather.py` | 283 | What the weather *is*, in the words people use: intensity classes, drizzle versus rain versus showers, thunder. |
|
| 173 |
| `pluvium/zarrlite.py` | 166 | A minimal reader for consolidated Zarr v2 stores over HTTPS (the public ERA5 stores on Google Cloud). |
|
| 174 |
|
| 175 |
+
167 modules, 25,197 lines.
|
docs/TRAINING.md
CHANGED
|
@@ -65,7 +65,7 @@ def sh(cmd, check=True):
|
|
| 65 |
print(f"$ {cmd}", flush=True)
|
| 66 |
return subprocess.run(cmd, shell=True, check=check).returncode
|
| 67 |
|
| 68 |
-
sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.
|
| 69 |
root = os.path.expanduser(ROOT)
|
| 70 |
pluvium = f"{sys.executable} -m pluvium.cli"
|
| 71 |
if not os.path.exists(f"{root}/area.json"):
|
|
@@ -162,6 +162,7 @@ is unchanged up to a batch of 8 and grows with the square root of the batch abov
|
|
| 162 |
| `PLUVIUM_VRAM_SHARE=0.95` | The share of the card a step may need. The rest is a margin. |
|
| 163 |
| `PLUVIUM_VRAM=fastest` | The rule used before 0.4.3: keep the batch that is fastest per forecast. On a 90 GB card that can leave most of the memory unused. |
|
| 164 |
| `PLUVIUM_BATCH=6` | A batch given by hand. Nothing is measured. |
|
|
|
|
| 165 |
|
| 166 |
A fuller card is not a faster one by itself. Forecasts per second stay about the same; each step holds more
|
| 167 |
forecasts and more cells, so there are fewer and steadier steps in the same time. If a step ever needs more than the
|
|
@@ -180,6 +181,11 @@ It is started again in the same way when it was killed (the machine ran out of R
|
|
| 180 |
(`PLUVIUM_STALL_MINUTES` changes that). The last 400 lines of every run that stopped are kept in
|
| 181 |
`logs/attempt-N.txt`, in the working folder and in the Hub repository, so the error can be read afterwards.
|
| 182 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
On an RTX PRO 6000 (96 GB), one Pluvium-500 forecast took 38.6 GB at its peak and trained at 0.24 forecasts a second,
|
| 184 |
so a step holds two forecasts at most and the rest of the card goes to more target cells.
|
| 185 |
|
|
|
|
| 65 |
print(f"$ {cmd}", flush=True)
|
| 66 |
return subprocess.run(cmd, shell=True, check=check).returncode
|
| 67 |
|
| 68 |
+
sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.6'")
|
| 69 |
root = os.path.expanduser(ROOT)
|
| 70 |
pluvium = f"{sys.executable} -m pluvium.cli"
|
| 71 |
if not os.path.exists(f"{root}/area.json"):
|
|
|
|
| 162 |
| `PLUVIUM_VRAM_SHARE=0.95` | The share of the card a step may need. The rest is a margin. |
|
| 163 |
| `PLUVIUM_VRAM=fastest` | The rule used before 0.4.3: keep the batch that is fastest per forecast. On a 90 GB card that can leave most of the memory unused. |
|
| 164 |
| `PLUVIUM_BATCH=6` | A batch given by hand. Nothing is measured. |
|
| 165 |
+
| `PLUVIUM_SESSION_HOURS=12` | How long one session of the machine may run. The run is published half an hour before the end. On molab it is 12 without being set, and 0 means no end. |
|
| 166 |
|
| 167 |
A fuller card is not a faster one by itself. Forecasts per second stay about the same; each step holds more
|
| 168 |
forecasts and more cells, so there are fewer and steadier steps in the same time. If a step ever needs more than the
|
|
|
|
| 181 |
(`PLUVIUM_STALL_MINUTES` changes that). The last 400 lines of every run that stopped are kept in
|
| 182 |
`logs/attempt-N.txt`, in the working folder and in the Hub repository, so the error can be read afterwards.
|
| 183 |
|
| 184 |
+
molab stops a notebook 12 hours after it starts. The program sees when it runs there and plans the run to be scored
|
| 185 |
+
and published half an hour before that, so on molab a run of `GPU_HOURS = 24` trains for about 10 hours in one
|
| 186 |
+
session. A run that is stopped anyway goes on from its last checkpoint when the cell is run again. On another machine
|
| 187 |
+
that stops its sessions, `PLUVIUM_SESSION_HOURS=12` says so.
|
| 188 |
+
|
| 189 |
On an RTX PRO 6000 (96 GB), one Pluvium-500 forecast took 38.6 GB at its peak and trained at 0.24 forecasts a second,
|
| 190 |
so a step holds two forecasts at most and the rest of the card goes to more target cells.
|
| 191 |
|
notebooks/pluvium_blend_cell.py
CHANGED
|
@@ -22,7 +22,7 @@ def sh(cmd: str, check: bool = True) -> int:
|
|
| 22 |
|
| 23 |
|
| 24 |
# 1. the package: from PyPI, or from the source tree when PyPI is not reachable
|
| 25 |
-
if sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.
|
| 26 |
sh(f"{sys.executable} -m pip install -q 'git+https://github.com/Wisk-inc/transformers.git"
|
| 27 |
f"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium'")
|
| 28 |
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
# 1. the package: from PyPI, or from the source tree when PyPI is not reachable
|
| 25 |
+
if sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.6'", check=False) != 0:
|
| 26 |
sh(f"{sys.executable} -m pip install -q 'git+https://github.com/Wisk-inc/transformers.git"
|
| 27 |
f"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium'")
|
| 28 |
|
notebooks/pluvium_everything.py
CHANGED
|
@@ -93,7 +93,7 @@ def step(what: str, action, skip: str = "") -> bool:
|
|
| 93 |
# 1. the package: from PyPI, or from the source tree when PyPI cannot be reached
|
| 94 |
def install() -> None:
|
| 95 |
try:
|
| 96 |
-
run([sys.executable, "-m", "pip", "install", "-q", "--upgrade", "pluvium>=0.5.
|
| 97 |
except RuntimeError:
|
| 98 |
run([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/Wisk-inc/transformers.git"
|
| 99 |
"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium"])
|
|
|
|
| 93 |
# 1. the package: from PyPI, or from the source tree when PyPI cannot be reached
|
| 94 |
def install() -> None:
|
| 95 |
try:
|
| 96 |
+
run([sys.executable, "-m", "pip", "install", "-q", "--upgrade", "pluvium>=0.5.6"])
|
| 97 |
except RuntimeError:
|
| 98 |
run([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/Wisk-inc/transformers.git"
|
| 99 |
"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium"])
|
pluvium/__init__.py
CHANGED
|
@@ -11,7 +11,7 @@ whose forecasts Pluvium reads (:mod:`pluvium.upstream`).
|
|
| 11 |
See :mod:`pluvium.model` for the architecture and its experimental components.
|
| 12 |
"""
|
| 13 |
|
| 14 |
-
__version__ = "0.5.
|
| 15 |
|
| 16 |
from .domain import Gazetteer, Geography, PointSet # noqa: E402
|
| 17 |
from .model import PluviumConfig, PluviumModel # noqa: E402
|
|
|
|
| 11 |
See :mod:`pluvium.model` for the architecture and its experimental components.
|
| 12 |
"""
|
| 13 |
|
| 14 |
+
__version__ = "0.5.6"
|
| 15 |
|
| 16 |
from .domain import Gazetteer, Geography, PointSet # noqa: E402
|
| 17 |
from .model import PluviumConfig, PluviumModel # noqa: E402
|
pluvium/program.py
CHANGED
|
@@ -19,6 +19,8 @@ forecast the live forecast for every town, if the driving models ar
|
|
| 19 |
Everything expensive is mirrored to the Hugging Face Hub as soon as it exists (``HF_REPO``, ``HF_TOKEN``), so a
|
| 20 |
second session -- or a machine that was stopped -- pulls it back in minutes and continues where the last one ended.
|
| 21 |
The time budget (``GPU_HOURS``) is wall clock from the program's start: training gets whatever the other stages leave.
|
|
|
|
|
|
|
| 22 |
"""
|
| 23 |
|
| 24 |
from __future__ import annotations
|
|
@@ -100,11 +102,106 @@ def choose_years(gpu_hours: float, holdout: int = 2021) -> tuple:
|
|
| 100 |
return tuple(range(holdout - n, holdout))
|
| 101 |
|
| 102 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
class Program:
|
| 104 |
def __init__(self, cfg: ProgramConfig) -> None:
|
| 105 |
self.cfg = cfg
|
| 106 |
self.started = time.time()
|
| 107 |
self.deadline = self.started + cfg.gpu_hours * 3600.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
self.root = Path(cfg.root).expanduser()
|
| 109 |
self.root.mkdir(parents=True, exist_ok=True)
|
| 110 |
self.state_path = self.root / "program.json"
|
|
@@ -121,6 +218,12 @@ class Program:
|
|
| 121 |
"""Hours left in the budget."""
|
| 122 |
return (self.deadline - time.time()) / 3600.0
|
| 123 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
def mark(self, stage: str, **info) -> None:
|
| 125 |
self.state[stage] = {"done": time.strftime("%Y-%m-%d %H:%M"), **info}
|
| 126 |
self.state_path.write_text(json.dumps(self.state, indent=1))
|
|
@@ -364,6 +467,13 @@ class Program:
|
|
| 364 |
log(f"GPU: {props.name}, {props.total_memory / 2**30:.0f} GB ({how}); {os.cpu_count()} CPU cores")
|
| 365 |
torch.backends.cuda.matmul.allow_tf32 = True
|
| 366 |
torch.backends.cudnn.allow_tf32 = True
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 367 |
geo, towns = self.geography()
|
| 368 |
era5, cmorph, stations = self.data(geo, self.years + (cfg.holdout_year,))
|
| 369 |
clim = self.climatology(geo, cmorph)
|
|
@@ -385,7 +495,7 @@ class Program:
|
|
| 385 |
f"{test_starts.size} for scoring")
|
| 386 |
model = build_model(geo, grids, cfg.size, st_points, **cfg.overrides)
|
| 387 |
loss = PluviumLoss(states=model.config.states)
|
| 388 |
-
reserve =
|
| 389 |
train_hours_total = max(0.25, self.left() - reserve)
|
| 390 |
share_a = train_hours_total * (0.85 if self.stage_b else 1.0)
|
| 391 |
trainer = Trainer(model, loss, builder, train_starts, val_starts, self.root / "stage_a",
|
|
@@ -554,7 +664,7 @@ class Program:
|
|
| 554 |
return None
|
| 555 |
cut = int(starts.size * 0.75)
|
| 556 |
train_b, held_b = starts[:cut], starts[cut:]
|
| 557 |
-
hours_b = max(0.2, self.left() -
|
| 558 |
cfg_b = self.train_cfg(hours_b, warmup_steps=50)
|
| 559 |
cfg_b.lr = self.cfg.lr / 3
|
| 560 |
# stage B's forecasts are built differently, so its step is sized afresh, unless a batch was given by hand
|
|
@@ -809,6 +919,13 @@ def retry_plan(vram: str, share: float) -> list:
|
|
| 809 |
|
| 810 |
#: a run that prints nothing for this many minutes is taken to have hung; it is stopped and started again
|
| 811 |
STALL_MINUTES = 120.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 812 |
#: lines of a stopped run's output that are kept, on disk and on the Hub, so that its error can be read afterwards
|
| 813 |
TAIL_LINES = 400
|
| 814 |
_SECRET = re.compile(r"hf_[A-Za-z0-9]{20,}|pypi-[A-Za-z0-9_-]{20,}|gh[pousr]_[A-Za-z0-9]{20,}")
|
|
@@ -842,7 +959,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
|
|
| 842 |
token=None, stall_minutes: float | None = None, api=None) -> int:
|
| 843 |
"""Run the program in a child process and keep it going. If the child stops with an error, is killed (the
|
| 844 |
machine ran out of RAM, say), or prints nothing for ``stall_minutes``, it is started again from its last
|
| 845 |
-
checkpoint with a smaller step
|
|
|
|
|
|
|
| 846 |
which some GPU errors leave unusable; the data and the checkpoints are on disk, so the next try is training
|
| 847 |
again within minutes.
|
| 848 |
|
|
@@ -857,6 +976,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
|
|
| 857 |
tries = retry_plan(vram, share)
|
| 858 |
limit = 60.0 * float(stall_minutes if stall_minutes is not None else os.environ.get("PLUVIUM_STALL_MINUTES") or STALL_MINUTES)
|
| 859 |
state = {"child": None}
|
|
|
|
|
|
|
|
|
|
| 860 |
|
| 861 |
def end_child() -> None:
|
| 862 |
child = state["child"]
|
|
@@ -873,11 +995,12 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
|
|
| 873 |
raise SystemExit(128 + signum)
|
| 874 |
|
| 875 |
previous = signal.signal(signal.SIGTERM, stop)
|
| 876 |
-
code = 1
|
| 877 |
try:
|
| 878 |
-
for k
|
| 879 |
-
|
| 880 |
-
|
|
|
|
| 881 |
extra = ["--vram", rule, "--vram-share", str(cut)]
|
| 882 |
if batch:
|
| 883 |
env["PLUVIUM_BATCH"] = str(batch)
|
|
@@ -910,10 +1033,13 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
|
|
| 910 |
settings = f"{rule}, share {cut}" + (f", batch {batch}" if batch else "")
|
| 911 |
saved = save_attempt(root, repo, token, k + 1, f"attempt {k + 1} of {len(tries)} ({settings}): the program {what}",
|
| 912 |
list(tail), api=api) if root else None
|
|
|
|
| 913 |
code = code if code > 0 else 1
|
| 914 |
if k + 1 < len(tries):
|
| 915 |
-
|
| 916 |
-
|
|
|
|
|
|
|
| 917 |
f"with each training step sized to {then_cut:.0%} of the GPU's memory" if then == "fill" else
|
| 918 |
"with the batch rule of 0.4.2 (--vram fastest)")
|
| 919 |
log(f"the program {what}" + (f"; the end of its output is in {saved}" if saved else "")
|
|
@@ -926,8 +1052,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
|
|
| 926 |
signal.signal(signal.SIGTERM, previous)
|
| 927 |
|
| 928 |
|
| 929 |
-
__all__ = ["DEFAULT_REPO", "FILES_PER_COMMIT", "
|
| 930 |
-
"
|
|
|
|
| 931 |
|
| 932 |
# ``python -m pluvium.program`` is how the notebook cells start a training run. Until 0.4.3 these two lines were
|
| 933 |
# missing: the command imported this module and ended at once with exit code 0, having trained nothing.
|
|
|
|
| 19 |
Everything expensive is mirrored to the Hugging Face Hub as soon as it exists (``HF_REPO``, ``HF_TOKEN``), so a
|
| 20 |
second session -- or a machine that was stopped -- pulls it back in minutes and continues where the last one ended.
|
| 21 |
The time budget (``GPU_HOURS``) is wall clock from the program's start: training gets whatever the other stages leave.
|
| 22 |
+
On a machine that stops a session after a fixed time (molab: 12 hours) the budget ends before that, so the run is
|
| 23 |
+
scored and published while the session is still up.
|
| 24 |
"""
|
| 25 |
|
| 26 |
from __future__ import annotations
|
|
|
|
| 102 |
return tuple(range(holdout - n, holdout))
|
| 103 |
|
| 104 |
|
| 105 |
+
# ------------------------------------------------------------------------------------- sessions --
|
| 106 |
+
#: molab, where these runs are made, stops a notebook 12 hours after it starts ("Notebooks can run for as long as 12
|
| 107 |
+
#: hours before molab shuts them down", docs.marimo.io/guides/molab). The first full run there trained for 11.6 of
|
| 108 |
+
#: its 24 hours and was stopped before it was scored and published.
|
| 109 |
+
MOLAB_HOURS = 12.0
|
| 110 |
+
#: how long before a session's end the run means to be done
|
| 111 |
+
SESSION_MARGIN_HOURS = 0.5
|
| 112 |
+
#: the least time kept for calibration, scoring, publishing and the forecast when the session has an end
|
| 113 |
+
SESSION_RESERVE_HOURS = 1.5
|
| 114 |
+
#: where molab keeps a notebook's packages (the tracebacks of the runs there show it)
|
| 115 |
+
MOLAB_PACKAGES = "/tmp/uv-venv"
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def _proc_stat(pid: int) -> list:
|
| 119 |
+
"""The fields of ``/proc/<pid>/stat`` after the command name, so that field 3 (the state) is the first."""
|
| 120 |
+
return Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def _ancestors(most: int = 16) -> list:
|
| 124 |
+
"""This process and the ones above it, nearest first: the program, its supervisor, the notebook, ..."""
|
| 125 |
+
out, pid = [], os.getpid()
|
| 126 |
+
while pid > 0 and len(out) < most:
|
| 127 |
+
out.append(pid)
|
| 128 |
+
try:
|
| 129 |
+
pid = int(_proc_stat(pid)[1])
|
| 130 |
+
except (OSError, ValueError, IndexError):
|
| 131 |
+
break
|
| 132 |
+
return out
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def _began(pid: int) -> float | None:
|
| 136 |
+
"""When a process started, in seconds since 1970, or None where there is no ``/proc``."""
|
| 137 |
+
try:
|
| 138 |
+
boot = next(float(line.split()[1]) for line in Path("/proc/stat").read_text().splitlines()
|
| 139 |
+
if line.startswith("btime"))
|
| 140 |
+
return boot + int(_proc_stat(pid)[19]) / os.sysconf("SC_CLK_TCK")
|
| 141 |
+
except (OSError, ValueError, IndexError, StopIteration):
|
| 142 |
+
return None
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
def _command(pid: int) -> str:
|
| 146 |
+
try:
|
| 147 |
+
return Path(f"/proc/{pid}/cmdline").read_bytes().replace(b"\0", b" ").decode(errors="replace")
|
| 148 |
+
except OSError:
|
| 149 |
+
return ""
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def on_molab(env=None, places=None, commands=None) -> bool:
|
| 153 |
+
"""Whether this runs in a molab notebook: the packages are where molab keeps them, a process above this one is
|
| 154 |
+
marimo, or marimo's settings are in the environment."""
|
| 155 |
+
env = os.environ if env is None else env
|
| 156 |
+
places = [sys.prefix, __file__] if places is None else places
|
| 157 |
+
commands = [_command(pid) for pid in _ancestors()[1:]] if commands is None else commands
|
| 158 |
+
return (any(str(p).startswith(MOLAB_PACKAGES) for p in places) or any("marimo" in c for c in commands)
|
| 159 |
+
or any(k.startswith("MARIMO") or "MOLAB" in k for k in env))
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def session_limit(env=None) -> tuple:
|
| 163 |
+
"""``(hours, why)``: how long one session on this machine may run, or ``(None, why)`` when nothing says it ends.
|
| 164 |
+
``PLUVIUM_SESSION_HOURS`` says so (0 for no end); otherwise a molab notebook has 12 hours."""
|
| 165 |
+
env = os.environ if env is None else env
|
| 166 |
+
given = str(env.get("PLUVIUM_SESSION_HOURS", "")).strip()
|
| 167 |
+
if given:
|
| 168 |
+
hours, why = float(given), env.get("PLUVIUM_SESSION_WHY") or "PLUVIUM_SESSION_HOURS"
|
| 169 |
+
return (hours, why) if hours > 0 else (None, why)
|
| 170 |
+
if on_molab(env):
|
| 171 |
+
return MOLAB_HOURS, "molab stops a notebook after 12 hours"
|
| 172 |
+
return None, "nothing says this machine stops its sessions"
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def session_start(hours: float, env=None, now: float | None = None) -> float:
|
| 176 |
+
"""When this session began: ``PLUVIUM_SESSION_START`` when the supervisor passes it on, otherwise the earliest
|
| 177 |
+
start among this process and the ones above it (the notebook, its server, the container) that is less than
|
| 178 |
+
``hours`` ago. A process older than that was there before the session."""
|
| 179 |
+
env = os.environ if env is None else env
|
| 180 |
+
now = time.time() if now is None else now
|
| 181 |
+
if env.get("PLUVIUM_SESSION_START"):
|
| 182 |
+
return float(env["PLUVIUM_SESSION_START"])
|
| 183 |
+
times = [t for t in map(_began, _ancestors()) if t is not None and now - hours * 3600.0 < t <= now]
|
| 184 |
+
return min(times, default=now)
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
def _clock(seconds: float) -> str:
|
| 188 |
+
return time.strftime("%H:%M", time.gmtime(seconds))
|
| 189 |
+
|
| 190 |
+
|
| 191 |
class Program:
|
| 192 |
def __init__(self, cfg: ProgramConfig) -> None:
|
| 193 |
self.cfg = cfg
|
| 194 |
self.started = time.time()
|
| 195 |
self.deadline = self.started + cfg.gpu_hours * 3600.0
|
| 196 |
+
#: ``(began, hours, why)`` when this machine stops its sessions after a fixed time. The run is planned to be
|
| 197 |
+
#: published half an hour before then; a run stopped before that publishes nothing
|
| 198 |
+
self.session = None
|
| 199 |
+
hours, why = session_limit()
|
| 200 |
+
if hours:
|
| 201 |
+
began = session_start(hours)
|
| 202 |
+
end = began + (hours - SESSION_MARGIN_HOURS) * 3600.0
|
| 203 |
+
if end < self.deadline:
|
| 204 |
+
self.deadline, self.session = end, (began, hours, why)
|
| 205 |
self.root = Path(cfg.root).expanduser()
|
| 206 |
self.root.mkdir(parents=True, exist_ok=True)
|
| 207 |
self.state_path = self.root / "program.json"
|
|
|
|
| 218 |
"""Hours left in the budget."""
|
| 219 |
return (self.deadline - time.time()) / 3600.0
|
| 220 |
|
| 221 |
+
def reserve(self) -> float:
|
| 222 |
+
"""Hours kept for calibration, scoring, publishing and the forecast. More when the session has an end: none
|
| 223 |
+
of these has run at full size on a GPU yet, and a session stopped in the middle of them publishes nothing."""
|
| 224 |
+
base = 0.9 + 0.1 * self.cfg.gpu_hours / 24
|
| 225 |
+
return max(base, SESSION_RESERVE_HOURS) if self.session else base
|
| 226 |
+
|
| 227 |
def mark(self, stage: str, **info) -> None:
|
| 228 |
self.state[stage] = {"done": time.strftime("%Y-%m-%d %H:%M"), **info}
|
| 229 |
self.state_path.write_text(json.dumps(self.state, indent=1))
|
|
|
|
| 467 |
log(f"GPU: {props.name}, {props.total_memory / 2**30:.0f} GB ({how}); {os.cpu_count()} CPU cores")
|
| 468 |
torch.backends.cuda.matmul.allow_tf32 = True
|
| 469 |
torch.backends.cudnn.allow_tf32 = True
|
| 470 |
+
if self.session:
|
| 471 |
+
began, hours, why = self.session
|
| 472 |
+
log(f"this session began at {_clock(began)} UTC and ends {hours:g} hours later ({why}): the network is "
|
| 473 |
+
f"trained, scored and published by {_clock(self.deadline)} UTC")
|
| 474 |
+
if self.left() < self.reserve() + 1.0:
|
| 475 |
+
log(f"only {self.left():.1f} hours of this session are left, so training gets little of them; a "
|
| 476 |
+
f"session started fresh has all {hours:g}")
|
| 477 |
geo, towns = self.geography()
|
| 478 |
era5, cmorph, stations = self.data(geo, self.years + (cfg.holdout_year,))
|
| 479 |
clim = self.climatology(geo, cmorph)
|
|
|
|
| 495 |
f"{test_starts.size} for scoring")
|
| 496 |
model = build_model(geo, grids, cfg.size, st_points, **cfg.overrides)
|
| 497 |
loss = PluviumLoss(states=model.config.states)
|
| 498 |
+
reserve = self.reserve() # calibration, scoring, publishing, forecast
|
| 499 |
train_hours_total = max(0.25, self.left() - reserve)
|
| 500 |
share_a = train_hours_total * (0.85 if self.stage_b else 1.0)
|
| 501 |
trainer = Trainer(model, loss, builder, train_starts, val_starts, self.root / "stage_a",
|
|
|
|
| 664 |
return None
|
| 665 |
cut = int(starts.size * 0.75)
|
| 666 |
train_b, held_b = starts[:cut], starts[cut:]
|
| 667 |
+
hours_b = max(0.2, self.left() - self.reserve())
|
| 668 |
cfg_b = self.train_cfg(hours_b, warmup_steps=50)
|
| 669 |
cfg_b.lr = self.cfg.lr / 3
|
| 670 |
# stage B's forecasts are built differently, so its step is sized afresh, unless a batch was given by hand
|
|
|
|
| 919 |
|
| 920 |
#: a run that prints nothing for this many minutes is taken to have hung; it is stopped and started again
|
| 921 |
STALL_MINUTES = 120.0
|
| 922 |
+
#: words in a stopped run's last lines that say the step was too large for the GPU, so the next try is smaller
|
| 923 |
+
SIZE_SIGNS = ("out of memory", "OutOfMemoryError", "CUBLAS_STATUS", "CUDNN_STATUS", "CUDA error", "cudaError")
|
| 924 |
+
|
| 925 |
+
|
| 926 |
+
def about_size(tail) -> bool:
|
| 927 |
+
"""Whether a stopped run's last lines say its step was too large for the GPU."""
|
| 928 |
+
return any(sign in line for line in list(tail)[-60:] for sign in SIZE_SIGNS)
|
| 929 |
#: lines of a stopped run's output that are kept, on disk and on the Hub, so that its error can be read afterwards
|
| 930 |
TAIL_LINES = 400
|
| 931 |
_SECRET = re.compile(r"hf_[A-Za-z0-9]{20,}|pypi-[A-Za-z0-9_-]{20,}|gh[pousr]_[A-Za-z0-9]{20,}")
|
|
|
|
| 959 |
token=None, stall_minutes: float | None = None, api=None) -> int:
|
| 960 |
"""Run the program in a child process and keep it going. If the child stops with an error, is killed (the
|
| 961 |
machine ran out of RAM, say), or prints nothing for ``stall_minutes``, it is started again from its last
|
| 962 |
+
checkpoint, with a smaller step (:func:`retry_plan`) when the stop could have come from the step's size. Any other
|
| 963 |
+
error is tried again at the same size: 0.5.5 stopped on a bug that had nothing to do with memory, and every try
|
| 964 |
+
after it had less of the card. A fresh process starts with a fresh CUDA context,
|
| 965 |
which some GPU errors leave unusable; the data and the checkpoints are on disk, so the next try is training
|
| 966 |
again within minutes.
|
| 967 |
|
|
|
|
| 976 |
tries = retry_plan(vram, share)
|
| 977 |
limit = 60.0 * float(stall_minutes if stall_minutes is not None else os.environ.get("PLUVIUM_STALL_MINUTES") or STALL_MINUTES)
|
| 978 |
state = {"child": None}
|
| 979 |
+
hours, why = session_limit() # found once, so that every try plans the same end
|
| 980 |
+
session = {"PLUVIUM_SESSION_HOURS": f"{hours or 0:g}", "PLUVIUM_SESSION_WHY": why,
|
| 981 |
+
"PLUVIUM_SESSION_START": f"{session_start(hours) if hours else time.time():.0f}"}
|
| 982 |
|
| 983 |
def end_child() -> None:
|
| 984 |
child = state["child"]
|
|
|
|
| 995 |
raise SystemExit(128 + signum)
|
| 996 |
|
| 997 |
previous = signal.signal(signal.SIGTERM, stop)
|
| 998 |
+
code, level = 1, 0
|
| 999 |
try:
|
| 1000 |
+
for k in range(len(tries)):
|
| 1001 |
+
rule, cut, batch = tries[level]
|
| 1002 |
+
env = dict(os.environ, **session, PLUVIUM_ATTEMPT=str(k + 1), PLUVIUM_VRAM=rule,
|
| 1003 |
+
PLUVIUM_VRAM_SHARE=str(cut), PYTHONUNBUFFERED="1")
|
| 1004 |
extra = ["--vram", rule, "--vram-share", str(cut)]
|
| 1005 |
if batch:
|
| 1006 |
env["PLUVIUM_BATCH"] = str(batch)
|
|
|
|
| 1033 |
settings = f"{rule}, share {cut}" + (f", batch {batch}" if batch else "")
|
| 1034 |
saved = save_attempt(root, repo, token, k + 1, f"attempt {k + 1} of {len(tries)} ({settings}): the program {what}",
|
| 1035 |
list(tail), api=api) if root else None
|
| 1036 |
+
smaller = hung or code < 0 or about_size(tail)
|
| 1037 |
code = code if code > 0 else 1
|
| 1038 |
if k + 1 < len(tries):
|
| 1039 |
+
level = min(level + 1, len(tries) - 1) if smaller else level
|
| 1040 |
+
then, then_cut, then_batch = tries[level]
|
| 1041 |
+
how = ("with the same settings, since the error was not about memory" if not smaller else
|
| 1042 |
+
f"with one forecast per step (--batch {then_batch})" if then_batch else
|
| 1043 |
f"with each training step sized to {then_cut:.0%} of the GPU's memory" if then == "fill" else
|
| 1044 |
"with the batch rule of 0.4.2 (--vram fastest)")
|
| 1045 |
log(f"the program {what}" + (f"; the end of its output is in {saved}" if saved else "")
|
|
|
|
| 1052 |
signal.signal(signal.SIGTERM, previous)
|
| 1053 |
|
| 1054 |
|
| 1055 |
+
__all__ = ["DEFAULT_REPO", "FILES_PER_COMMIT", "MOLAB_HOURS", "Program", "ProgramConfig", "RETRY_SHARE_DROP",
|
| 1056 |
+
"SESSION_MARGIN_HOURS", "SESSION_RESERVE_HOURS", "STALL_MINUTES", "about_size", "choose_years", "main",
|
| 1057 |
+
"mirror_files", "on_molab", "retry_plan", "save_attempt", "session_limit", "session_start", "supervise"]
|
| 1058 |
|
| 1059 |
# ``python -m pluvium.program`` is how the notebook cells start a training run. Until 0.4.3 these two lines were
|
| 1060 |
# missing: the command imported this module and ended at once with exit code 0, having trained nothing.
|
pluvium/training/ema.py
CHANGED
|
@@ -27,6 +27,8 @@ class EMA:
|
|
| 27 |
d = min(self.decay, (1 + self.updates) / (10 + self.updates)) # warm-up: short memory at the start
|
| 28 |
for k, v in model.state_dict().items():
|
| 29 |
if k in self.shadow:
|
|
|
|
|
|
|
| 30 |
self.shadow[k].mul_(d).add_(v.detach().float(), alpha=1 - d)
|
| 31 |
|
| 32 |
@contextlib.contextmanager
|
|
@@ -43,8 +45,12 @@ class EMA:
|
|
| 43 |
return {"decay": self.decay, "updates": self.updates, "shadow": self.shadow}
|
| 44 |
|
| 45 |
def load_state_dict(self, state: dict) -> None:
|
|
|
|
|
|
|
|
|
|
| 46 |
self.decay, self.updates = state["decay"], state["updates"]
|
| 47 |
-
self.shadow = {k: v.float()
|
|
|
|
| 48 |
|
| 49 |
|
| 50 |
__all__ = ["EMA"]
|
|
|
|
| 27 |
d = min(self.decay, (1 + self.updates) / (10 + self.updates)) # warm-up: short memory at the start
|
| 28 |
for k, v in model.state_dict().items():
|
| 29 |
if k in self.shadow:
|
| 30 |
+
if self.shadow[k].device != v.device: # the averages follow the weights
|
| 31 |
+
self.shadow[k] = self.shadow[k].to(v.device)
|
| 32 |
self.shadow[k].mul_(d).add_(v.detach().float(), alpha=1 - d)
|
| 33 |
|
| 34 |
@contextlib.contextmanager
|
|
|
|
| 45 |
return {"decay": self.decay, "updates": self.updates, "shadow": self.shadow}
|
| 46 |
|
| 47 |
def load_state_dict(self, state: dict) -> None:
|
| 48 |
+
"""The averages are put where the model's weights are. A checkpoint is read onto the CPU, and until 0.5.6
|
| 49 |
+
they stayed there: a run resumed on a GPU stopped at its first step with "Expected all tensors to be on the
|
| 50 |
+
same device", after 11.6 hours of training had been saved."""
|
| 51 |
self.decay, self.updates = state["decay"], state["updates"]
|
| 52 |
+
self.shadow = {k: v.float().to(self.shadow[k].device if k in self.shadow else v.device)
|
| 53 |
+
for k, v in state["shadow"].items()}
|
| 54 |
|
| 55 |
|
| 56 |
__all__ = ["EMA"]
|
pyproject.toml
CHANGED
|
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "pluvium"
|
| 7 |
-
version = "0.5.
|
| 8 |
description = "A rain model that says when each parish might get rain, down to the exact town or city, and the time when rain is going to fall. Made for Jamaica first, works for any country."
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
|
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "pluvium"
|
| 7 |
+
version = "0.5.6"
|
| 8 |
description = "A rain model that says when each parish might get rain, down to the exact town or city, and the time when rain is going to fall. Made for Jamaica first, works for any country."
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
tests/test_fit.py
CHANGED
|
@@ -248,11 +248,12 @@ def test_a_run_that_stops_is_started_again_with_smaller_steps_then_the_old_rule(
|
|
| 248 |
|
| 249 |
log = tmp_path / "tries.txt"
|
| 250 |
|
| 251 |
-
def child(succeed_at: int) -> list:
|
| 252 |
return [sys.executable, "-c",
|
| 253 |
"import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_VRAM'] + ' ' + "
|
| 254 |
"os.environ['PLUVIUM_VRAM_SHARE'] + ' ' + os.environ.get('PLUVIUM_BATCH', '-') + ' ' + "
|
| 255 |
-
f"' '.join(sys.argv[2:]) + '\\n');
|
|
|
|
| 256 |
str(log)]
|
| 257 |
|
| 258 |
assert supervise(["--hours", "1"], 0.95, command=child(3)) == 0
|
|
@@ -264,6 +265,12 @@ def test_a_run_that_stops_is_started_again_with_smaller_steps_then_the_old_rule(
|
|
| 264 |
assert supervise(["--hours", "1"], 0.95, command=child(9), vram="fastest") == 1
|
| 265 |
assert log.read_text().splitlines() == ["fastest 0.95 - --hours 1 --vram fastest --vram-share 0.95",
|
| 266 |
"fastest 0.95 1 --hours 1 --vram fastest --vram-share 0.95 --batch 1"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
once = [sys.executable, "-c", "import sys; sys.exit(0)"]
|
| 268 |
assert supervise([], 0.95, command=once) == 0 # a run that works is run once
|
| 269 |
always = [sys.executable, "-c", "import sys; sys.exit(3)"]
|
|
@@ -402,3 +409,28 @@ def test_one_batch_that_cannot_be_trained_on_is_left_out_and_five_in_a_row_stop_
|
|
| 402 |
with pytest.raises(ValueError, match="every batch fails"):
|
| 403 |
always.run()
|
| 404 |
assert always.step == 4 # four left out, the fifth stopped it
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 248 |
|
| 249 |
log = tmp_path / "tries.txt"
|
| 250 |
|
| 251 |
+
def child(succeed_at: int, says: str = "torch.OutOfMemoryError: CUDA out of memory.") -> list:
|
| 252 |
return [sys.executable, "-c",
|
| 253 |
"import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_VRAM'] + ' ' + "
|
| 254 |
"os.environ['PLUVIUM_VRAM_SHARE'] + ' ' + os.environ.get('PLUVIUM_BATCH', '-') + ' ' + "
|
| 255 |
+
f"' '.join(sys.argv[2:]) + '\\n'); ok = os.environ['PLUVIUM_ATTEMPT'] == '{succeed_at}'; "
|
| 256 |
+
f"print('' if ok else {says!r}); sys.exit(0 if ok else 1)",
|
| 257 |
str(log)]
|
| 258 |
|
| 259 |
assert supervise(["--hours", "1"], 0.95, command=child(3)) == 0
|
|
|
|
| 265 |
assert supervise(["--hours", "1"], 0.95, command=child(9), vram="fastest") == 1
|
| 266 |
assert log.read_text().splitlines() == ["fastest 0.95 - --hours 1 --vram fastest --vram-share 0.95",
|
| 267 |
"fastest 0.95 1 --hours 1 --vram fastest --vram-share 0.95 --batch 1"]
|
| 268 |
+
log.unlink()
|
| 269 |
+
# an error that has nothing to do with memory is tried again at the same size. 0.5.5 stopped on such an error
|
| 270 |
+
# (the averaged weights on the CPU) and every try after it had less of the card
|
| 271 |
+
same = child(2, "RuntimeError: Expected all tensors to be on the same device, but found at least two devices")
|
| 272 |
+
assert supervise(["--hours", "1"], 0.95, command=same) == 0
|
| 273 |
+
assert log.read_text().splitlines() == ["fill 0.95 - --hours 1 --vram fill --vram-share 0.95"] * 2
|
| 274 |
once = [sys.executable, "-c", "import sys; sys.exit(0)"]
|
| 275 |
assert supervise([], 0.95, command=once) == 0 # a run that works is run once
|
| 276 |
always = [sys.executable, "-c", "import sys; sys.exit(3)"]
|
|
|
|
| 409 |
with pytest.raises(ValueError, match="every batch fails"):
|
| 410 |
always.run()
|
| 411 |
assert always.step == 4 # four left out, the fifth stopped it
|
| 412 |
+
|
| 413 |
+
|
| 414 |
+
def test_averaged_weights_read_from_a_checkpoint_follow_the_model_to_its_device(world, tmp_path):
|
| 415 |
+
"""0.5.5 read a checkpoint onto the CPU and left the averaged weights there. On the GPU the resumed run stopped
|
| 416 |
+
at its first step (8,626, after 11.6 hours) with "Expected all tensors to be on the same device". The meta
|
| 417 |
+
device stands in for the GPU: it does not raise that error, but it shows where the averages are."""
|
| 418 |
+
import torch
|
| 419 |
+
|
| 420 |
+
from pluvium.training import EMA
|
| 421 |
+
|
| 422 |
+
model = torch.nn.Linear(3, 2)
|
| 423 |
+
saved = EMA(model, 0.9).state_dict() # on the CPU, as a checkpoint is read
|
| 424 |
+
model = model.to("meta")
|
| 425 |
+
ema = EMA(model, 0.9)
|
| 426 |
+
ema.load_state_dict(saved)
|
| 427 |
+
assert {v.device.type for v in ema.shadow.values()} == {"meta"} and ema.updates == 0
|
| 428 |
+
stray = EMA(torch.nn.Linear(3, 2), 0.9) # made on the CPU, the model moved after
|
| 429 |
+
stray.update(model)
|
| 430 |
+
assert {v.device.type for v in stray.shadow.values()} == {"meta"} and stray.updates == 1
|
| 431 |
+
# and a trainer that resumes keeps its averages where its model is
|
| 432 |
+
trainer, _ = make_trainer(world, tmp_path / "a", ema_decay=0.99)
|
| 433 |
+
trainer.run()
|
| 434 |
+
again, _ = make_trainer(world, tmp_path / "a", ema_decay=0.99)
|
| 435 |
+
where = {p.device for p in again.model.parameters()}
|
| 436 |
+
assert again.step == 2 and again.ema.updates == 2 and {v.device for v in again.ema.shadow.values()} == where
|
tests/test_program.py
CHANGED
|
@@ -160,3 +160,54 @@ def test_the_background_fetch_does_not_keep_a_finished_run_alive(tmp_path):
|
|
| 160 |
began = time.time()
|
| 161 |
prog.stop_prefetch()
|
| 162 |
assert not prog._prefetch.is_alive() and time.time() - began < 30
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
began = time.time()
|
| 161 |
prog.stop_prefetch()
|
| 162 |
assert not prog._prefetch.is_alive() and time.time() - began < 30
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
def test_a_molab_session_is_planned_to_be_published_before_molab_stops_it(tmp_path, monkeypatch):
|
| 166 |
+
"""molab stops a notebook 12 hours after it starts. The first full run there trained for 11.6 of its 24 hours and
|
| 167 |
+
was stopped before it was scored and published. A run there now plans to be done before the session ends."""
|
| 168 |
+
import time
|
| 169 |
+
|
| 170 |
+
from pluvium import program as P
|
| 171 |
+
|
| 172 |
+
for name in ("PLUVIUM_SESSION_HOURS", "PLUVIUM_SESSION_START", "PLUVIUM_SESSION_WHY"):
|
| 173 |
+
monkeypatch.delenv(name, raising=False)
|
| 174 |
+
molab = "/tmp/uv-venv/lib/python3.13/site-packages/pluvium/program.py" # as the tracebacks there show
|
| 175 |
+
assert P.on_molab({}, places=[molab], commands=[])
|
| 176 |
+
assert P.on_molab({}, places=["/usr/lib"], commands=["/usr/bin/python -m marimo edit notebook.py"])
|
| 177 |
+
assert P.on_molab({"MARIMO_MANAGE_SCRIPT_METADATA": "true"}, places=["/usr/lib"], commands=[])
|
| 178 |
+
assert not P.on_molab({"HOME": "/root"}, places=["/usr/lib"], commands=["bash", "python -m pluvium.program"])
|
| 179 |
+
assert P.session_limit({"PLUVIUM_SESSION_HOURS": "6"}) == (6.0, "PLUVIUM_SESSION_HOURS")
|
| 180 |
+
assert P.session_limit({"PLUVIUM_SESSION_HOURS": "0"})[0] is None # 0: no end, even on molab
|
| 181 |
+
began = time.time() - 40 * 60
|
| 182 |
+
assert P.session_start(12, {"PLUVIUM_SESSION_START": str(began)}) == began
|
| 183 |
+
found = P.session_start(12, {}) # the earliest process of this session
|
| 184 |
+
assert time.time() - 12 * 3600 < found <= time.time()
|
| 185 |
+
|
| 186 |
+
monkeypatch.setattr(P, "on_molab", lambda env=None, places=None, commands=None: True)
|
| 187 |
+
assert P.session_limit({}) == (12.0, "molab stops a notebook after 12 hours")
|
| 188 |
+
# the notebook started 40 minutes before the program: published 11.5 hours after that, not 24 hours after
|
| 189 |
+
monkeypatch.setenv("PLUVIUM_SESSION_START", str(began))
|
| 190 |
+
prog = Program(ProgramConfig(root=str(tmp_path), hub_repo=None, gpu_hours=24))
|
| 191 |
+
assert prog.session is not None and abs(prog.deadline - (began + 11.5 * 3600)) < 1
|
| 192 |
+
assert 10.7 < prog.left() < 10.9 and prog.reserve() == 1.5
|
| 193 |
+
# no end known: 24 hours from the program's start, as before
|
| 194 |
+
monkeypatch.setattr(P, "on_molab", lambda env=None, places=None, commands=None: False)
|
| 195 |
+
prog = Program(ProgramConfig(root=str(tmp_path), hub_repo=None, gpu_hours=24))
|
| 196 |
+
assert prog.session is None and 23.9 < prog.left() <= 24 and prog.reserve() == 1.0
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
def test_the_supervisor_finds_the_session_once_and_every_try_gets_it(tmp_path, monkeypatch):
|
| 200 |
+
import sys
|
| 201 |
+
import time
|
| 202 |
+
|
| 203 |
+
from pluvium.program import supervise
|
| 204 |
+
|
| 205 |
+
seen = tmp_path / "env.txt"
|
| 206 |
+
child = [sys.executable, "-c", "import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_SESSION_HOURS'] "
|
| 207 |
+
"+ ' ' + os.environ['PLUVIUM_SESSION_START'] + '\\n'); sys.exit(0 if os.environ['PLUVIUM_ATTEMPT'] == '2' "
|
| 208 |
+
"else 1)", str(seen)]
|
| 209 |
+
monkeypatch.delenv("PLUVIUM_SESSION_START", raising=False)
|
| 210 |
+
monkeypatch.setenv("PLUVIUM_SESSION_HOURS", "12")
|
| 211 |
+
assert supervise([], 0.95, command=child) == 0
|
| 212 |
+
first, second = seen.read_text().splitlines()
|
| 213 |
+
assert first == second and first.split()[0] == "12" and float(first.split()[1]) <= time.time()
|