Sigmandndnns commited on
Commit
816b99f
·
verified ·
1 Parent(s): b114b3f

Pluvium: code, model card, papers and benchmarks

Browse files
CHANGELOG.md CHANGED
@@ -1,5 +1,20 @@
1
  # Changelog
2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ## 0.5.5 (November 2026)
4
 
5
  The third GPU run trained, then stopped on a missing file. Its log was on the Hub this time, so the cause is known.
 
1
  # Changelog
2
 
3
+ ## 0.5.6 (November 2026)
4
+
5
+ The GPU run trained on molab for 11.6 hours, to step 8,625, and its best held-out loss was 0.734. Then two things
6
+ stopped it.
7
+
8
+ - molab shuts a notebook down 12 hours after it starts and the run was planned for 24, so it was stopped while it
9
+ was still training, before it was scored and published. On molab the run now ends half an hour before the 12
10
+ hours are up, counted from when the notebook started, with 1.5 hours kept for scoring and publishing.
11
+ `PLUVIUM_SESSION_HOURS` sets the limit on other machines.
12
+ - Run again, it went on from step 8,625 and stopped at the next step with "Expected all tensors to be on the same
13
+ device". The checkpoint is read onto the CPU and the averaged weights stayed there while the model was on the
14
+ GPU. They are put on the model's device now.
15
+ - The three restarts after that came back to the same error, each with less of the card. Only a stop that has to
16
+ do with memory makes the next try smaller now. Any other error is tried again at the same size.
17
+
18
  ## 0.5.5 (November 2026)
19
 
20
  The third GPU run trained, then stopped on a missing file. Its log was on the Hub this time, so the cause is known.
docs/MODULES.md CHANGED
@@ -114,7 +114,7 @@ Every module of the package, what it is for, and its size. Generated from the mo
114
  | `pluvium/prob/extremes.py` | 91 | The tail: cloudbursts and their return periods. |
115
  | `pluvium/prob/scores.py` | 88 | Scores that a single mean error hides. |
116
  | `pluvium/products.py` | 192 | From the heads' raw outputs to what a person asks: will it rain here, when will it start, how hard, will there be thunder, how long will it last, how much will fall, and why. |
117
- | `pluvium/program.py` | 935 | Pluvium-500 from nothing on one GPU, in stages that each finish or resume: |
118
  | `pluvium/quick.py` | 617 | From nothing to a forecast in one call. |
119
  | `pluvium/render/__init__.py` | 16 | Forecasts in the forms people and agencies use: |
120
  | `pluvium/render/alerts.py` | 77 | Alerts in the Common Alerting Protocol (OASIS CAP 1.2). |
@@ -156,12 +156,12 @@ Every module of the package, what it is for, and its size. Generated from the mo
156
  | `pluvium/tails/twcrps.py` | 45 | Scoring the tail on its own: the threshold-weighted CRPS. |
157
  | `pluvium/tails/withheld.py` | 189 | What happens when the storm is bigger than anything in training? |
158
  | `pluvium/timing.py` | 234 | When in the day it rains: the first three days of the forecast, split into six-hour periods. |
159
- | `pluvium/train.py` | 981 | Training on one GPU: batches built on every CPU core, bf16 on the GPU, a step sized by measuring to fill the GPU's memory (:mod:`pluvium.training.fit`), a learning-rate schedule that ends when the time budget does, checkpoints that resume exactly, and (optionally) a mirror of every checkpoint on the Hugging Face Hub so a rented machine that stops loses nothing. |
160
  | `pluvium/training/__init__.py` | 23 | How the network is taught, beyond the loss: |
161
  | `pluvium/training/augment.py` | 53 | Perturbations with a physical meaning, applied to training examples. |
162
  | `pluvium/training/curriculum.py` | 58 | Showing the network the rare events. |
163
  | `pluvium/training/distill.py` | 38 | Distillation: a small Pluvium that forecasts like the large one. |
164
- | `pluvium/training/ema.py` | 50 | Polyak averaging of the weights. |
165
  | `pluvium/training/fit.py` | 228 | Sizing a training step to the GPU's memory. |
166
  | `pluvium/upstream/__init__.py` | 22 | The published weights of the AI weather models whose forecasts Pluvium reads, and how to keep a copy of them. |
167
  | `pluvium/upstream/fetch.py` | 215 | Copy a published set of weights to a local folder, unchanged, with its licence and a record of where it came from. |
@@ -172,4 +172,4 @@ Every module of the package, what it is for, and its size. Generated from the mo
172
  | `pluvium/weather.py` | 283 | What the weather *is*, in the words people use: intensity classes, drizzle versus rain versus showers, thunder. |
173
  | `pluvium/zarrlite.py` | 166 | A minimal reader for consolidated Zarr v2 stores over HTTPS (the public ERA5 stores on Google Cloud). |
174
 
175
- 167 modules, 25,058 lines.
 
114
  | `pluvium/prob/extremes.py` | 91 | The tail: cloudbursts and their return periods. |
115
  | `pluvium/prob/scores.py` | 88 | Scores that a single mean error hides. |
116
  | `pluvium/products.py` | 192 | From the heads' raw outputs to what a person asks: will it rain here, when will it start, how hard, will there be thunder, how long will it last, how much will fall, and why. |
117
+ | `pluvium/program.py` | 1062 | Pluvium-500 from nothing on one GPU, in stages that each finish or resume: |
118
  | `pluvium/quick.py` | 617 | From nothing to a forecast in one call. |
119
  | `pluvium/render/__init__.py` | 16 | Forecasts in the forms people and agencies use: |
120
  | `pluvium/render/alerts.py` | 77 | Alerts in the Common Alerting Protocol (OASIS CAP 1.2). |
 
156
  | `pluvium/tails/twcrps.py` | 45 | Scoring the tail on its own: the threshold-weighted CRPS. |
157
  | `pluvium/tails/withheld.py` | 189 | What happens when the storm is bigger than anything in training? |
158
  | `pluvium/timing.py` | 234 | When in the day it rains: the first three days of the forecast, split into six-hour periods. |
159
+ | `pluvium/train.py` | 987 | Training on one GPU: batches built on every CPU core, bf16 on the GPU, a step sized by measuring to fill the GPU's memory (:mod:`pluvium.training.fit`), a learning-rate schedule that ends when the time budget does, checkpoints that resume exactly, and (optionally) a mirror of every checkpoint on the Hugging Face Hub so a rented machine that stops loses nothing. |
160
  | `pluvium/training/__init__.py` | 23 | How the network is taught, beyond the loss: |
161
  | `pluvium/training/augment.py` | 53 | Perturbations with a physical meaning, applied to training examples. |
162
  | `pluvium/training/curriculum.py` | 58 | Showing the network the rare events. |
163
  | `pluvium/training/distill.py` | 38 | Distillation: a small Pluvium that forecasts like the large one. |
164
+ | `pluvium/training/ema.py` | 56 | Polyak averaging of the weights. |
165
  | `pluvium/training/fit.py` | 228 | Sizing a training step to the GPU's memory. |
166
  | `pluvium/upstream/__init__.py` | 22 | The published weights of the AI weather models whose forecasts Pluvium reads, and how to keep a copy of them. |
167
  | `pluvium/upstream/fetch.py` | 215 | Copy a published set of weights to a local folder, unchanged, with its licence and a record of where it came from. |
 
172
  | `pluvium/weather.py` | 283 | What the weather *is*, in the words people use: intensity classes, drizzle versus rain versus showers, thunder. |
173
  | `pluvium/zarrlite.py` | 166 | A minimal reader for consolidated Zarr v2 stores over HTTPS (the public ERA5 stores on Google Cloud). |
174
 
175
+ 167 modules, 25,197 lines.
docs/TRAINING.md CHANGED
@@ -65,7 +65,7 @@ def sh(cmd, check=True):
65
  print(f"$ {cmd}", flush=True)
66
  return subprocess.run(cmd, shell=True, check=check).returncode
67
 
68
- sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.5'")
69
  root = os.path.expanduser(ROOT)
70
  pluvium = f"{sys.executable} -m pluvium.cli"
71
  if not os.path.exists(f"{root}/area.json"):
@@ -162,6 +162,7 @@ is unchanged up to a batch of 8 and grows with the square root of the batch abov
162
  | `PLUVIUM_VRAM_SHARE=0.95` | The share of the card a step may need. The rest is a margin. |
163
  | `PLUVIUM_VRAM=fastest` | The rule used before 0.4.3: keep the batch that is fastest per forecast. On a 90 GB card that can leave most of the memory unused. |
164
  | `PLUVIUM_BATCH=6` | A batch given by hand. Nothing is measured. |
 
165
 
166
  A fuller card is not a faster one by itself. Forecasts per second stay about the same; each step holds more
167
  forecasts and more cells, so there are fewer and steadier steps in the same time. If a step ever needs more than the
@@ -180,6 +181,11 @@ It is started again in the same way when it was killed (the machine ran out of R
180
  (`PLUVIUM_STALL_MINUTES` changes that). The last 400 lines of every run that stopped are kept in
181
  `logs/attempt-N.txt`, in the working folder and in the Hub repository, so the error can be read afterwards.
182
 
 
 
 
 
 
183
  On an RTX PRO 6000 (96 GB), one Pluvium-500 forecast took 38.6 GB at its peak and trained at 0.24 forecasts a second,
184
  so a step holds two forecasts at most and the rest of the card goes to more target cells.
185
 
 
65
  print(f"$ {cmd}", flush=True)
66
  return subprocess.run(cmd, shell=True, check=check).returncode
67
 
68
+ sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.6'")
69
  root = os.path.expanduser(ROOT)
70
  pluvium = f"{sys.executable} -m pluvium.cli"
71
  if not os.path.exists(f"{root}/area.json"):
 
162
  | `PLUVIUM_VRAM_SHARE=0.95` | The share of the card a step may need. The rest is a margin. |
163
  | `PLUVIUM_VRAM=fastest` | The rule used before 0.4.3: keep the batch that is fastest per forecast. On a 90 GB card that can leave most of the memory unused. |
164
  | `PLUVIUM_BATCH=6` | A batch given by hand. Nothing is measured. |
165
+ | `PLUVIUM_SESSION_HOURS=12` | How long one session of the machine may run. The run is published half an hour before the end. On molab it is 12 without being set, and 0 means no end. |
166
 
167
  A fuller card is not a faster one by itself. Forecasts per second stay about the same; each step holds more
168
  forecasts and more cells, so there are fewer and steadier steps in the same time. If a step ever needs more than the
 
181
  (`PLUVIUM_STALL_MINUTES` changes that). The last 400 lines of every run that stopped are kept in
182
  `logs/attempt-N.txt`, in the working folder and in the Hub repository, so the error can be read afterwards.
183
 
184
+ molab stops a notebook 12 hours after it starts. The program sees when it runs there and plans the run to be scored
185
+ and published half an hour before that, so on molab a run of `GPU_HOURS = 24` trains for about 10 hours in one
186
+ session. A run that is stopped anyway goes on from its last checkpoint when the cell is run again. On another machine
187
+ that stops its sessions, `PLUVIUM_SESSION_HOURS=12` says so.
188
+
189
  On an RTX PRO 6000 (96 GB), one Pluvium-500 forecast took 38.6 GB at its peak and trained at 0.24 forecasts a second,
190
  so a step holds two forecasts at most and the rest of the card goes to more target cells.
191
 
notebooks/pluvium_blend_cell.py CHANGED
@@ -22,7 +22,7 @@ def sh(cmd: str, check: bool = True) -> int:
22
 
23
 
24
  # 1. the package: from PyPI, or from the source tree when PyPI is not reachable
25
- if sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.5'", check=False) != 0:
26
  sh(f"{sys.executable} -m pip install -q 'git+https://github.com/Wisk-inc/transformers.git"
27
  f"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium'")
28
 
 
22
 
23
 
24
  # 1. the package: from PyPI, or from the source tree when PyPI is not reachable
25
+ if sh(f"{sys.executable} -m pip install -q --upgrade 'pluvium>=0.5.6'", check=False) != 0:
26
  sh(f"{sys.executable} -m pip install -q 'git+https://github.com/Wisk-inc/transformers.git"
27
  f"@claude/linear-context-architecture-a35c6h#subdirectory=pluvium'")
28
 
notebooks/pluvium_everything.py CHANGED
@@ -93,7 +93,7 @@ def step(what: str, action, skip: str = "") -> bool:
93
  # 1. the package: from PyPI, or from the source tree when PyPI cannot be reached
94
  def install() -> None:
95
  try:
96
- run([sys.executable, "-m", "pip", "install", "-q", "--upgrade", "pluvium>=0.5.5"])
97
  except RuntimeError:
98
  run([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/Wisk-inc/transformers.git"
99
  "@claude/linear-context-architecture-a35c6h#subdirectory=pluvium"])
 
93
  # 1. the package: from PyPI, or from the source tree when PyPI cannot be reached
94
  def install() -> None:
95
  try:
96
+ run([sys.executable, "-m", "pip", "install", "-q", "--upgrade", "pluvium>=0.5.6"])
97
  except RuntimeError:
98
  run([sys.executable, "-m", "pip", "install", "-q", "git+https://github.com/Wisk-inc/transformers.git"
99
  "@claude/linear-context-architecture-a35c6h#subdirectory=pluvium"])
pluvium/__init__.py CHANGED
@@ -11,7 +11,7 @@ whose forecasts Pluvium reads (:mod:`pluvium.upstream`).
11
  See :mod:`pluvium.model` for the architecture and its experimental components.
12
  """
13
 
14
- __version__ = "0.5.5"
15
 
16
  from .domain import Gazetteer, Geography, PointSet # noqa: E402
17
  from .model import PluviumConfig, PluviumModel # noqa: E402
 
11
  See :mod:`pluvium.model` for the architecture and its experimental components.
12
  """
13
 
14
+ __version__ = "0.5.6"
15
 
16
  from .domain import Gazetteer, Geography, PointSet # noqa: E402
17
  from .model import PluviumConfig, PluviumModel # noqa: E402
pluvium/program.py CHANGED
@@ -19,6 +19,8 @@ forecast the live forecast for every town, if the driving models ar
19
  Everything expensive is mirrored to the Hugging Face Hub as soon as it exists (``HF_REPO``, ``HF_TOKEN``), so a
20
  second session -- or a machine that was stopped -- pulls it back in minutes and continues where the last one ended.
21
  The time budget (``GPU_HOURS``) is wall clock from the program's start: training gets whatever the other stages leave.
 
 
22
  """
23
 
24
  from __future__ import annotations
@@ -100,11 +102,106 @@ def choose_years(gpu_hours: float, holdout: int = 2021) -> tuple:
100
  return tuple(range(holdout - n, holdout))
101
 
102
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
103
  class Program:
104
  def __init__(self, cfg: ProgramConfig) -> None:
105
  self.cfg = cfg
106
  self.started = time.time()
107
  self.deadline = self.started + cfg.gpu_hours * 3600.0
 
 
 
 
 
 
 
 
 
108
  self.root = Path(cfg.root).expanduser()
109
  self.root.mkdir(parents=True, exist_ok=True)
110
  self.state_path = self.root / "program.json"
@@ -121,6 +218,12 @@ class Program:
121
  """Hours left in the budget."""
122
  return (self.deadline - time.time()) / 3600.0
123
 
 
 
 
 
 
 
124
  def mark(self, stage: str, **info) -> None:
125
  self.state[stage] = {"done": time.strftime("%Y-%m-%d %H:%M"), **info}
126
  self.state_path.write_text(json.dumps(self.state, indent=1))
@@ -364,6 +467,13 @@ class Program:
364
  log(f"GPU: {props.name}, {props.total_memory / 2**30:.0f} GB ({how}); {os.cpu_count()} CPU cores")
365
  torch.backends.cuda.matmul.allow_tf32 = True
366
  torch.backends.cudnn.allow_tf32 = True
 
 
 
 
 
 
 
367
  geo, towns = self.geography()
368
  era5, cmorph, stations = self.data(geo, self.years + (cfg.holdout_year,))
369
  clim = self.climatology(geo, cmorph)
@@ -385,7 +495,7 @@ class Program:
385
  f"{test_starts.size} for scoring")
386
  model = build_model(geo, grids, cfg.size, st_points, **cfg.overrides)
387
  loss = PluviumLoss(states=model.config.states)
388
- reserve = 0.9 + 0.1 * cfg.gpu_hours / 24 # calibration, scoring, publishing, forecast
389
  train_hours_total = max(0.25, self.left() - reserve)
390
  share_a = train_hours_total * (0.85 if self.stage_b else 1.0)
391
  trainer = Trainer(model, loss, builder, train_starts, val_starts, self.root / "stage_a",
@@ -554,7 +664,7 @@ class Program:
554
  return None
555
  cut = int(starts.size * 0.75)
556
  train_b, held_b = starts[:cut], starts[cut:]
557
- hours_b = max(0.2, self.left() - (0.9 + 0.1 * self.cfg.gpu_hours / 24))
558
  cfg_b = self.train_cfg(hours_b, warmup_steps=50)
559
  cfg_b.lr = self.cfg.lr / 3
560
  # stage B's forecasts are built differently, so its step is sized afresh, unless a batch was given by hand
@@ -809,6 +919,13 @@ def retry_plan(vram: str, share: float) -> list:
809
 
810
  #: a run that prints nothing for this many minutes is taken to have hung; it is stopped and started again
811
  STALL_MINUTES = 120.0
 
 
 
 
 
 
 
812
  #: lines of a stopped run's output that are kept, on disk and on the Hub, so that its error can be read afterwards
813
  TAIL_LINES = 400
814
  _SECRET = re.compile(r"hf_[A-Za-z0-9]{20,}|pypi-[A-Za-z0-9_-]{20,}|gh[pousr]_[A-Za-z0-9]{20,}")
@@ -842,7 +959,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
842
  token=None, stall_minutes: float | None = None, api=None) -> int:
843
  """Run the program in a child process and keep it going. If the child stops with an error, is killed (the
844
  machine ran out of RAM, say), or prints nothing for ``stall_minutes``, it is started again from its last
845
- checkpoint with a smaller step each time (:func:`retry_plan`). A fresh process starts with a fresh CUDA context,
 
 
846
  which some GPU errors leave unusable; the data and the checkpoints are on disk, so the next try is training
847
  again within minutes.
848
 
@@ -857,6 +976,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
857
  tries = retry_plan(vram, share)
858
  limit = 60.0 * float(stall_minutes if stall_minutes is not None else os.environ.get("PLUVIUM_STALL_MINUTES") or STALL_MINUTES)
859
  state = {"child": None}
 
 
 
860
 
861
  def end_child() -> None:
862
  child = state["child"]
@@ -873,11 +995,12 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
873
  raise SystemExit(128 + signum)
874
 
875
  previous = signal.signal(signal.SIGTERM, stop)
876
- code = 1
877
  try:
878
- for k, (rule, cut, batch) in enumerate(tries):
879
- env = dict(os.environ, PLUVIUM_ATTEMPT=str(k + 1), PLUVIUM_VRAM=rule, PLUVIUM_VRAM_SHARE=str(cut),
880
- PYTHONUNBUFFERED="1")
 
881
  extra = ["--vram", rule, "--vram-share", str(cut)]
882
  if batch:
883
  env["PLUVIUM_BATCH"] = str(batch)
@@ -910,10 +1033,13 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
910
  settings = f"{rule}, share {cut}" + (f", batch {batch}" if batch else "")
911
  saved = save_attempt(root, repo, token, k + 1, f"attempt {k + 1} of {len(tries)} ({settings}): the program {what}",
912
  list(tail), api=api) if root else None
 
913
  code = code if code > 0 else 1
914
  if k + 1 < len(tries):
915
- then, then_cut, then_batch = tries[k + 1]
916
- how = (f"with one forecast per step (--batch {then_batch})" if then_batch else
 
 
917
  f"with each training step sized to {then_cut:.0%} of the GPU's memory" if then == "fill" else
918
  "with the batch rule of 0.4.2 (--vram fastest)")
919
  log(f"the program {what}" + (f"; the end of its output is in {saved}" if saved else "")
@@ -926,8 +1052,9 @@ def supervise(argv: list, share: float, command: list | None = None, vram: str =
926
  signal.signal(signal.SIGTERM, previous)
927
 
928
 
929
- __all__ = ["DEFAULT_REPO", "FILES_PER_COMMIT", "Program", "ProgramConfig", "RETRY_SHARE_DROP", "STALL_MINUTES",
930
- "choose_years", "main", "mirror_files", "retry_plan", "save_attempt", "supervise"]
 
931
 
932
  # ``python -m pluvium.program`` is how the notebook cells start a training run. Until 0.4.3 these two lines were
933
  # missing: the command imported this module and ended at once with exit code 0, having trained nothing.
 
19
  Everything expensive is mirrored to the Hugging Face Hub as soon as it exists (``HF_REPO``, ``HF_TOKEN``), so a
20
  second session -- or a machine that was stopped -- pulls it back in minutes and continues where the last one ended.
21
  The time budget (``GPU_HOURS``) is wall clock from the program's start: training gets whatever the other stages leave.
22
+ On a machine that stops a session after a fixed time (molab: 12 hours) the budget ends before that, so the run is
23
+ scored and published while the session is still up.
24
  """
25
 
26
  from __future__ import annotations
 
102
  return tuple(range(holdout - n, holdout))
103
 
104
 
105
+ # ------------------------------------------------------------------------------------- sessions --
106
+ #: molab, where these runs are made, stops a notebook 12 hours after it starts ("Notebooks can run for as long as 12
107
+ #: hours before molab shuts them down", docs.marimo.io/guides/molab). The first full run there trained for 11.6 of
108
+ #: its 24 hours and was stopped before it was scored and published.
109
+ MOLAB_HOURS = 12.0
110
+ #: how long before a session's end the run means to be done
111
+ SESSION_MARGIN_HOURS = 0.5
112
+ #: the least time kept for calibration, scoring, publishing and the forecast when the session has an end
113
+ SESSION_RESERVE_HOURS = 1.5
114
+ #: where molab keeps a notebook's packages (the tracebacks of the runs there show it)
115
+ MOLAB_PACKAGES = "/tmp/uv-venv"
116
+
117
+
118
+ def _proc_stat(pid: int) -> list:
119
+ """The fields of ``/proc/<pid>/stat`` after the command name, so that field 3 (the state) is the first."""
120
+ return Path(f"/proc/{pid}/stat").read_text().rsplit(")", 1)[1].split()
121
+
122
+
123
+ def _ancestors(most: int = 16) -> list:
124
+ """This process and the ones above it, nearest first: the program, its supervisor, the notebook, ..."""
125
+ out, pid = [], os.getpid()
126
+ while pid > 0 and len(out) < most:
127
+ out.append(pid)
128
+ try:
129
+ pid = int(_proc_stat(pid)[1])
130
+ except (OSError, ValueError, IndexError):
131
+ break
132
+ return out
133
+
134
+
135
+ def _began(pid: int) -> float | None:
136
+ """When a process started, in seconds since 1970, or None where there is no ``/proc``."""
137
+ try:
138
+ boot = next(float(line.split()[1]) for line in Path("/proc/stat").read_text().splitlines()
139
+ if line.startswith("btime"))
140
+ return boot + int(_proc_stat(pid)[19]) / os.sysconf("SC_CLK_TCK")
141
+ except (OSError, ValueError, IndexError, StopIteration):
142
+ return None
143
+
144
+
145
+ def _command(pid: int) -> str:
146
+ try:
147
+ return Path(f"/proc/{pid}/cmdline").read_bytes().replace(b"\0", b" ").decode(errors="replace")
148
+ except OSError:
149
+ return ""
150
+
151
+
152
+ def on_molab(env=None, places=None, commands=None) -> bool:
153
+ """Whether this runs in a molab notebook: the packages are where molab keeps them, a process above this one is
154
+ marimo, or marimo's settings are in the environment."""
155
+ env = os.environ if env is None else env
156
+ places = [sys.prefix, __file__] if places is None else places
157
+ commands = [_command(pid) for pid in _ancestors()[1:]] if commands is None else commands
158
+ return (any(str(p).startswith(MOLAB_PACKAGES) for p in places) or any("marimo" in c for c in commands)
159
+ or any(k.startswith("MARIMO") or "MOLAB" in k for k in env))
160
+
161
+
162
+ def session_limit(env=None) -> tuple:
163
+ """``(hours, why)``: how long one session on this machine may run, or ``(None, why)`` when nothing says it ends.
164
+ ``PLUVIUM_SESSION_HOURS`` says so (0 for no end); otherwise a molab notebook has 12 hours."""
165
+ env = os.environ if env is None else env
166
+ given = str(env.get("PLUVIUM_SESSION_HOURS", "")).strip()
167
+ if given:
168
+ hours, why = float(given), env.get("PLUVIUM_SESSION_WHY") or "PLUVIUM_SESSION_HOURS"
169
+ return (hours, why) if hours > 0 else (None, why)
170
+ if on_molab(env):
171
+ return MOLAB_HOURS, "molab stops a notebook after 12 hours"
172
+ return None, "nothing says this machine stops its sessions"
173
+
174
+
175
+ def session_start(hours: float, env=None, now: float | None = None) -> float:
176
+ """When this session began: ``PLUVIUM_SESSION_START`` when the supervisor passes it on, otherwise the earliest
177
+ start among this process and the ones above it (the notebook, its server, the container) that is less than
178
+ ``hours`` ago. A process older than that was there before the session."""
179
+ env = os.environ if env is None else env
180
+ now = time.time() if now is None else now
181
+ if env.get("PLUVIUM_SESSION_START"):
182
+ return float(env["PLUVIUM_SESSION_START"])
183
+ times = [t for t in map(_began, _ancestors()) if t is not None and now - hours * 3600.0 < t <= now]
184
+ return min(times, default=now)
185
+
186
+
187
+ def _clock(seconds: float) -> str:
188
+ return time.strftime("%H:%M", time.gmtime(seconds))
189
+
190
+
191
  class Program:
192
  def __init__(self, cfg: ProgramConfig) -> None:
193
  self.cfg = cfg
194
  self.started = time.time()
195
  self.deadline = self.started + cfg.gpu_hours * 3600.0
196
+ #: ``(began, hours, why)`` when this machine stops its sessions after a fixed time. The run is planned to be
197
+ #: published half an hour before then; a run stopped before that publishes nothing
198
+ self.session = None
199
+ hours, why = session_limit()
200
+ if hours:
201
+ began = session_start(hours)
202
+ end = began + (hours - SESSION_MARGIN_HOURS) * 3600.0
203
+ if end < self.deadline:
204
+ self.deadline, self.session = end, (began, hours, why)
205
  self.root = Path(cfg.root).expanduser()
206
  self.root.mkdir(parents=True, exist_ok=True)
207
  self.state_path = self.root / "program.json"
 
218
  """Hours left in the budget."""
219
  return (self.deadline - time.time()) / 3600.0
220
 
221
+ def reserve(self) -> float:
222
+ """Hours kept for calibration, scoring, publishing and the forecast. More when the session has an end: none
223
+ of these has run at full size on a GPU yet, and a session stopped in the middle of them publishes nothing."""
224
+ base = 0.9 + 0.1 * self.cfg.gpu_hours / 24
225
+ return max(base, SESSION_RESERVE_HOURS) if self.session else base
226
+
227
  def mark(self, stage: str, **info) -> None:
228
  self.state[stage] = {"done": time.strftime("%Y-%m-%d %H:%M"), **info}
229
  self.state_path.write_text(json.dumps(self.state, indent=1))
 
467
  log(f"GPU: {props.name}, {props.total_memory / 2**30:.0f} GB ({how}); {os.cpu_count()} CPU cores")
468
  torch.backends.cuda.matmul.allow_tf32 = True
469
  torch.backends.cudnn.allow_tf32 = True
470
+ if self.session:
471
+ began, hours, why = self.session
472
+ log(f"this session began at {_clock(began)} UTC and ends {hours:g} hours later ({why}): the network is "
473
+ f"trained, scored and published by {_clock(self.deadline)} UTC")
474
+ if self.left() < self.reserve() + 1.0:
475
+ log(f"only {self.left():.1f} hours of this session are left, so training gets little of them; a "
476
+ f"session started fresh has all {hours:g}")
477
  geo, towns = self.geography()
478
  era5, cmorph, stations = self.data(geo, self.years + (cfg.holdout_year,))
479
  clim = self.climatology(geo, cmorph)
 
495
  f"{test_starts.size} for scoring")
496
  model = build_model(geo, grids, cfg.size, st_points, **cfg.overrides)
497
  loss = PluviumLoss(states=model.config.states)
498
+ reserve = self.reserve() # calibration, scoring, publishing, forecast
499
  train_hours_total = max(0.25, self.left() - reserve)
500
  share_a = train_hours_total * (0.85 if self.stage_b else 1.0)
501
  trainer = Trainer(model, loss, builder, train_starts, val_starts, self.root / "stage_a",
 
664
  return None
665
  cut = int(starts.size * 0.75)
666
  train_b, held_b = starts[:cut], starts[cut:]
667
+ hours_b = max(0.2, self.left() - self.reserve())
668
  cfg_b = self.train_cfg(hours_b, warmup_steps=50)
669
  cfg_b.lr = self.cfg.lr / 3
670
  # stage B's forecasts are built differently, so its step is sized afresh, unless a batch was given by hand
 
919
 
920
  #: a run that prints nothing for this many minutes is taken to have hung; it is stopped and started again
921
  STALL_MINUTES = 120.0
922
+ #: words in a stopped run's last lines that say the step was too large for the GPU, so the next try is smaller
923
+ SIZE_SIGNS = ("out of memory", "OutOfMemoryError", "CUBLAS_STATUS", "CUDNN_STATUS", "CUDA error", "cudaError")
924
+
925
+
926
+ def about_size(tail) -> bool:
927
+ """Whether a stopped run's last lines say its step was too large for the GPU."""
928
+ return any(sign in line for line in list(tail)[-60:] for sign in SIZE_SIGNS)
929
  #: lines of a stopped run's output that are kept, on disk and on the Hub, so that its error can be read afterwards
930
  TAIL_LINES = 400
931
  _SECRET = re.compile(r"hf_[A-Za-z0-9]{20,}|pypi-[A-Za-z0-9_-]{20,}|gh[pousr]_[A-Za-z0-9]{20,}")
 
959
  token=None, stall_minutes: float | None = None, api=None) -> int:
960
  """Run the program in a child process and keep it going. If the child stops with an error, is killed (the
961
  machine ran out of RAM, say), or prints nothing for ``stall_minutes``, it is started again from its last
962
+ checkpoint, with a smaller step (:func:`retry_plan`) when the stop could have come from the step's size. Any other
963
+ error is tried again at the same size: 0.5.5 stopped on a bug that had nothing to do with memory, and every try
964
+ after it had less of the card. A fresh process starts with a fresh CUDA context,
965
  which some GPU errors leave unusable; the data and the checkpoints are on disk, so the next try is training
966
  again within minutes.
967
 
 
976
  tries = retry_plan(vram, share)
977
  limit = 60.0 * float(stall_minutes if stall_minutes is not None else os.environ.get("PLUVIUM_STALL_MINUTES") or STALL_MINUTES)
978
  state = {"child": None}
979
+ hours, why = session_limit() # found once, so that every try plans the same end
980
+ session = {"PLUVIUM_SESSION_HOURS": f"{hours or 0:g}", "PLUVIUM_SESSION_WHY": why,
981
+ "PLUVIUM_SESSION_START": f"{session_start(hours) if hours else time.time():.0f}"}
982
 
983
  def end_child() -> None:
984
  child = state["child"]
 
995
  raise SystemExit(128 + signum)
996
 
997
  previous = signal.signal(signal.SIGTERM, stop)
998
+ code, level = 1, 0
999
  try:
1000
+ for k in range(len(tries)):
1001
+ rule, cut, batch = tries[level]
1002
+ env = dict(os.environ, **session, PLUVIUM_ATTEMPT=str(k + 1), PLUVIUM_VRAM=rule,
1003
+ PLUVIUM_VRAM_SHARE=str(cut), PYTHONUNBUFFERED="1")
1004
  extra = ["--vram", rule, "--vram-share", str(cut)]
1005
  if batch:
1006
  env["PLUVIUM_BATCH"] = str(batch)
 
1033
  settings = f"{rule}, share {cut}" + (f", batch {batch}" if batch else "")
1034
  saved = save_attempt(root, repo, token, k + 1, f"attempt {k + 1} of {len(tries)} ({settings}): the program {what}",
1035
  list(tail), api=api) if root else None
1036
+ smaller = hung or code < 0 or about_size(tail)
1037
  code = code if code > 0 else 1
1038
  if k + 1 < len(tries):
1039
+ level = min(level + 1, len(tries) - 1) if smaller else level
1040
+ then, then_cut, then_batch = tries[level]
1041
+ how = ("with the same settings, since the error was not about memory" if not smaller else
1042
+ f"with one forecast per step (--batch {then_batch})" if then_batch else
1043
  f"with each training step sized to {then_cut:.0%} of the GPU's memory" if then == "fill" else
1044
  "with the batch rule of 0.4.2 (--vram fastest)")
1045
  log(f"the program {what}" + (f"; the end of its output is in {saved}" if saved else "")
 
1052
  signal.signal(signal.SIGTERM, previous)
1053
 
1054
 
1055
+ __all__ = ["DEFAULT_REPO", "FILES_PER_COMMIT", "MOLAB_HOURS", "Program", "ProgramConfig", "RETRY_SHARE_DROP",
1056
+ "SESSION_MARGIN_HOURS", "SESSION_RESERVE_HOURS", "STALL_MINUTES", "about_size", "choose_years", "main",
1057
+ "mirror_files", "on_molab", "retry_plan", "save_attempt", "session_limit", "session_start", "supervise"]
1058
 
1059
  # ``python -m pluvium.program`` is how the notebook cells start a training run. Until 0.4.3 these two lines were
1060
  # missing: the command imported this module and ended at once with exit code 0, having trained nothing.
pluvium/training/ema.py CHANGED
@@ -27,6 +27,8 @@ class EMA:
27
  d = min(self.decay, (1 + self.updates) / (10 + self.updates)) # warm-up: short memory at the start
28
  for k, v in model.state_dict().items():
29
  if k in self.shadow:
 
 
30
  self.shadow[k].mul_(d).add_(v.detach().float(), alpha=1 - d)
31
 
32
  @contextlib.contextmanager
@@ -43,8 +45,12 @@ class EMA:
43
  return {"decay": self.decay, "updates": self.updates, "shadow": self.shadow}
44
 
45
  def load_state_dict(self, state: dict) -> None:
 
 
 
46
  self.decay, self.updates = state["decay"], state["updates"]
47
- self.shadow = {k: v.float() for k, v in state["shadow"].items()}
 
48
 
49
 
50
  __all__ = ["EMA"]
 
27
  d = min(self.decay, (1 + self.updates) / (10 + self.updates)) # warm-up: short memory at the start
28
  for k, v in model.state_dict().items():
29
  if k in self.shadow:
30
+ if self.shadow[k].device != v.device: # the averages follow the weights
31
+ self.shadow[k] = self.shadow[k].to(v.device)
32
  self.shadow[k].mul_(d).add_(v.detach().float(), alpha=1 - d)
33
 
34
  @contextlib.contextmanager
 
45
  return {"decay": self.decay, "updates": self.updates, "shadow": self.shadow}
46
 
47
  def load_state_dict(self, state: dict) -> None:
48
+ """The averages are put where the model's weights are. A checkpoint is read onto the CPU, and until 0.5.6
49
+ they stayed there: a run resumed on a GPU stopped at its first step with "Expected all tensors to be on the
50
+ same device", after 11.6 hours of training had been saved."""
51
  self.decay, self.updates = state["decay"], state["updates"]
52
+ self.shadow = {k: v.float().to(self.shadow[k].device if k in self.shadow else v.device)
53
+ for k, v in state["shadow"].items()}
54
 
55
 
56
  __all__ = ["EMA"]
pyproject.toml CHANGED
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
4
 
5
  [project]
6
  name = "pluvium"
7
- version = "0.5.5"
8
  description = "A rain model that says when each parish might get rain, down to the exact town or city, and the time when rain is going to fall. Made for Jamaica first, works for any country."
9
  readme = "README.md"
10
  requires-python = ">=3.10"
 
4
 
5
  [project]
6
  name = "pluvium"
7
+ version = "0.5.6"
8
  description = "A rain model that says when each parish might get rain, down to the exact town or city, and the time when rain is going to fall. Made for Jamaica first, works for any country."
9
  readme = "README.md"
10
  requires-python = ">=3.10"
tests/test_fit.py CHANGED
@@ -248,11 +248,12 @@ def test_a_run_that_stops_is_started_again_with_smaller_steps_then_the_old_rule(
248
 
249
  log = tmp_path / "tries.txt"
250
 
251
- def child(succeed_at: int) -> list:
252
  return [sys.executable, "-c",
253
  "import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_VRAM'] + ' ' + "
254
  "os.environ['PLUVIUM_VRAM_SHARE'] + ' ' + os.environ.get('PLUVIUM_BATCH', '-') + ' ' + "
255
- f"' '.join(sys.argv[2:]) + '\\n'); sys.exit(0 if os.environ['PLUVIUM_ATTEMPT'] == '{succeed_at}' else 1)",
 
256
  str(log)]
257
 
258
  assert supervise(["--hours", "1"], 0.95, command=child(3)) == 0
@@ -264,6 +265,12 @@ def test_a_run_that_stops_is_started_again_with_smaller_steps_then_the_old_rule(
264
  assert supervise(["--hours", "1"], 0.95, command=child(9), vram="fastest") == 1
265
  assert log.read_text().splitlines() == ["fastest 0.95 - --hours 1 --vram fastest --vram-share 0.95",
266
  "fastest 0.95 1 --hours 1 --vram fastest --vram-share 0.95 --batch 1"]
 
 
 
 
 
 
267
  once = [sys.executable, "-c", "import sys; sys.exit(0)"]
268
  assert supervise([], 0.95, command=once) == 0 # a run that works is run once
269
  always = [sys.executable, "-c", "import sys; sys.exit(3)"]
@@ -402,3 +409,28 @@ def test_one_batch_that_cannot_be_trained_on_is_left_out_and_five_in_a_row_stop_
402
  with pytest.raises(ValueError, match="every batch fails"):
403
  always.run()
404
  assert always.step == 4 # four left out, the fifth stopped it
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
248
 
249
  log = tmp_path / "tries.txt"
250
 
251
+ def child(succeed_at: int, says: str = "torch.OutOfMemoryError: CUDA out of memory.") -> list:
252
  return [sys.executable, "-c",
253
  "import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_VRAM'] + ' ' + "
254
  "os.environ['PLUVIUM_VRAM_SHARE'] + ' ' + os.environ.get('PLUVIUM_BATCH', '-') + ' ' + "
255
+ f"' '.join(sys.argv[2:]) + '\\n'); ok = os.environ['PLUVIUM_ATTEMPT'] == '{succeed_at}'; "
256
+ f"print('' if ok else {says!r}); sys.exit(0 if ok else 1)",
257
  str(log)]
258
 
259
  assert supervise(["--hours", "1"], 0.95, command=child(3)) == 0
 
265
  assert supervise(["--hours", "1"], 0.95, command=child(9), vram="fastest") == 1
266
  assert log.read_text().splitlines() == ["fastest 0.95 - --hours 1 --vram fastest --vram-share 0.95",
267
  "fastest 0.95 1 --hours 1 --vram fastest --vram-share 0.95 --batch 1"]
268
+ log.unlink()
269
+ # an error that has nothing to do with memory is tried again at the same size. 0.5.5 stopped on such an error
270
+ # (the averaged weights on the CPU) and every try after it had less of the card
271
+ same = child(2, "RuntimeError: Expected all tensors to be on the same device, but found at least two devices")
272
+ assert supervise(["--hours", "1"], 0.95, command=same) == 0
273
+ assert log.read_text().splitlines() == ["fill 0.95 - --hours 1 --vram fill --vram-share 0.95"] * 2
274
  once = [sys.executable, "-c", "import sys; sys.exit(0)"]
275
  assert supervise([], 0.95, command=once) == 0 # a run that works is run once
276
  always = [sys.executable, "-c", "import sys; sys.exit(3)"]
 
409
  with pytest.raises(ValueError, match="every batch fails"):
410
  always.run()
411
  assert always.step == 4 # four left out, the fifth stopped it
412
+
413
+
414
+ def test_averaged_weights_read_from_a_checkpoint_follow_the_model_to_its_device(world, tmp_path):
415
+ """0.5.5 read a checkpoint onto the CPU and left the averaged weights there. On the GPU the resumed run stopped
416
+ at its first step (8,626, after 11.6 hours) with "Expected all tensors to be on the same device". The meta
417
+ device stands in for the GPU: it does not raise that error, but it shows where the averages are."""
418
+ import torch
419
+
420
+ from pluvium.training import EMA
421
+
422
+ model = torch.nn.Linear(3, 2)
423
+ saved = EMA(model, 0.9).state_dict() # on the CPU, as a checkpoint is read
424
+ model = model.to("meta")
425
+ ema = EMA(model, 0.9)
426
+ ema.load_state_dict(saved)
427
+ assert {v.device.type for v in ema.shadow.values()} == {"meta"} and ema.updates == 0
428
+ stray = EMA(torch.nn.Linear(3, 2), 0.9) # made on the CPU, the model moved after
429
+ stray.update(model)
430
+ assert {v.device.type for v in stray.shadow.values()} == {"meta"} and stray.updates == 1
431
+ # and a trainer that resumes keeps its averages where its model is
432
+ trainer, _ = make_trainer(world, tmp_path / "a", ema_decay=0.99)
433
+ trainer.run()
434
+ again, _ = make_trainer(world, tmp_path / "a", ema_decay=0.99)
435
+ where = {p.device for p in again.model.parameters()}
436
+ assert again.step == 2 and again.ema.updates == 2 and {v.device for v in again.ema.shadow.values()} == where
tests/test_program.py CHANGED
@@ -160,3 +160,54 @@ def test_the_background_fetch_does_not_keep_a_finished_run_alive(tmp_path):
160
  began = time.time()
161
  prog.stop_prefetch()
162
  assert not prog._prefetch.is_alive() and time.time() - began < 30
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
160
  began = time.time()
161
  prog.stop_prefetch()
162
  assert not prog._prefetch.is_alive() and time.time() - began < 30
163
+
164
+
165
+ def test_a_molab_session_is_planned_to_be_published_before_molab_stops_it(tmp_path, monkeypatch):
166
+ """molab stops a notebook 12 hours after it starts. The first full run there trained for 11.6 of its 24 hours and
167
+ was stopped before it was scored and published. A run there now plans to be done before the session ends."""
168
+ import time
169
+
170
+ from pluvium import program as P
171
+
172
+ for name in ("PLUVIUM_SESSION_HOURS", "PLUVIUM_SESSION_START", "PLUVIUM_SESSION_WHY"):
173
+ monkeypatch.delenv(name, raising=False)
174
+ molab = "/tmp/uv-venv/lib/python3.13/site-packages/pluvium/program.py" # as the tracebacks there show
175
+ assert P.on_molab({}, places=[molab], commands=[])
176
+ assert P.on_molab({}, places=["/usr/lib"], commands=["/usr/bin/python -m marimo edit notebook.py"])
177
+ assert P.on_molab({"MARIMO_MANAGE_SCRIPT_METADATA": "true"}, places=["/usr/lib"], commands=[])
178
+ assert not P.on_molab({"HOME": "/root"}, places=["/usr/lib"], commands=["bash", "python -m pluvium.program"])
179
+ assert P.session_limit({"PLUVIUM_SESSION_HOURS": "6"}) == (6.0, "PLUVIUM_SESSION_HOURS")
180
+ assert P.session_limit({"PLUVIUM_SESSION_HOURS": "0"})[0] is None # 0: no end, even on molab
181
+ began = time.time() - 40 * 60
182
+ assert P.session_start(12, {"PLUVIUM_SESSION_START": str(began)}) == began
183
+ found = P.session_start(12, {}) # the earliest process of this session
184
+ assert time.time() - 12 * 3600 < found <= time.time()
185
+
186
+ monkeypatch.setattr(P, "on_molab", lambda env=None, places=None, commands=None: True)
187
+ assert P.session_limit({}) == (12.0, "molab stops a notebook after 12 hours")
188
+ # the notebook started 40 minutes before the program: published 11.5 hours after that, not 24 hours after
189
+ monkeypatch.setenv("PLUVIUM_SESSION_START", str(began))
190
+ prog = Program(ProgramConfig(root=str(tmp_path), hub_repo=None, gpu_hours=24))
191
+ assert prog.session is not None and abs(prog.deadline - (began + 11.5 * 3600)) < 1
192
+ assert 10.7 < prog.left() < 10.9 and prog.reserve() == 1.5
193
+ # no end known: 24 hours from the program's start, as before
194
+ monkeypatch.setattr(P, "on_molab", lambda env=None, places=None, commands=None: False)
195
+ prog = Program(ProgramConfig(root=str(tmp_path), hub_repo=None, gpu_hours=24))
196
+ assert prog.session is None and 23.9 < prog.left() <= 24 and prog.reserve() == 1.0
197
+
198
+
199
+ def test_the_supervisor_finds_the_session_once_and_every_try_gets_it(tmp_path, monkeypatch):
200
+ import sys
201
+ import time
202
+
203
+ from pluvium.program import supervise
204
+
205
+ seen = tmp_path / "env.txt"
206
+ child = [sys.executable, "-c", "import os, sys; open(sys.argv[1], 'a').write(os.environ['PLUVIUM_SESSION_HOURS'] "
207
+ "+ ' ' + os.environ['PLUVIUM_SESSION_START'] + '\\n'); sys.exit(0 if os.environ['PLUVIUM_ATTEMPT'] == '2' "
208
+ "else 1)", str(seen)]
209
+ monkeypatch.delenv("PLUVIUM_SESSION_START", raising=False)
210
+ monkeypatch.setenv("PLUVIUM_SESSION_HOURS", "12")
211
+ assert supervise([], 0.95, command=child) == 0
212
+ first, second = seen.read_text().splitlines()
213
+ assert first == second and first.split()[0] == "12" and float(first.split()[1]) <= time.time()