File size: 5,189 Bytes
b84c286 5bd97d2 b84c286 cad1287 b84c286 2d97ae0 b84c286 2d97ae0 b84c286 2d97ae0 b84c286 2d97ae0 b84c286 5bd97d2 b84c286 2d97ae0 b84c286 5bd97d2 b84c286 5bd97d2 b84c286 5bd97d2 b84c286 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | ---
license: cc-by-4.0
library_name: pytorch
tags:
- time-series
- panel-data
- missing-data
- smoothing
- imputation
- air-quality
- pytorch
datasets:
- prteek/sparse-panel-demo
---
# Prevailing level network for sparse, gappy panels
A small PyTorch network that estimates a smooth **prevailing level** for every location and day of a panel,
for two related quantities at once, when the observations are noisy daily means with long gaps and some series
missing entirely. It was developed for the blog post
[Smooth Levels from Sparse, Gappy Panels](https://huggingface.co/blog/prteek/sparse-panel-modelling) and powers the
[sparse panel demo Space](https://huggingface.co/spaces/prteek/sparse-panel-demo).
**This repo holds the code and the training recipe, not trained weights.** The network learns an embedding for
each location and a time basis over a fixed date range, so a fitted model only applies to the panel it was fitted
on. To use it, fit it to your own panel: on the 12-station, four-year Beijing data a fit takes a few seconds on a
CPU.
## Files
- `model.py`: the encoding, the network, the loss and `fit` / `predict`
- `example.py`: fits the network to a small synthetic panel and predicts every cell (runs as is)
## Model
Each row is one location and day, with up to two targets \\( y_1, y_2 \\) (log values, standardised).
$$
\begin{aligned}
x &= \big[\, e_{\text{area}} + e_{\text{station}},\ b(t) \,\big] \\
h &= \operatorname{SiLU}\!\big(W_2 \operatorname{SiLU}(W_1 x)\big) \\
\mu_1 &= w_1^\top h, \quad \mu_2 = w_2^\top h
\end{aligned}
$$
- **Location**: an 8-dim area embedding plus a station offset that starts near zero and is penalised, so a
station with little data falls back to its area.
- **Time**: fixed Gaussian bumps \\( b_k(t) = \exp\big(-\tfrac12((t - c_k)/w)^2\big) \\) with centres every \\( w \\) days, so the
level is a smooth function of time.
- **Shared trunk, two heads**: two layers of 64 with SiLU, then one linear head per target. A day where only one
target was observed still trains everything the other depends on.
**Loss.** An observed value is a mean of \\( n \\) readings, so its variance around the level is \\( \sigma_j^2 / n + \tau_j^2 \\),
with \\( \sigma_j \\), \\( \tau_j \\) learned per target. The Gaussian negative log-likelihood is averaged over observed cells
only: a missing target is **masked**, not dropped. Added to it:
- a **curvature penalty** \\( \lambda \cdot \overline{(\mu_{t+1} - 2\mu_t + \mu_{t-1})^2} \\) over every location's
predicted series on the **whole** day grid, observed or not, which keeps the level smooth through gaps;
- a small L2 penalty on the station offsets.
## Training recipe
Full-batch Adam with a cosine learning-rate schedule. The settings below were chosen by look-ahead
cross-validation in the blog post:
| Setting | Value |
|---|---|
| curvature penalty \\( \lambda \\) (`rough`) | 1e5 |
| bump spacing \\( w \\) (`spacing`) | 30 days |
| hidden width | 64 |
| learning rate | 5e-3 |
| weight decay | 1e-4 |
| station offset L2 (`station_l2`) | 1e-3 |
| schedule length | 1000 |
| training steps | 100 |
| seed | 0 |
For a single-target model, pass `use=["PM2.5"]` (or the other target) to `fit`.
## Input format
Long format, one row per location × date × target that has data:
| column | meaning |
|---|---|
| `station` | location id |
| `area` | group the location belongs to (it falls back to this when its own data is thin) |
| `date` | daily timestamp |
| `pollutant` | target name, one of `model.TARGETS` |
| `value` | the daily mean (> 0; modelled on the log scale) |
| `n` | number of readings behind that mean (weights the observation) |
The two target names are set in `model.TARGETS` (`["PM2.5", "PM10"]`); change them for other quantities.
```python
from model import Encoding, fit, predict
enc = Encoding.fit(panel, end=panel["date"].max())
model = fit(enc, enc.wide(panel), CFG, steps=100)
level = predict(model, enc, enc.grid()) # every location x day, observed or not
```
## Limitations
- **Not transferable.** Location embeddings and the time basis are specific to one panel and date range. Refit
for new locations or dates.
- **Smoothness is a choice.** Cross-validation that scores against noisy readings rewards following short
episodes, so it tends to under-smooth; the blog post discusses this. Treat `rough` and `spacing` as decisions about
how smooth the level should be.
- **Log-scale bias.** The model is fitted to logs of means over few readings, so its level sits slightly below a
moving average of full data (Jensen's inequality).
- **A GAM is a strong alternative.** In the blog post, a mixed-effects GAM with hand-set smoothness does as well or
better.
- **Memory.** Training is full batch and the penalty covers every location × day, so memory grows with panel size.
## Data used in the blog post
Beijing Multi-Site Air Quality, Song Chen (2017), UCI Machine Learning Repository,
[doi:10.24432/C5RK5G](https://doi.org/10.24432/C5RK5G), CC BY 4.0. The demo's sparse readings and predictions
are in the dataset [prteek/sparse-panel-demo](https://huggingface.co/datasets/prteek/sparse-panel-demo).
|