Smooth Levels from Sparse, Gappy Panels

Masking, Weighting and a Curvature Penalty in PyTorch

Real-world panel data is rarely tidy. Local price indices, sensor networks and small-area statistics all produce many short time series, one per location, that are noisy, full of gaps, and sometimes missing a whole series. Often what we actually want is not the noisy reading but a prevailing level: a smooth estimate of each series at each location, including the periods where nothing was observed.

This post walks through a small PyTorch network that does this for two related targets at once, and tests it on air-quality data from Beijing that we deliberately break. Everything is open:

• 🎮 Try it: Space: sparse-panel-demo
• 🧠 Model code and recipe: prteek/sparse-panel-demo
• 📊 Data: datasets/prteek/sparse-panel-demo
• 📄 Paper: PM2.5, PM10 ~ Station + Time, Analysis on alphaXiv
• 💻 Code: github.com/prteek/sparse-panel-modelling

TL;DR

• Model the two targets jointly, and treat a missing target as masked in the loss rather than dropping the row.
• Weight each observation by how many raw readings went into it.
• Make smoothness explicit with fixed Gaussian time bumps and a curvature penalty over every day, observed or not.
• On deliberately thinned Beijing PM2.5/PM10 data, the network carries the level through long gaps and very sparse stations far better than a moving average, and the joint model recovers a series it never saw. The price: where data is plentiful, separate single-target models are slightly more accurate.

The problem

Picture a panel where every location has its own time series of two related quantities: two nearby sensors, or rents and sale prices in the same neighbourhood. Each observation is a mean over some raw readings, so it is noisy. Some locations observe only one of the two series, some observe neither for weeks at a time, and some never observe one of them at all.

Smoothing each series on its own breaks down exactly where you need it most: long gaps and very noisy series. To do better, the model has to borrow information, and this kind of panel offers two sources:

• Across locations: nearby or similar locations are related, so a sparse series can lean on its peers.
• Across targets: the two quantities are related, so a day where only one was observed still says something about the other.

The network below uses both, and the experiment measures how much each one helps.

The data: breaking a good dataset on purpose

To evaluate on sparse data while still knowing the truth, we start from a very complete dataset and thin it. The UCI Beijing Multi-Site Air Quality dataset has hourly PM2.5 and PM10 at 12 stations from March 2013 to February 2017, with more than 99% of station-days covered. The two pollutants are closely related, which makes them a good fit for the two-target setting.

We work with daily means and keep $nn$, the number of hours behind each mean, as a measure of how reliable it is. Then we thin the data:

• For each station and day, keep a random Poisson(1.2) number of hours, drawn separately for PM2.5 and PM10. Often only one of the two is observed on a given day.
• From each station and pollutant series, remove two blocks of 20–60 consecutive days.

After thinning, both pollutants are observed on 44% of station-days and exactly one of them on 45%.

The ground truth is a 31-day centred moving average of the daily (log) means of the full data.

Dongsi, PM2.5, one year. Grey: full-data daily means. Dots: sparse readings (size = hours). Black: the 31-day moving average of the full data, our target.

Daily pollution easily varies by a factor of two from one day to the next with the weather, and a mean of one or two hours adds a lot of noise on top. The prevailing level is the slow black line.

The model

Architecture

Each row is one station-day with up to two targets, $y_1=\log \text{PM2.5}y\_1\; =\; \backslash log\; \backslash text\{PM2.5\}$ and $y_2=\log \text{PM10}y\_2\; =\; \backslash log\; \backslash text\{PM10\}$ (standardised). The network maps (station, day) to a level for both:

• Location is an 8-dim area embedding plus a station offset that starts near zero and is penalised, so a station with little data falls back to its area.
• Time enters only through fixed Gaussian bumps, $b_k(t)=\exp \{-\frac{1}{2}((t-c_k)\mathrm{/}w{)}^2\}b\_k(t)\; =\; \backslash exp\backslash \{-\backslash tfrac12\; ((t\; -\; c\_k)/w)^2\backslash \}$, with centres $c_{k}c\_k$ every $ww$ days. Neighbouring days get almost identical bump vectors, so the level cannot jump from one day to the next.
• A shared trunk of two 64-unit SiLU layers feeds two linear heads, one per pollutant. Everything before the heads is shared, so a day that observed only PM2.5 still trains everything PM10 depends on.

In PyTorch this is small:

class TimeBasis(nn.Module):
    def __init__(self, n_days: int, spacing: int):
        super().__init__()
        self.register_buffer("centres", torch.arange(-spacing, n_days + 2 * spacing, spacing, dtype=torch.float32))
        self.width = float(spacing)

    def forward(self, day):
        return torch.exp(-0.5 * ((day.float()[:, None] - self.centres[None, :]) / self.width) ** 2)


class PrevailingNet(nn.Module):
    ...
    def forward(self, x):
        loc = self.area(x["area"]) + self.station(x["station"])
        h = self.trunk(torch.cat([loc, self.time(x["day"])], dim=1))
        return torch.cat([head(h) for head in self.heads], dim=1)

Loss: mask, weight, smooth

Each daily observation is a mean of $nn$ hourly readings, so its variance around the level is the reading variance divided by $nn$, plus a floor ${\tau }^2\backslash tau^2$ for day-to-day movements the level should not follow. For target $jj$:

$${\mathcal{L}}_{\text{fit}}=\frac{1}{\sum m_{ij}}\sum _{i,j}{m}_{ij}[\frac{1}{2}\log v_{ij}+\frac{(y_{ij}-{\mu }_{ij}{)}^2}{2v_{ij}}],v_{ij}=\frac{{\sigma }_j^2}{n_{ij}}+{\tau }_j^2\backslash mathcal\{L\}\_\{\backslash text\{fit\}\}\; =\; \backslash frac\{1\}\{\backslash sum\; m\_\{ij\}\}\; \backslash sum\_\{i,j\}\; m\_\{ij\}\; \backslash left[\; \backslash tfrac12\; \backslash log\; v\_\{ij\}\; +\; \backslash frac\{(y\_\{ij\}\; -\; \backslash mu\_\{ij\})^2\}\{2\; v\_\{ij\}\}\; \backslash right],\; \backslash qquad\; v\_\{ij\}\; =\; \backslash frac\{\backslash sigma\_j^2\}\{n\_\{ij\}\}\; +\; \backslash tau\_j^2$$

Here $m_{ij}m\_\{ij\}$ is 1 if target $jj$ was observed on row $ii$ and 0 otherwise: a missing target is masked, not dropped. ${\sigma }_j\backslash sigma\_j$ and ${\tau }_j\backslash tau\_j$ are learned per target, and a day with 24 readings counts far more than a day with one.

Smoothness comes from a penalty on the curvature of every station's predicted series over the whole day grid, observed or not:

$$\mathcal{L}={\mathcal{L}}_{\text{fit}}+\lambda \overline{{({\mu }_{t+1}-2{\mu }_t+{\mu }_{t-1})}^2}+\gamma \overline{\parallel e_{\text{station}}{\parallel }^2}\backslash mathcal\{L\}\; =\; \backslash mathcal\{L\}\_\{\backslash text\{fit\}\}\; +\; \backslash lambda\; \backslash ,\; \backslash overline\{\backslash left(\backslash mu\_\{t+1\}\; -\; 2\backslash mu\_t\; +\; \backslash mu\_\{t-1\}\backslash right)^2\}\; +\; \backslash gamma\; \backslash ,\; \backslash overline\{\backslash lVert\; e\_\{\backslash text\{station\}\}\; \backslash rVert^2\}$$

Because the network produces predictions for days without data too, the penalty controls the curve inside gaps and at the edges of the series. The whole loss is a few lines:

def nll(mu, model, y, mask, n):
    var = torch.exp(2 * model.log_sigma) / n + torch.exp(2 * model.log_tau)
    out = 0.5 * (torch.log(var) + (y - mu) ** 2 / var)
    return (out * mask).sum() / mask.sum()


def roughness(model):
    mu = model.all_series()  # (stations, days, targets), every day on the grid
    return (mu[:, 2:] - 2 * mu[:, 1:-1] + mu[:, :-2]).pow(2).mean()


loss = (nll(model(x), model, y, mask, n) + cfg["rough"] * roughness(model)
        + cfg["station_l2"] * model.station.weight.pow(2).mean())

Training is full-batch Adam with a cosine schedule. On the 12-station, four-year panel a fit takes a few seconds on a CPU.

The experiment

To test the worst cases, two stations are degraded further before fitting:

• A dropped series: every PM10 reading at Dingling is removed; only its PM2.5 remains.
• A very sparse station: Huairou keeps only 15% of its (already sparse) days.

Three estimators are fitted to the same data:

1. Joint: the network above, modelling both pollutants together.
2. Single-target: the same network trained on one pollutant at a time (two separate models).
3. ±15-day mean: for each station and pollutant, the hour-weighted mean of the sparse readings within 15 days; where a station has no readings at all, the same within its area.

Choosing the settings

The curvature weight $\lambda \backslash lambda$, bump spacing $ww$ and number of training steps are chosen by look-ahead cross-validation: 12 sequential folds, 30 days apart over the final year. Each fold trains on everything up to its cutoff and is scored on the next 15 days of sparse readings, with every station hidden at once. The winner is $\lambda ={10}^5\backslash lambda\; =\; 10^5$, $w=30w\; =\; 30$ days and 100 training steps.

The validation scheme matters more than it looks. We also tried hiding one station per area while its neighbours kept reporting:

• Every station hidden asks what persists into the near future. Episodes of two to three weeks don't carry past the cutoff, so a curve that chases them is punished. This defines the level as the part of the series that persists.
• One station hidden asks what this station is doing right now, given its neighbours right now. That rewards following shared short episodes and picks a much rougher curve.

Neither is wrong, but a prevailing level is the first one.

Results

Error is the mean absolute log error against the 31-day moving average of the full data (0.15 ≈ 15%). Every station-day and pollutant falls into one group: the dropped series (Dingling PM10), the very sparse station (Huairou), and, for every other station, observed days, unobserved days outside the gap blocks, the subset of those where the other pollutant was observed, and days inside a long gap.

For reference, a raw sparse reading on an observed day is off by roughly 0.7–0.8, so all three estimators remove most of the noise. The differences are in the hard cases.

Long gaps. The networks carry the level through (joint 0.156, single-target 0.144) while the ±15-day mean runs out of data (0.292). Once its window empties, the moving average either stops or swings on the few readings at the edges; the networks carry on smoothly.

A very sparse station. With 15% of its days, Huairou is too thin for a moving average (0.496). Both networks do as well as on regular stations (0.152 and 0.141), because strong smoothing and the shared area pattern fill in what the station lacks.

A dropped series. This is where modelling both pollutants together pays off. With no PM10 at Dingling at all, the joint model infers it from Dingling's PM2.5 (0.144). The single-target PM10 model has nothing station-specific to go on and returns the suburban area's level, which is too high (0.182). The joint model also beats the area moving average (0.156), though its curve runs a little low and misses some peaks.

Everywhere else, sharing costs a little. On observed, unobserved and long-gap days, the single-target models are slightly more accurate (0.147 vs 0.154 on observed days). When a station has its own data for a pollutant, borrowing from the other adds little and forces one shared trunk to serve both.

With dense data, simple smoothing is hard to beat. On observed days the ±15-day mean (0.153) is on par with the networks, helped by the fact that its 31-day window matches the benchmark's. Over a well-observed year at Dongsi, the moving average tracks the level closely, while the network curves follow the seasons but flatten shorter episodes such as the December 2015 peak.

A bias to fix

The network's level sits about 0.11 below the moving average on the log scale (roughly 11%). It is fitted to logs of means over one or two hours, and the log of a mean over few readings is lower on average than the log of a mean over 24 (Jensen's inequality). The usual lognormal correction $+\frac{1}{2}{\sigma }^2+\backslash tfrac12\backslash sigma^2$ overshoots by about as much the other way, because the reading-level spread mixes within-day and between-day variation.

Lessons learned

• Mask, don't drop. Treating a missing target as masked in the loss, instead of discarding the row, is what lets one target fill in for the other. It paid off for an entirely missing series.
• Weight by information. A variance of ${\sigma }^2\mathrm{/}n+{\tau }^2\backslash sigma^2/n\; +\; \backslash tau^2$ lets a 24-hour mean outweigh a single reading and stops the model chasing day-to-day noise.
• Make smoothness explicit. Smooth time bumps plus a curvature penalty over every day, observed or not, keep the level stable through gaps and at the edges.
• Bumps alone don't make a curve smooth where there is no data. Without the curvature penalty, filling gaps inside the data improves slightly (0.13 vs 0.15), but 15 days past the end the error jumps from 0.25 to 0.5–2.9: the terminal bumps have nothing pinning them down. Training length becomes the only brake, and 1,000 steps instead of 100 nearly doubles the error.
• Share across locations with offsets. A thin station falls back to its area instead of doing nothing.
• Know what the network cannot see. Pollution across Beijing moves together day to day because the weather is citywide. Moving averages pick that up from same-day readings; the network's only notion of time is a smooth curve. A shared day-level effect is the obvious next step.
• Decide the smoothness, don't tune it. Validated on noisy readings, a weaker penalty always looks slightly better because it follows real short episodes. How smooth the level should be is a choice about what the level means. Freeze it and let cross-validation pick the rest.
• A mixed-effects GAM is a strong alternative. Area curves with station deviations shrunk towards them, fitted jointly to both pollutants, beat the network in every group when the smoothness is set by hand. Left to REML it follows weekly swings and does much worse. Tuned with the same look-ahead CV as the network, it is pushed to a nearly flat curve and does worse than the network: the CV only checks the 15 days just past the end of the data, where any flexible spline goes wrong, so it picks the stiffest one. The tuning target, not the model, sets the smoothness.

Try it on your own panel

The model repo holds code and a training recipe rather than trained weights: location embeddings and the time basis are specific to one panel and date range, so you fit it to your own data. Input is long format, one row per location × date × target with data (station, area, date, pollutant, value, n).

from model import Encoding, fit, predict

CFG = dict(rough=1e5, spacing=30, lr=5e-3, hidden=64, weight_decay=1e-4,
           station_l2=1e-3, schedule=1000, seed=0)

enc = Encoding.fit(panel, end=panel["date"].max())
model = fit(enc, enc.wide(panel), CFG, steps=100)  # use=["PM2.5"] for a single-target model
level = predict(model, enc, enc.grid())             # every location × day, observed or not

example.py in the model repo runs this end to end on a small synthetic panel. Or open the Space, choose how many readings to keep and which stations to look at, and compare the joint network, single-target networks and the ±15-day mean against the true level.

References

• Song Chen (2017). Beijing Multi-Site Air Quality. UCI Machine Learning Repository. doi:10.24432/C5RK5G
• Zhang et al. (2017). Cautionary tales on air-quality improvement in Beijing. Proc. R. Soc. A 473(2205). doi:10.1098/rspa.2017.0457
• Paszke et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. NeurIPS 32.
• Elfwing, Uchibe & Doya (2018). Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks 107. doi:10.1016/j.neunet.2017.12.012
• Eilers & Marx (1996). Flexible smoothing with B-splines and penalties. Statistical Science 11(2). doi:10.1214/ss/1038425655
• Wood (2017). Generalized Additive Models: An Introduction with R, 2nd ed. CRC. doi:10.1201/9781315370279

This work was developed with AI assistance.