Publish DINCAE reproduction

.gitattributes

@@ -1,35 +1,3 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,117 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- OneScience
+- Earth-science
+- sea-surface-temperature
+- gap-filling
+frameworks: PyTorch
+---
+
+<p align="center"><strong><span style="font-size: 30px;">DINCAE</span></strong></p>
+
+# Model Introduction
+
+DINCAE reconstructs daily sea-surface temperature obscured by clouds with a probabilistic convolutional autoencoder and provides pixel-wise reconstruction uncertainty.
+
+Paper: DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations  
+https://doi.org/10.5194/gmd-13-1609-2020
+
+# Model Description
+
+The method was proposed by research teams from the University of Liege and the National Institute of Biology in Slovenia. The paper trains and validates the method on AVHRR Pathfinder daily sea-surface temperature from 1985-2009. The model reconstructs cloud-obscured SST while estimating reconstruction-error variance.
+
+# Use Cases
+
+| Use Case | Description |
+| :---: | :--- |
+| SST gap reconstruction | Reconstruct cloud-obscured SST from current, previous-day, and next-day observations. |
+| Probabilistic error estimation | Jointly output reconstruction means and pixel-wise error variances. |
+| Local engineering validation | Validate training, inference, and evaluation with structured data at reduced spatial scale. |
+| ModelScope/OneCode execution | Validate structured data, training, inference, SST reconstruction metrics, and visualization in ModelScope or OneCode environments. |
+| Multi-GPU training | Validate distributed training and the checkpoint workflow through `torchrun`. |
+
+# Usage
+
+## 1.OneCode
+
+[Try one-click AI4S programming in OneCode](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
+
+## 2. Download and Installation
+
+```bash
+hf download OneScience-Group/DINCAE --local-dir ./DINCAE
+cd DINCAE
+```
+
+### Environment Dependencies
+
+**Hardware Requirements**
+
+- A GPU or DCU is recommended.
+- A CPU can be used for connectivity validation with the default small-sample configuration.
+- DCU users must install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.
+
+### Training Data
+
+```bash
+python scripts/fake_data.py
+```
+
+The paper data contain 5,266 daily `112×112` AVHRR SST time steps, with current, previous-day, and next-day observations, precision, coordinates, and seasonal inputs. The default uses eight structured synthetic `32×32` time steps for efficient CPU validation, while paper dimensions and parameters remain recorded under `paper_model`. The synthetic data validate engineering only and do not represent the real AVHRR distribution, training scale, or paper performance.
+
+### Training
+
+For single-GPU training, use:
+
+```bash
+python scripts/train.py
+```
+
+For multi-GPU training, use:
+
+```bash
+torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
+```
+
+Training uses random cloud masks and masked Gaussian NLL; the default reduces grid size, samples, convolution widths, bottleneck size, and epochs while retaining the paper's `112×112` setup under `paper_model`. Formal experiments require complete AVHRR data and paper-scale training, with artifacts saved to:
+
+```text
+result/checkpoints/dincae.pt
+result/training/metrics.json
+```
+
+### Trained Weights
+
+The paper does not provide directly loadable official model weights, and this repository bundles no weights under `weight/`. The locally trained checkpoint is saved to `result/checkpoints/dincae.pt` and must not be represented as an official pretrained weight.
+
+### Inference
+
+```bash
+python scripts/inference.py
+```
+
+This writes reconstructed means, variances, targets, missing masks, and dates to `result/output/predictions.npz`.
+
+### Evaluation and Visualization
+
+```bash
+python scripts/result.py
+```
+
+Evaluation reports RMSE, centered RMSE (CRMSE), bias, standardized-residual calibration, and a DINEOF-like iterative rank-13 baseline. Outputs are `result/evaluation/metrics.json` and `result/evaluation/comparison.png`. Synthetic-data metrics demonstrate connectivity only and are not paper results.
+
+# Official OneScience Information
+
+| Platform | OneScience | OneSkills |
+| --- | --- | --- |
+| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
+| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
+
+# Citation and License
+
+This repository is an independent engineering reproduction of the public DINCAE specifications.
+
+Use of this repository's code, official model weights, and data remains subject to the licenses and terms of their respective projects.

README_zh.md

@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+tags:
+- OneScience
+- 地球科学
+- 海表温度
+- 缺测重建
+frameworks: PyTorch
+---
+
+<p align="center"><strong><span style="font-size: 30px;">DINCAE</span></strong></p>
+
+# 模型介绍
+
+DINCAE 通过概率卷积自编码器重建云遮挡造成的逐日海表温度缺测，并提供像素级重建不确定性。
+
+论文：DINCAE 1.0: a convolutional neural network with error estimates to reconstruct sea surface temperature satellite observations  
+https://doi.org/10.5194/gmd-13-1609-2020
+
+# 模型描述
+
+该方法由列日大学和斯洛文尼亚国家生物研究所的研究团队提出。论文使用 1985-2009 年 AVHRR Pathfinder 逐日海表温度资料开展训练与验证。模型适用于重建云遮挡区域的 SST，并同时估计重建误差方差。
+
+# 适用场景
+
+| 场景 | 说明 |
+| :---: | :--- |
+| 海表温度缺测重建 | 根据当前、前一日和后一日观测重建云遮挡 SST。 |
+| 概率误差估计 | 联合输出重建均值和像素级误差方差。 |
+| 本地工程验证 | 使用缩小空间尺度的结构化虚拟数据验证训练、推理和评估流程。 |
+| ModelScope/OneCode 运行 | 在 ModelScope 或 OneCode 环境中验证结构化数据、训练、推理、SST 重建指标和可视化流程。 |
+| 多卡训练 | 通过 `torchrun` 验证分布式训练和 checkpoint 流程。 |
+
+# 使用说明
+
+## 1.OneCode
+
+可通过 OneCode 在线环境体验智能化一键式 AI4S 编程：
+
+[点击体验智能化一键式 AI4S 编程](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
+
+## 2.下载安装
+
+```bash
+modelscope download --model OneScience/DINCAE --local_dir ./DINCAE
+cd DINCAE
+```
+
+### 环境依赖
+
+**硬件要求**
+
+- 推荐使用 GPU 或 DCU 运行。
+- CPU 可用于默认小样本配置的连通性验证。
+- DCU 用户需预先安装 DTK，建议使用 DTK 25.04.2 以上版本或与当前集群匹配的 OneScience 推荐版本。
+
+### 训练数据介绍
+
+```bash
+python scripts/fake_data.py
+```
+
+论文数据包含 5266 个 `112×112` AVHRR 逐日 SST 时刻，模型输入当前、前一日和后一日观测及精度、坐标和季节信息。本仓库默认使用 8 个 `32×32` 结构化虚拟时刻以控制 CPU 验证成本，论文真实网格和模型参数单独记录在 `paper_model`。虚拟数据仅用于验证工程流程，不代表 AVHRR 的真实数据分布、训练规模或论文性能。
+
+### 训练
+
+单卡训练可使用：
+
+```bash
+python scripts/train.py
+```
+
+多卡训练可使用：
+
+```bash
+torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
+```
+
+训练使用随机云掩码和 masked Gaussian NLL；默认缩小网格、样本数、卷积宽度、瓶颈和训练轮数，论文配置保留 `112×112` 网格与 1000 epochs。正式实验需要完整 AVHRR 数据和论文规模模型，训练产物保存到：
+
+```text
+result/checkpoints/dincae.pt
+result/training/metrics.json
+```
+
+### 训练权重
+
+论文未提供可直接加载的官方模型权重，本仓库不在 `weight/` 中内置权重。本地训练生成的 checkpoint 保存到 `result/checkpoints/dincae.pt`，不得描述为官方预训练权重。
+
+### 推理
+
+```bash
+python scripts/inference.py
+```
+
+输出 `result/output/predictions.npz`，包含重建均值、方差、目标、缺测掩膜和日期。
+
+### 评估和可视化
+
+```bash
+python scripts/result.py
+```
+
+评估输出 RMSE、中心化 RMSE（CRMSE）、bias、标准化残差 calibration，以及 DINEOF-like rank-13 迭代低秩基线。结果写入 `result/evaluation/metrics.json` 和 `result/evaluation/comparison.png`。少样本指标只证明流程连通，不代表论文性能。
+
+# OneScience官方信息
+
+| 平台 | OneScience 主仓库 | Skills 仓库 |
+| --- | --- | --- |
+| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
+| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
+
+# 引用与许可证
+
+本仓库为 DINCAE 公开规格的独立工程复现版本。
+
+本仓库代码、官方模型权重和数据的使用仍应以各自项目中的许可证及使用条款为准。

conf/config.yaml

@@ -0,0 +1,46 @@
+seed: 2026
+data:
+  root: data
+  file: avhrr_daily_sst.npz
+  format_version: dincae_avhrr_v1
+  samples: 8
+  train_samples: 6
+  start_date: "2008-01-01"
+  grid: [32, 32]
+  paper_period: "1985-2009"
+  paper_time_steps: 5266
+  random_cloud_fraction: 0.20
+model:
+  in_channels: 10
+  out_channels: 2
+  filters: [8, 12, 18, 27]
+  bottleneck_features: 22
+  dropout: 0.3
+  kernel_size: 3
+  grid_size: 32
+training:
+  epochs: 1
+  batch_size: 2
+  learning_rate: 0.001
+  gamma: 10.0
+  delta: 0.001
+  paper_epochs: 1000
+  paper_batch_size: 50
+runtime:
+  device: cpu
+  num_workers: 0
+paths:
+  checkpoint: result/checkpoints/dincae.pt
+  training_metrics: result/training/metrics.json
+  inference: result/output/predictions.npz
+  evaluation_dir: result/evaluation
+evaluation:
+  dineof_rank: 13
+  calibration_bins: 8
+paper_model:
+  grid: [112, 112]
+  time_steps: 5266
+  filters: [16, 24, 36, 54]
+  bottleneck_features: 529
+  epochs: 1000
+  batch_size: 50

config.json

@@ -0,0 +1,45 @@
+{
+  "model_name": "DINCAE",
+  "model_type": "dincae",
+  "architectures": ["DINCAE"],
+  "framework": "PyTorch",
+  "domain": "ocean",
+  "task": "probabilistic-sst-gap-reconstruction",
+  "implementation": {
+    "entry_point": "model/dincae.py",
+    "scope": "core-method reduced-spatial-scale engineering reproduction with paper dimensions recorded separately",
+    "train_script": "scripts/train.py",
+    "inference_script": "scripts/inference.py",
+    "evaluation_script": "scripts/result.py",
+    "synthetic_data_script": "scripts/fake_data.py"
+  },
+  "architecture": {
+    "engineering_input_shape": ["B", 10, 32, 32],
+    "engineering_output_shape": ["B", 2, 32, 32],
+    "paper_input_shape": ["B", 10, 112, 112],
+    "paper_output_shape": ["B", 2, 112, 112],
+    "engineering_filters": [8, 12, 18, 27],
+    "paper_filters": [16, 24, 36, 54],
+    "paper_bottleneck": [2646, 529, 2646],
+    "decoder": "nearest-neighbor upsampling with encoder skips"
+  },
+  "data": {
+    "dataset": "AVHRR Pathfinder daily SST",
+    "format_version": "dincae_avhrr_v1",
+    "paper_period": "1985-2009",
+    "paper_time_steps": 5266,
+    "paper_grid": [112, 112],
+    "engineering_grid": [32, 32],
+    "input_channels": 10,
+    "output_channels": 2,
+    "synthetic": true
+  },
+  "configuration_sources": [
+    "conf/config.yaml",
+    "model/dincae.py",
+    "scripts/fake_data.py",
+    "scripts/train.py",
+    "scripts/inference.py",
+    "scripts/result.py"
+  ]
+}

configuration.json

@@ -0,0 +1,12 @@
+{
+  "framework": "PyTorch",
+  "task": "sea surface temperature gap reconstruction with uncertainty",
+  "model": "DINCAE",
+  "input_format": "BCHW: engineering [B,10,32,32], paper [B,10,112,112]",
+  "protocol": "probabilistic daily SST reconstruction from cloudy observations",
+  "default_config": "conf/config.yaml",
+  "training": "scripts/train.py",
+  "inference": "scripts/inference.py",
+  "evaluation": "scripts/result.py",
+  "visualization": "scripts/result.py"
+}

model/dincae.py

@@ -0,0 +1,122 @@
+"""Independent PyTorch implementation of DINCAE for daily SST gap filling."""
+
+from __future__ import annotations
+
+from typing import Dict, Tuple
+
+import numpy as np
+import torch
+from torch import Tensor, nn
+from torch.nn import functional as F
+
+
+class ConvBlock(nn.Sequential):
+    def __init__(self, in_channels: int, out_channels: int, kernel_size: int = 3):
+        padding = kernel_size // 2
+        super().__init__(
+            nn.Conv2d(in_channels, out_channels, kernel_size, padding=padding),
+            nn.LeakyReLU(0.1, inplace=True),
+            nn.Conv2d(out_channels, out_channels, kernel_size, padding=padding),
+            nn.LeakyReLU(0.1, inplace=True),
+        )
+
+
+class DINCAE(nn.Module):
+    """Four-level convolutional autoencoder with the paper-sized dense bottleneck."""
+
+    def __init__(self, in_channels: int = 10, out_channels: int = 2,
+                 filters=(16, 24, 36, 54),
+                 bottleneck_features: int = 529, dropout: float = 0.3,
+                 kernel_size: int = 3, grid_size: int = 112):
+        super().__init__()
+        if len(filters) != 4 or out_channels != 2:
+            raise ValueError("DINCAE requires four encoder levels and two output channels")
+        if grid_size < 16 or grid_size % 16:
+            raise ValueError("grid_size must be at least 16 and divisible by 16")
+        self.input_shape = (in_channels, grid_size, grid_size)
+        self.encoders = nn.ModuleList()
+        previous = in_channels
+        for width in filters:
+            self.encoders.append(ConvBlock(previous, width, kernel_size))
+            previous = width
+        self.pool = nn.AvgPool2d(2)
+        latent_size = grid_size // 16
+        flattened = filters[-1] * latent_size * latent_size
+        self.latent_size = latent_size
+        self.dense_encode = nn.Linear(flattened, bottleneck_features)
+        self.dropout = nn.Dropout(dropout)
+        self.dense_decode = nn.Linear(bottleneck_features, flattened)
+        decoder_inputs = (filters[3] + filters[3], filters[3] + filters[2],
+                          filters[2] + filters[1], filters[1] + filters[0])
+        decoder_outputs = (filters[3], filters[2], filters[1], filters[0])
+        self.decoders = nn.ModuleList([
+            ConvBlock(cin, cout, kernel_size)
+            for cin, cout in zip(decoder_inputs, decoder_outputs)
+        ])
+        self.output = nn.Conv2d(filters[0], out_channels, 1)
+
+    def forward(self, x: Tensor) -> Tensor:
+        if tuple(x.shape[-2:]) != self.input_shape[1:] or x.shape[1] != self.input_shape[0]:
+            raise ValueError(f"expected [B,{self.input_shape[0]},{self.input_shape[1]},{self.input_shape[2]}], got {tuple(x.shape)}")
+        skips = []
+        for encoder in self.encoders:
+            x = encoder(x)
+            skips.append(x)
+            x = self.pool(x)
+        x = x.flatten(1)
+        x = self.dense_decode(self.dropout(F.leaky_relu(self.dense_encode(x), 0.1)))
+        x = x.view(x.shape[0], -1, self.latent_size, self.latent_size)
+        for decoder, skip in zip(self.decoders, reversed(skips)):
+            x = F.interpolate(x, size=skip.shape[-2:], mode="nearest")
+            x = decoder(torch.cat((x, skip), dim=1))
+        return self.output(x)
+
+
+def output_distribution(output: Tensor, gamma: float = 10.0,
+                        delta: float = 1e-3) -> Tuple[Tensor, Tensor, Tensor]:
+    """Recover precision, variance, and mean from [log precision, temperature*precision]."""
+    log_precision = output[:, 0]
+    precision = torch.maximum(torch.exp(torch.minimum(log_precision, log_precision.new_tensor(gamma))),
+                              log_precision.new_tensor(delta))
+    variance = precision.reciprocal()
+    mean = output[:, 1] / precision
+    return mean, variance, precision
+
+
+def masked_gaussian_nll(output: Tensor, target: Tensor, mask: Tensor,
+                        gamma: float = 10.0, delta: float = 1e-3) -> Tensor:
+    mean, _, precision = output_distribution(output, gamma, delta)
+    target = target[:, 0]
+    valid = mask[:, 0].bool() & torch.isfinite(target)
+    if not torch.any(valid):
+        return output.sum() * 0.0
+    error = target[valid] - mean[valid]
+    return (0.5 * (precision[valid] * error.square() - torch.log(precision[valid]))).mean()
+
+
+def reconstruction_metrics(target: np.ndarray, prediction: np.ndarray,
+                           mask: np.ndarray) -> Dict[str, float]:
+    valid = mask.astype(bool) & np.isfinite(target) & np.isfinite(prediction)
+    if not np.any(valid):
+        return {"rmse": float("nan"), "crmse": float("nan"), "bias": float("nan")}
+    error = prediction[valid].astype(np.float64) - target[valid].astype(np.float64)
+    bias = float(error.mean())
+    return {
+        "rmse": float(np.sqrt(np.mean(error ** 2))),
+        "crmse": float(np.sqrt(np.mean((error - bias) ** 2))),
+        "bias": bias,
+    }
+
+
+def build_input(observed: np.ndarray, precision: np.ndarray, index: int,
+                longitude: np.ndarray, latitude: np.ndarray, day_of_year: int) -> np.ndarray:
+    """Build 10 channels: three anomaly*precision, three precision, and four auxiliaries."""
+    frames = []
+    for offset in (0, -1, 1):
+        j = min(max(index + offset, 0), observed.shape[0] - 1)
+        anomaly = np.nan_to_num(observed[j], nan=0.0)
+        frames.extend((anomaly * precision[j], precision[j]))
+    angle = 2.0 * np.pi * day_of_year / 365.25
+    frames.extend((longitude, latitude, np.full_like(longitude, np.cos(angle)),
+                   np.full_like(longitude, np.sin(angle))))
+    return np.stack(frames).astype(np.float32)

scripts/fake_data.py

@@ -0,0 +1,65 @@
+"""Generate compact AVHRR-like daily SST data for engineering validation."""
+
+from datetime import date, timedelta
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def main():
+    config = yaml.safe_load((ROOT / "conf/config.yaml").read_text())
+    data_config = config["data"]
+    n = int(data_config["samples"])
+    height, width = map(int, data_config["grid"])
+    if height != width or height < 16 or height % 16 or n < 4:
+        raise ValueError("DINCAE engineering grids must be square, divisible by 16, and contain at least four days")
+    rng = np.random.default_rng(int(config["seed"]))
+    yy, xx = np.mgrid[-1:1:complex(height), -1:1:complex(width)]
+    longitude = (12.0 + 6.0 * xx).astype(np.float32)
+    latitude = (38.0 + 6.0 * yy).astype(np.float32)
+    ocean = ((xx + 0.28) ** 2 + (yy - 0.10) ** 2 > 0.055) & (xx - 0.75 * yy < 1.12)
+    sst = np.empty((n, height, width), np.float32)
+    observed = np.empty_like(sst)
+    precision = np.empty_like(sst)
+    start = date.fromisoformat(data_config["start_date"])
+    timestamps = []
+    for t in range(n):
+        current = start + timedelta(days=t)
+        timestamps.append(current.isoformat())
+        phase = 2 * np.pi * t / max(n, 16)
+        eddy = 1.9 * np.exp(-((xx - 0.35 * np.sin(phase)) ** 2 +
+                              (yy - 0.25 * np.cos(phase)) ** 2) / 0.055)
+        field = 18.0 - 3.1 * yy + 0.7 * np.sin(3 * xx + phase) + eddy
+        field += rng.normal(0, 0.06, (height, width))
+        field[~ocean] = np.nan
+        cloud_noise = rng.normal(size=(height, width))
+        cloud = (np.sin(5 * xx + phase) + np.cos(4 * yy - phase) + cloud_noise > 1.05)
+        valid = ocean & ~cloud
+        sigma = (0.22 + 0.08 * (1 + np.sin(2 * xx - yy + phase))).astype(np.float32)
+        obs = field + rng.normal(0, sigma)
+        obs[~valid] = np.nan
+        sst[t], observed[t] = field, obs
+        precision[t] = np.where(valid, 1.0 / sigma ** 2, 0.0)
+    climatology = np.zeros((height, width), np.float32)
+    climatology[ocean] = np.mean(sst[:, ocean], axis=0)
+    anomaly = sst - climatology
+    observed_anomaly = observed - climatology
+    output = ROOT / data_config["root"]
+    output.mkdir(parents=True, exist_ok=True)
+    np.savez_compressed(output / data_config["file"],
+                        format_version=np.array(data_config["format_version"]),
+                        sst_anomaly=anomaly, observed_anomaly=observed_anomaly,
+                        precision=precision, climatology=climatology,
+                        ocean_mask=ocean, longitude=longitude, latitude=latitude,
+                        timestamps=np.asarray(timestamps), units=np.array("degree_Celsius"),
+                        grid=np.array([height, width], np.int32),
+                        source=np.array("synthetic AVHRR-like engineering data"))
+    print(f"generated={n} grid={height}x{width} observed_fraction={np.isfinite(observed).mean():.4f}")
+
+
+if __name__ == "__main__":
+    main()

scripts/inference.py

@@ -0,0 +1,51 @@
+"""Run probabilistic DINCAE reconstruction on held-out daily SST fields."""
+
+from datetime import date
+from pathlib import Path
+import sys
+
+import numpy as np
+import torch
+import yaml
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from model.dincae import DINCAE, build_input, output_distribution
+
+
+def main():
+    config = yaml.safe_load((ROOT / "conf/config.yaml").read_text())
+    device = torch.device("cuda" if torch.cuda.is_available() and config["runtime"]["device"] != "cpu" else "cpu")
+    data_path = ROOT / config["data"]["root"] / config["data"]["file"]
+    with np.load(data_path) as loaded:
+        data = {key: loaded[key] for key in loaded.files}
+    checkpoint = torch.load(ROOT / config["paths"]["checkpoint"], map_location=device, weights_only=False)
+    model = DINCAE(**checkpoint["model_config"]).to(device)
+    model.load_state_dict(checkpoint["model"])
+    model.eval()
+    start = min(int(config["data"]["train_samples"]), len(data["timestamps"]) - 1)
+    means, variances, targets, missing_masks, inputs_observed = [], [], [], [], []
+    with torch.no_grad():
+        for index in range(start, len(data["timestamps"])):
+            timestamp = date.fromisoformat(str(data["timestamps"][index]))
+            model_input = build_input(data["observed_anomaly"], data["precision"], index,
+                                      data["longitude"], data["latitude"], timestamp.timetuple().tm_yday)
+            output = model(torch.from_numpy(model_input[None]).to(device))
+            mean, variance, _ = output_distribution(output, checkpoint["gamma"], checkpoint["delta"])
+            means.append(mean[0].cpu().numpy() + data["climatology"])
+            variances.append(variance[0].cpu().numpy())
+            targets.append(data["sst_anomaly"][index] + data["climatology"])
+            missing_masks.append((data["precision"][index] == 0) & data["ocean_mask"])
+            inputs_observed.append(data["observed_anomaly"][index] + data["climatology"])
+    output_path = ROOT / config["paths"]["inference"]
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    np.savez_compressed(output_path, prediction=np.asarray(means), variance=np.asarray(variances),
+                        target=np.asarray(targets), missing_mask=np.asarray(missing_masks),
+                        observed=np.asarray(inputs_observed), ocean_mask=data["ocean_mask"],
+                        timestamps=data["timestamps"][start:], units=data["units"])
+    print(f"predictions={len(means)} shape={np.asarray(means).shape} output={output_path}")
+
+
+if __name__ == "__main__":
+    main()

scripts/result.py

@@ -0,0 +1,89 @@
+"""Evaluate DINCAE, residual calibration, and a rank-13 DINEOF-like baseline."""
+
+import json
+from pathlib import Path
+import sys
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import yaml
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from model.dincae import reconstruction_metrics
+
+
+def dineof_like(observed, ocean_mask, requested_rank):
+    matrix = observed.reshape(observed.shape[0], -1).astype(np.float64)
+    sea = ocean_mask.ravel()
+    values = matrix[:, sea]
+    missing = ~np.isfinite(values)
+    column_mean = np.nanmean(values, axis=0)
+    column_mean = np.nan_to_num(column_mean, nan=float(np.nanmean(values)))
+    filled = np.where(missing, column_mean[None], values)
+    rank = min(int(requested_rank), min(filled.shape) - 1)
+    for _ in range(4):
+        mean = filled.mean(axis=0, keepdims=True)
+        u, singular, vt = np.linalg.svd(filled - mean, full_matrices=False)
+        reconstructed = (u[:, :rank] * singular[:rank]) @ vt[:rank] + mean
+        filled[missing] = reconstructed[missing]
+    result = np.full_like(matrix, np.nan)
+    result[:, sea] = filled
+    return result.reshape(observed.shape).astype(np.float32), rank
+
+
+def main():
+    config = yaml.safe_load((ROOT / "conf/config.yaml").read_text())
+    with np.load(ROOT / config["paths"]["inference"]) as loaded:
+        pred = {key: loaded[key] for key in loaded.files}
+    with np.load(ROOT / config["data"]["root"] / config["data"]["file"]) as loaded:
+        full = {key: loaded[key] for key in loaded.files}
+    mask = pred["missing_mask"]
+    metrics = {"dincae": reconstruction_metrics(pred["target"], pred["prediction"], mask)}
+    std_residual = (pred["prediction"] - pred["target"]) / np.sqrt(np.maximum(pred["variance"], 1e-12))
+    residual = std_residual[mask & np.isfinite(std_residual)]
+    bins = int(config["evaluation"]["calibration_bins"])
+    edges = np.linspace(-4, 4, bins + 1)
+    counts, _ = np.histogram(residual, bins=edges)
+    metrics["standardized_residual_calibration"] = {
+        "mean": float(residual.mean()), "std": float(residual.std()),
+        "within_1sigma": float(np.mean(np.abs(residual) <= 1)),
+        "within_2sigma": float(np.mean(np.abs(residual) <= 2)),
+        "histogram_edges": edges.tolist(), "histogram_counts": counts.tolist(),
+    }
+    baseline_all, effective_rank = dineof_like(full["observed_anomaly"] + full["climatology"],
+                                                full["ocean_mask"], config["evaluation"]["dineof_rank"])
+    start = min(int(config["data"]["train_samples"]), len(full["timestamps"]) - 1)
+    baseline = baseline_all[start:]
+    metrics["dineof_like"] = reconstruction_metrics(pred["target"], baseline, mask)
+    metrics["dineof_like"]["requested_rank"] = int(config["evaluation"]["dineof_rank"])
+    metrics["dineof_like"]["effective_rank"] = effective_rank
+    out_dir = ROOT / config["paths"]["evaluation_dir"]
+    out_dir.mkdir(parents=True, exist_ok=True)
+    (out_dir / "metrics.json").write_text(json.dumps(metrics, indent=2))
+    index = 0
+    fields = (pred["observed"][index], pred["target"][index], pred["prediction"][index],
+              baseline[index], pred["prediction"][index] - pred["target"][index])
+    titles = ("Cloudy AVHRR-like input", "Complete target", "DINCAE", "DINEOF-like rank 13", "DINCAE error")
+    fig, axes = plt.subplots(1, 5, figsize=(17, 3.6), constrained_layout=True)
+    lower, upper = np.nanpercentile(pred["target"][index], (2, 98))
+    for axis, field, title in zip(axes, fields, titles):
+        if title.endswith("error"):
+            limit = max(float(np.nanpercentile(np.abs(field), 98)), 1e-3)
+            image = axis.imshow(field, cmap="RdBu_r", vmin=-limit, vmax=limit)
+        else:
+            image = axis.imshow(field, cmap="turbo", vmin=lower, vmax=upper)
+        axis.set_title(title, fontsize=9)
+        axis.set_axis_off()
+        fig.colorbar(image, ax=axis, fraction=0.046)
+    fig.savefig(out_dir / "comparison.png", dpi=150)
+    plt.close(fig)
+    print(json.dumps(metrics, indent=2))
+    print(f"comparison={out_dir / 'comparison.png'}")
+
+
+if __name__ == "__main__":
+    main()

scripts/train.py

@@ -0,0 +1,108 @@
+"""Train DINCAE with random cloud masking and masked Gaussian NLL."""
+
+import json
+import os
+import sys
+from datetime import date
+from pathlib import Path
+
+import numpy as np
+import torch
+import yaml
+from torch.nn.parallel import DistributedDataParallel
+from torch.utils.data import DataLoader, Dataset, DistributedSampler
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT))
+
+from model.dincae import DINCAE, build_input, masked_gaussian_nll
+
+
+class SSTDataset(Dataset):
+    def __init__(self, arrays, indices, cloud_fraction, seed):
+        self.data, self.indices = arrays, list(indices)
+        self.cloud_fraction, self.seed = float(cloud_fraction), int(seed)
+
+    def __len__(self):
+        return len(self.indices)
+
+    def __getitem__(self, item):
+        index = self.indices[item]
+        timestamp = date.fromisoformat(str(self.data["timestamps"][index]))
+        observed = self.data["observed_anomaly"].copy()
+        precision = self.data["precision"].copy()
+        rng = np.random.default_rng(self.seed + index)
+        original_valid = np.isfinite(observed[index]) & self.data["ocean_mask"]
+        withheld = original_valid & (rng.random(original_valid.shape) < self.cloud_fraction)
+        observed[index, withheld] = np.nan
+        precision[index, withheld] = 0.0
+        inputs = build_input(observed, precision, index, self.data["longitude"],
+                             self.data["latitude"], timestamp.timetuple().tm_yday)
+        target = self.data["sst_anomaly"][index][None].astype(np.float32)
+        return torch.from_numpy(inputs), torch.from_numpy(target), torch.from_numpy(withheld[None])
+
+
+def main():
+    config = yaml.safe_load((ROOT / "conf/config.yaml").read_text())
+    rank = int(os.environ.get("RANK", "0"))
+    world_size = int(os.environ.get("WORLD_SIZE", "1"))
+    local_rank = int(os.environ.get("LOCAL_RANK", "0"))
+    distributed = world_size > 1
+    if distributed:
+        backend = "nccl" if torch.cuda.is_available() else "gloo"
+        torch.distributed.init_process_group(backend=backend)
+    requested = config["runtime"]["device"]
+    use_cuda = torch.cuda.is_available() and requested != "cpu"
+    device = torch.device(f"cuda:{local_rank}" if use_cuda else "cpu")
+    if use_cuda:
+        torch.cuda.set_device(device)
+    torch.manual_seed(int(config["seed"]) + rank)
+    path = ROOT / config["data"]["root"] / config["data"]["file"]
+    with np.load(path) as loaded:
+        arrays = {key: loaded[key] for key in loaded.files}
+    train_count = min(int(config["data"]["train_samples"]), len(arrays["timestamps"]) - 1)
+    dataset = SSTDataset(arrays, range(train_count), config["data"]["random_cloud_fraction"], config["seed"])
+    sampler = DistributedSampler(dataset, shuffle=True, seed=int(config["seed"])) if distributed else None
+    loader = DataLoader(dataset, batch_size=int(config["training"]["batch_size"]),
+                        shuffle=sampler is None, sampler=sampler,
+                        num_workers=int(config["runtime"]["num_workers"]))
+    model = DINCAE(**config["model"]).to(device)
+    if distributed:
+        model = DistributedDataParallel(model, device_ids=[local_rank] if use_cuda else None)
+    optimizer = torch.optim.Adam(model.parameters(), lr=float(config["training"]["learning_rate"]))
+    history = []
+    for epoch in range(int(config["training"]["epochs"])):
+        if sampler is not None:
+            sampler.set_epoch(epoch)
+        model.train()
+        total, batches = 0.0, 0
+        for inputs, target, mask in loader:
+            inputs, target, mask = inputs.to(device), target.to(device), mask.to(device)
+            optimizer.zero_grad(set_to_none=True)
+            output = model(inputs)
+            loss = masked_gaussian_nll(output, target, mask,
+                                       config["training"]["gamma"], config["training"]["delta"])
+            loss.backward()
+            optimizer.step()
+            total, batches = total + float(loss.detach()), batches + 1
+        history.append({"epoch": epoch + 1, "masked_gaussian_nll": total / max(batches, 1)})
+        if rank == 0:
+            print(f"epoch={epoch + 1} loss={history[-1]['masked_gaussian_nll']:.6f}")
+    if rank == 0:
+        raw_model = model.module if distributed else model
+        checkpoint = ROOT / config["paths"]["checkpoint"]
+        checkpoint.parent.mkdir(parents=True, exist_ok=True)
+        torch.save({"model": raw_model.state_dict(), "model_config": config["model"],
+                    "gamma": config["training"]["gamma"], "delta": config["training"]["delta"],
+                    "format_version": "dincae_checkpoint_v1"}, checkpoint)
+        metrics = ROOT / config["paths"]["training_metrics"]
+        metrics.parent.mkdir(parents=True, exist_ok=True)
+        metrics.write_text(json.dumps({"history": history, "world_size": world_size,
+                                       "parameter_count": sum(p.numel() for p in raw_model.parameters())}, indent=2))
+        print(f"checkpoint={checkpoint}")
+    if distributed:
+        torch.distributed.destroy_process_group()
+
+
+if __name__ == "__main__":
+    main()