Spaces:
Sleeping
Sleeping
Add comprehensive production implementation guide
Browse files- Complete DWD ICON deployment guide for production systems
- Background downloader service with 6-hour update schedule
- Fast API server for instant global forecast queries
- Monitoring, alerting, and reliability best practices
- Performance optimization strategies and caching
- Legal requirements and proper attribution
- Deployment checklist and maintenance procedures
- Production-ready architecture for scaling weather services
π€ Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- PRODUCTION_GUIDE.md +842 -0
PRODUCTION_GUIDE.md
ADDED
|
@@ -0,0 +1,842 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# DWD ICON Weather Data - Production Implementation Guide
|
| 2 |
+
|
| 3 |
+
## Overview
|
| 4 |
+
This guide covers implementing a production weather forecasting system using real-time DWD ICON global model data from the German Weather Service.
|
| 5 |
+
|
| 6 |
+
## Table of Contents
|
| 7 |
+
- [Data Source Information](#data-source-information)
|
| 8 |
+
- [Update Schedule](#update-schedule)
|
| 9 |
+
- [Architecture Overview](#architecture-overview)
|
| 10 |
+
- [Production Implementation](#production-implementation)
|
| 11 |
+
- [API Endpoints](#api-endpoints)
|
| 12 |
+
- [Monitoring & Reliability](#monitoring--reliability)
|
| 13 |
+
- [Performance Optimization](#performance-optimization)
|
| 14 |
+
- [Legal & Attribution](#legal--attribution)
|
| 15 |
+
|
| 16 |
+
## Data Source Information
|
| 17 |
+
|
| 18 |
+
### Source Details
|
| 19 |
+
- **Provider**: German Weather Service (Deutscher Wetterdienst - DWD)
|
| 20 |
+
- **Model**: ICON Global Weather Model
|
| 21 |
+
- **Data Server**: https://opendata.dwd.de/weather/nwp/icon/grib/
|
| 22 |
+
- **License**: Open Government Data (commercial use permitted)
|
| 23 |
+
- **Format**: GRIB2 compressed with bzip2
|
| 24 |
+
- **Grid**: Icosahedral unstructured grid (global coverage)
|
| 25 |
+
- **Resolution**: ~13km globally
|
| 26 |
+
|
| 27 |
+
### Available Parameters
|
| 28 |
+
**Essential Parameters (recommended for production):**
|
| 29 |
+
- `t_2m`: Temperature at 2m (Kelvin β Celsius)
|
| 30 |
+
- `u_10m`: U-component wind at 10m (m/s)
|
| 31 |
+
- `v_10m`: V-component wind at 10m (m/s)
|
| 32 |
+
- `tot_prec`: Total precipitation (kg/mΒ²/s β mm/h)
|
| 33 |
+
- `snow_gsp`: Grid-scale snow (kg/mΒ²/s β mm/h)
|
| 34 |
+
- `clct`: Total cloud cover (fraction β percentage)
|
| 35 |
+
- `cape_con`: Convective Available Potential Energy (J/kg)
|
| 36 |
+
- `vmax_10m`: Wind gusts at 10m (m/s)
|
| 37 |
+
|
| 38 |
+
**Additional Parameters Available:**
|
| 39 |
+
- `relhum_2m`: Relative humidity at 2m
|
| 40 |
+
- `pmsl`: Pressure at mean sea level
|
| 41 |
+
- `rain_con`: Convective rain
|
| 42 |
+
- `rain_gsp`: Grid-scale rain
|
| 43 |
+
- `snow_con`: Convective snow
|
| 44 |
+
- `asob_s`: Net shortwave radiation
|
| 45 |
+
- Pressure level data (850, 700, 500, 300 hPa)
|
| 46 |
+
|
| 47 |
+
## Update Schedule
|
| 48 |
+
|
| 49 |
+
### Model Run Times (UTC)
|
| 50 |
+
- **00:00 UTC** - Available ~03:30 UTC
|
| 51 |
+
- **06:00 UTC** - Available ~09:30 UTC
|
| 52 |
+
- **12:00 UTC** - Available ~15:30 UTC
|
| 53 |
+
- **18:00 UTC** - Available ~21:30 UTC
|
| 54 |
+
|
| 55 |
+
### Data Availability Delay
|
| 56 |
+
- **Typical delay**: 3-4 hours after model run time
|
| 57 |
+
- **Coordinate files**: Only available from 00Z run (time-invariant)
|
| 58 |
+
- **Forecast range**: 0-180 hours (7.5 days)
|
| 59 |
+
|
| 60 |
+
### Recommended Update Strategy
|
| 61 |
+
```cron
|
| 62 |
+
# Download every 6 hours at 30 minutes past availability
|
| 63 |
+
30 4,10,16,22 * * * /path/to/download_dwd_data.py
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
## Architecture Overview
|
| 67 |
+
|
| 68 |
+
### Optimal Production Architecture
|
| 69 |
+
|
| 70 |
+
```
|
| 71 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 72 |
+
β PRODUCTION SYSTEM β
|
| 73 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
|
| 74 |
+
β βββββββββββββββββββ βββββββββββββββββββ ββββββββββββββββ β
|
| 75 |
+
β β Background β β Data Storage β β API Server β
|
| 76 |
+
β β Downloader βββββΆβ & Processing βββββΆβ (Instant β
|
| 77 |
+
β β (Every 6hrs) β β β β Response) β
|
| 78 |
+
β βββββββββββββββββββ βββββββββββββββββββ ββββββββββββββββ β
|
| 79 |
+
β β β β β
|
| 80 |
+
β βΌ βΌ βΌ β
|
| 81 |
+
β β’ Download GRIBs β’ Parse & Store β’ Extract β
|
| 82 |
+
β β’ Validate data β’ Index by location β’ Generate β
|
| 83 |
+
β β’ Handle failures β’ Cache coordinates β’ Serve JSON β
|
| 84 |
+
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
### File URL Structure
|
| 88 |
+
```
|
| 89 |
+
# Coordinate files (time-invariant, only from 00Z run)
|
| 90 |
+
https://opendata.dwd.de/weather/nwp/icon/grib/00/clat/icon_global_icosahedral_time-invariant_YYYYMMDD00_CLAT.grib2.bz2
|
| 91 |
+
https://opendata.dwd.de/weather/nwp/icon/grib/00/clon/icon_global_icosahedral_time-invariant_YYYYMMDD00_CLON.grib2.bz2
|
| 92 |
+
|
| 93 |
+
# Weather data files
|
| 94 |
+
https://opendata.dwd.de/weather/nwp/icon/grib/{RUN_HOUR}/{PARAMETER}/icon_global_icosahedral_single-level_{YYYYMMDD}{RUN_HOUR}_{FORECAST_HOUR:03d}_{PARAMETER}.grib2.bz2
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
### Example URLs
|
| 98 |
+
```
|
| 99 |
+
# Temperature at 2m, 12Z run, +006 forecast hour
|
| 100 |
+
https://opendata.dwd.de/weather/nwp/icon/grib/12/t_2m/icon_global_icosahedral_single-level_2025092412_006_T_2M.grib2.bz2
|
| 101 |
+
|
| 102 |
+
# Wind gusts, 00Z run, +024 forecast hour
|
| 103 |
+
https://opendata.dwd.de/weather/nwp/icon/grib/00/vmax_10m/icon_global_icosahedral_single-level_2025092400_024_VMAX_10M.grib2.bz2
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
## Production Implementation
|
| 107 |
+
|
| 108 |
+
### 1. Background Data Downloader
|
| 109 |
+
|
| 110 |
+
```python
|
| 111 |
+
#!/usr/bin/env python3
|
| 112 |
+
"""
|
| 113 |
+
DWD ICON Data Downloader - Production Service
|
| 114 |
+
Downloads global weather data every 6 hours
|
| 115 |
+
"""
|
| 116 |
+
|
| 117 |
+
import requests
|
| 118 |
+
import tempfile
|
| 119 |
+
import logging
|
| 120 |
+
from datetime import datetime, timedelta, timezone
|
| 121 |
+
from pathlib import Path
|
| 122 |
+
import os
|
| 123 |
+
import bz2
|
| 124 |
+
|
| 125 |
+
# Configuration
|
| 126 |
+
DATA_DIR = Path("/var/lib/weather-data")
|
| 127 |
+
LOG_FILE = "/var/log/dwd-downloader.log"
|
| 128 |
+
MAX_RETRIES = 3
|
| 129 |
+
TIMEOUT = 300 # 5 minutes per file
|
| 130 |
+
|
| 131 |
+
# Essential parameters for production
|
| 132 |
+
PARAMETERS = {
|
| 133 |
+
't_2m': 'T_2M',
|
| 134 |
+
'u_10m': 'U_10M',
|
| 135 |
+
'v_10m': 'V_10M',
|
| 136 |
+
'tot_prec': 'TOT_PREC',
|
| 137 |
+
'snow_gsp': 'SNOW_GSP',
|
| 138 |
+
'clct': 'CLCT',
|
| 139 |
+
'cape_con': 'CAPE_CON',
|
| 140 |
+
'vmax_10m': 'VMAX_10M'
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
# Optimized forecast hours: every 3hrs for 48hrs, then 24hr intervals
|
| 144 |
+
FORECAST_HOURS = [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 72, 96]
|
| 145 |
+
|
| 146 |
+
def get_latest_dwd_run():
|
| 147 |
+
"""Get the latest available DWD ICON model run"""
|
| 148 |
+
now = datetime.now(timezone.utc)
|
| 149 |
+
available_time = now - timedelta(hours=4) # 4-hour delay
|
| 150 |
+
|
| 151 |
+
run_hours = [0, 6, 12, 18]
|
| 152 |
+
current_hour = available_time.hour
|
| 153 |
+
latest_run = max([h for h in run_hours if h <= current_hour], default=18)
|
| 154 |
+
|
| 155 |
+
if latest_run > current_hour:
|
| 156 |
+
available_time = available_time - timedelta(days=1)
|
| 157 |
+
latest_run = 18
|
| 158 |
+
|
| 159 |
+
return available_time.replace(hour=latest_run, minute=0, second=0, microsecond=0)
|
| 160 |
+
|
| 161 |
+
def download_coordinate_files(run_date, data_dir):
|
| 162 |
+
"""Download coordinate files (only from 00Z run)"""
|
| 163 |
+
base_url = "https://opendata.dwd.de/weather/nwp/icon/grib"
|
| 164 |
+
date_str = run_date.strftime("%Y%m%d")
|
| 165 |
+
|
| 166 |
+
coord_dir = data_dir / "coordinates" / date_str
|
| 167 |
+
coord_dir.mkdir(parents=True, exist_ok=True)
|
| 168 |
+
|
| 169 |
+
files = {
|
| 170 |
+
'clat': f"icon_global_icosahedral_time-invariant_{date_str}00_CLAT.grib2.bz2",
|
| 171 |
+
'clon': f"icon_global_icosahedral_time-invariant_{date_str}00_CLON.grib2.bz2"
|
| 172 |
+
}
|
| 173 |
+
|
| 174 |
+
for coord_type, filename in files.items():
|
| 175 |
+
url = f"{base_url}/00/{coord_type}/{filename}"
|
| 176 |
+
output_path = coord_dir / filename
|
| 177 |
+
|
| 178 |
+
if output_path.exists():
|
| 179 |
+
logging.info(f"Coordinate file exists: {output_path}")
|
| 180 |
+
continue
|
| 181 |
+
|
| 182 |
+
logging.info(f"Downloading coordinate file: {url}")
|
| 183 |
+
download_file(url, output_path)
|
| 184 |
+
|
| 185 |
+
return coord_dir
|
| 186 |
+
|
| 187 |
+
def download_weather_data(run_date, data_dir):
|
| 188 |
+
"""Download weather parameter files"""
|
| 189 |
+
base_url = "https://opendata.dwd.de/weather/nwp/icon/grib"
|
| 190 |
+
date_str = run_date.strftime("%Y%m%d")
|
| 191 |
+
run_hour = f"{run_date.hour:02d}"
|
| 192 |
+
|
| 193 |
+
weather_dir = data_dir / "weather" / f"{date_str}_{run_hour}"
|
| 194 |
+
weather_dir.mkdir(parents=True, exist_ok=True)
|
| 195 |
+
|
| 196 |
+
total_files = len(PARAMETERS) * len(FORECAST_HOURS)
|
| 197 |
+
downloaded = 0
|
| 198 |
+
|
| 199 |
+
for param_key, param_dwd in PARAMETERS.items():
|
| 200 |
+
param_dir = weather_dir / param_key
|
| 201 |
+
param_dir.mkdir(exist_ok=True)
|
| 202 |
+
|
| 203 |
+
for forecast_hour in FORECAST_HOURS:
|
| 204 |
+
filename = f"icon_global_icosahedral_single-level_{date_str}{run_hour}_{forecast_hour:03d}_{param_dwd}.grib2.bz2"
|
| 205 |
+
url = f"{base_url}/{run_hour}/{param_key}/{filename}"
|
| 206 |
+
output_path = param_dir / filename
|
| 207 |
+
|
| 208 |
+
if output_path.exists():
|
| 209 |
+
logging.info(f"File exists: {output_path}")
|
| 210 |
+
downloaded += 1
|
| 211 |
+
continue
|
| 212 |
+
|
| 213 |
+
logging.info(f"Downloading [{downloaded+1}/{total_files}]: {param_key} +{forecast_hour:03d}h")
|
| 214 |
+
|
| 215 |
+
if download_file(url, output_path):
|
| 216 |
+
downloaded += 1
|
| 217 |
+
else:
|
| 218 |
+
logging.error(f"Failed to download: {url}")
|
| 219 |
+
|
| 220 |
+
logging.info(f"Downloaded {downloaded}/{total_files} files")
|
| 221 |
+
return weather_dir
|
| 222 |
+
|
| 223 |
+
def download_file(url, output_path):
|
| 224 |
+
"""Download a single file with retries"""
|
| 225 |
+
for attempt in range(MAX_RETRIES):
|
| 226 |
+
try:
|
| 227 |
+
response = requests.get(url, timeout=TIMEOUT, stream=True)
|
| 228 |
+
response.raise_for_status()
|
| 229 |
+
|
| 230 |
+
# Stream download to handle large files
|
| 231 |
+
with open(output_path, 'wb') as f:
|
| 232 |
+
for chunk in response.iter_content(chunk_size=8192):
|
| 233 |
+
f.write(chunk)
|
| 234 |
+
|
| 235 |
+
file_size = output_path.stat().st_size
|
| 236 |
+
logging.info(f"Downloaded: {output_path.name} ({file_size / 1024 / 1024:.1f} MB)")
|
| 237 |
+
return True
|
| 238 |
+
|
| 239 |
+
except Exception as e:
|
| 240 |
+
logging.warning(f"Download attempt {attempt + 1} failed: {e}")
|
| 241 |
+
if output_path.exists():
|
| 242 |
+
output_path.unlink()
|
| 243 |
+
|
| 244 |
+
if attempt == MAX_RETRIES - 1:
|
| 245 |
+
logging.error(f"Failed to download after {MAX_RETRIES} attempts: {url}")
|
| 246 |
+
return False
|
| 247 |
+
|
| 248 |
+
return False
|
| 249 |
+
|
| 250 |
+
def cleanup_old_data(data_dir, keep_days=3):
|
| 251 |
+
"""Remove data older than keep_days"""
|
| 252 |
+
cutoff_date = datetime.now() - timedelta(days=keep_days)
|
| 253 |
+
|
| 254 |
+
for data_type in ['coordinates', 'weather']:
|
| 255 |
+
type_dir = data_dir / data_type
|
| 256 |
+
if not type_dir.exists():
|
| 257 |
+
continue
|
| 258 |
+
|
| 259 |
+
for item in type_dir.iterdir():
|
| 260 |
+
if item.is_dir():
|
| 261 |
+
try:
|
| 262 |
+
# Parse date from directory name
|
| 263 |
+
if data_type == 'coordinates':
|
| 264 |
+
item_date = datetime.strptime(item.name, '%Y%m%d')
|
| 265 |
+
else: # weather
|
| 266 |
+
item_date = datetime.strptime(item.name[:8], '%Y%m%d')
|
| 267 |
+
|
| 268 |
+
if item_date < cutoff_date:
|
| 269 |
+
logging.info(f"Removing old data: {item}")
|
| 270 |
+
import shutil
|
| 271 |
+
shutil.rmtree(item)
|
| 272 |
+
|
| 273 |
+
except ValueError:
|
| 274 |
+
continue # Skip items that don't match date pattern
|
| 275 |
+
|
| 276 |
+
def main():
|
| 277 |
+
"""Main download process"""
|
| 278 |
+
logging.basicConfig(
|
| 279 |
+
level=logging.INFO,
|
| 280 |
+
format='%(asctime)s - %(levelname)s - %(message)s',
|
| 281 |
+
handlers=[
|
| 282 |
+
logging.FileHandler(LOG_FILE),
|
| 283 |
+
logging.StreamHandler()
|
| 284 |
+
]
|
| 285 |
+
)
|
| 286 |
+
|
| 287 |
+
try:
|
| 288 |
+
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
| 289 |
+
|
| 290 |
+
run_date = get_latest_dwd_run()
|
| 291 |
+
logging.info(f"Downloading DWD ICON data for run: {run_date.strftime('%Y-%m-%d %H:%M UTC')}")
|
| 292 |
+
|
| 293 |
+
# Download coordinate files
|
| 294 |
+
coord_dir = download_coordinate_files(run_date, DATA_DIR)
|
| 295 |
+
|
| 296 |
+
# Download weather data
|
| 297 |
+
weather_dir = download_weather_data(run_date, DATA_DIR)
|
| 298 |
+
|
| 299 |
+
# Cleanup old data
|
| 300 |
+
cleanup_old_data(DATA_DIR)
|
| 301 |
+
|
| 302 |
+
logging.info("Download process completed successfully")
|
| 303 |
+
|
| 304 |
+
except Exception as e:
|
| 305 |
+
logging.error(f"Download process failed: {e}")
|
| 306 |
+
raise
|
| 307 |
+
|
| 308 |
+
if __name__ == "__main__":
|
| 309 |
+
main()
|
| 310 |
+
```
|
| 311 |
+
|
| 312 |
+
### 2. Data Processing Service
|
| 313 |
+
|
| 314 |
+
```python
|
| 315 |
+
#!/usr/bin/env python3
|
| 316 |
+
"""
|
| 317 |
+
DWD ICON Data Processor - Production Service
|
| 318 |
+
Processes GRIB files into queryable format
|
| 319 |
+
"""
|
| 320 |
+
|
| 321 |
+
import xarray as xr
|
| 322 |
+
import numpy as np
|
| 323 |
+
from pathlib import Path
|
| 324 |
+
import sqlite3
|
| 325 |
+
import json
|
| 326 |
+
import logging
|
| 327 |
+
from scipy.spatial import cKDTree
|
| 328 |
+
import pickle
|
| 329 |
+
|
| 330 |
+
def process_coordinates(coord_dir):
|
| 331 |
+
"""Process coordinate files and build spatial index"""
|
| 332 |
+
clat_file = next(coord_dir.glob("*_CLAT.grib2.bz2"))
|
| 333 |
+
clon_file = next(coord_dir.glob("*_CLON.grib2.bz2"))
|
| 334 |
+
|
| 335 |
+
# Load coordinate data
|
| 336 |
+
clat_ds = xr.open_dataset(clat_file, engine='cfgrib')
|
| 337 |
+
clon_ds = xr.open_dataset(clon_file, engine='cfgrib')
|
| 338 |
+
|
| 339 |
+
# Extract coordinates (handle different variable names)
|
| 340 |
+
if 'clat' in clat_ds:
|
| 341 |
+
lats = clat_ds.clat.values
|
| 342 |
+
else:
|
| 343 |
+
lats = clat_ds[list(clat_ds.data_vars.keys())[0]].values
|
| 344 |
+
|
| 345 |
+
if 'clon' in clon_ds:
|
| 346 |
+
lons = clon_ds.clon.values
|
| 347 |
+
else:
|
| 348 |
+
lons = clon_ds[list(clon_ds.data_vars.keys())[0]].values
|
| 349 |
+
|
| 350 |
+
# Build spatial index for fast lookups
|
| 351 |
+
coords = np.column_stack([lats.ravel(), lons.ravel()])
|
| 352 |
+
tree = cKDTree(np.radians(coords))
|
| 353 |
+
|
| 354 |
+
return {
|
| 355 |
+
'lats': lats,
|
| 356 |
+
'lons': lons,
|
| 357 |
+
'tree': tree,
|
| 358 |
+
'coords': coords
|
| 359 |
+
}
|
| 360 |
+
|
| 361 |
+
def find_nearest_point(lat, lon, spatial_index):
|
| 362 |
+
"""Find nearest grid point using spatial index"""
|
| 363 |
+
target = np.radians([lat, lon])
|
| 364 |
+
distance, index = spatial_index['tree'].query(target)
|
| 365 |
+
|
| 366 |
+
grid_shape = spatial_index['lats'].shape
|
| 367 |
+
return np.unravel_index(index, grid_shape)
|
| 368 |
+
|
| 369 |
+
def extract_forecast_data(weather_dir, spatial_index, lat, lon):
|
| 370 |
+
"""Extract forecast data for specific location"""
|
| 371 |
+
nearest_idx = find_nearest_point(lat, lon, spatial_index)
|
| 372 |
+
|
| 373 |
+
forecast_data = {
|
| 374 |
+
'location': {'lat': lat, 'lon': lon},
|
| 375 |
+
'grid_point': {
|
| 376 |
+
'lat': float(spatial_index['lats'][nearest_idx]),
|
| 377 |
+
'lon': float(spatial_index['lons'][nearest_idx])
|
| 378 |
+
},
|
| 379 |
+
'forecast': []
|
| 380 |
+
}
|
| 381 |
+
|
| 382 |
+
# Process each parameter
|
| 383 |
+
for param_key in PARAMETERS.keys():
|
| 384 |
+
param_dir = weather_dir / param_key
|
| 385 |
+
if not param_dir.exists():
|
| 386 |
+
continue
|
| 387 |
+
|
| 388 |
+
param_data = []
|
| 389 |
+
|
| 390 |
+
for forecast_hour in FORECAST_HOURS:
|
| 391 |
+
grib_files = list(param_dir.glob(f"*_{forecast_hour:03d}_*.grib2.bz2"))
|
| 392 |
+
if not grib_files:
|
| 393 |
+
param_data.append(None)
|
| 394 |
+
continue
|
| 395 |
+
|
| 396 |
+
try:
|
| 397 |
+
ds = xr.open_dataset(grib_files[0], engine='cfgrib')
|
| 398 |
+
var_name = list(ds.data_vars.keys())[0]
|
| 399 |
+
value = ds[var_name].values[nearest_idx]
|
| 400 |
+
param_data.append(float(value))
|
| 401 |
+
|
| 402 |
+
except Exception as e:
|
| 403 |
+
logging.warning(f"Error processing {param_key} +{forecast_hour:03d}h: {e}")
|
| 404 |
+
param_data.append(None)
|
| 405 |
+
|
| 406 |
+
forecast_data[param_key] = param_data
|
| 407 |
+
|
| 408 |
+
return forecast_data
|
| 409 |
+
```
|
| 410 |
+
|
| 411 |
+
### 3. Fast API Server
|
| 412 |
+
|
| 413 |
+
```python
|
| 414 |
+
#!/usr/bin/env python3
|
| 415 |
+
"""
|
| 416 |
+
DWD Weather API - Production Server
|
| 417 |
+
Serves instant forecasts from processed data
|
| 418 |
+
"""
|
| 419 |
+
|
| 420 |
+
from fastapi import FastAPI, HTTPException
|
| 421 |
+
from fastapi.responses import JSONResponse
|
| 422 |
+
from pydantic import BaseModel
|
| 423 |
+
import uvicorn
|
| 424 |
+
from pathlib import Path
|
| 425 |
+
import pickle
|
| 426 |
+
import json
|
| 427 |
+
from datetime import datetime, timedelta
|
| 428 |
+
import logging
|
| 429 |
+
|
| 430 |
+
app = FastAPI(
|
| 431 |
+
title="DWD ICON Weather API",
|
| 432 |
+
description="Real-time weather forecasts from German Weather Service",
|
| 433 |
+
version="1.0.0"
|
| 434 |
+
)
|
| 435 |
+
|
| 436 |
+
# Global variables for cached data
|
| 437 |
+
spatial_index = None
|
| 438 |
+
latest_run_date = None
|
| 439 |
+
data_cache = {}
|
| 440 |
+
|
| 441 |
+
class ForecastRequest(BaseModel):
|
| 442 |
+
latitude: float
|
| 443 |
+
longitude: float
|
| 444 |
+
|
| 445 |
+
class ForecastResponse(BaseModel):
|
| 446 |
+
location: dict
|
| 447 |
+
grid_point: dict
|
| 448 |
+
forecast_run: str
|
| 449 |
+
forecast_data: dict
|
| 450 |
+
|
| 451 |
+
@app.on_event("startup")
|
| 452 |
+
async def startup_event():
|
| 453 |
+
"""Load latest data on startup"""
|
| 454 |
+
global spatial_index, latest_run_date
|
| 455 |
+
|
| 456 |
+
try:
|
| 457 |
+
# Load spatial index
|
| 458 |
+
index_file = Path("/var/lib/weather-data/spatial_index.pkl")
|
| 459 |
+
if index_file.exists():
|
| 460 |
+
with open(index_file, 'rb') as f:
|
| 461 |
+
spatial_index = pickle.load(f)
|
| 462 |
+
logging.info("Loaded spatial index")
|
| 463 |
+
|
| 464 |
+
# Determine latest run
|
| 465 |
+
weather_dir = Path("/var/lib/weather-data/weather")
|
| 466 |
+
if weather_dir.exists():
|
| 467 |
+
run_dirs = sorted([d for d in weather_dir.iterdir() if d.is_dir()])
|
| 468 |
+
if run_dirs:
|
| 469 |
+
latest_run_date = run_dirs[-1].name
|
| 470 |
+
logging.info(f"Latest data run: {latest_run_date}")
|
| 471 |
+
|
| 472 |
+
except Exception as e:
|
| 473 |
+
logging.error(f"Startup failed: {e}")
|
| 474 |
+
|
| 475 |
+
@app.get("/")
|
| 476 |
+
async def root():
|
| 477 |
+
return {"message": "DWD ICON Weather API", "status": "operational"}
|
| 478 |
+
|
| 479 |
+
@app.get("/health")
|
| 480 |
+
async def health_check():
|
| 481 |
+
"""Health check endpoint"""
|
| 482 |
+
if spatial_index is None:
|
| 483 |
+
raise HTTPException(status_code=503, detail="Spatial index not loaded")
|
| 484 |
+
|
| 485 |
+
if latest_run_date is None:
|
| 486 |
+
raise HTTPException(status_code=503, detail="No weather data available")
|
| 487 |
+
|
| 488 |
+
return {
|
| 489 |
+
"status": "healthy",
|
| 490 |
+
"latest_run": latest_run_date,
|
| 491 |
+
"data_points": len(spatial_index['coords']) if spatial_index else 0
|
| 492 |
+
}
|
| 493 |
+
|
| 494 |
+
@app.post("/forecast", response_model=ForecastResponse)
|
| 495 |
+
async def get_forecast(request: ForecastRequest):
|
| 496 |
+
"""Get weather forecast for specific location"""
|
| 497 |
+
if spatial_index is None:
|
| 498 |
+
raise HTTPException(status_code=503, detail="Service not ready")
|
| 499 |
+
|
| 500 |
+
try:
|
| 501 |
+
# Extract forecast data
|
| 502 |
+
weather_dir = Path(f"/var/lib/weather-data/weather/{latest_run_date}")
|
| 503 |
+
forecast_data = extract_forecast_data(
|
| 504 |
+
weather_dir,
|
| 505 |
+
spatial_index,
|
| 506 |
+
request.latitude,
|
| 507 |
+
request.longitude
|
| 508 |
+
)
|
| 509 |
+
|
| 510 |
+
return ForecastResponse(
|
| 511 |
+
location=forecast_data['location'],
|
| 512 |
+
grid_point=forecast_data['grid_point'],
|
| 513 |
+
forecast_run=latest_run_date,
|
| 514 |
+
forecast_data={k: v for k, v in forecast_data.items()
|
| 515 |
+
if k not in ['location', 'grid_point']}
|
| 516 |
+
)
|
| 517 |
+
|
| 518 |
+
except Exception as e:
|
| 519 |
+
logging.error(f"Forecast generation failed: {e}")
|
| 520 |
+
raise HTTPException(status_code=500, detail="Forecast generation failed")
|
| 521 |
+
|
| 522 |
+
@app.get("/locations/nearest")
|
| 523 |
+
async def get_nearest_grid_point(lat: float, lon: float):
|
| 524 |
+
"""Get nearest grid point information"""
|
| 525 |
+
if spatial_index is None:
|
| 526 |
+
raise HTTPException(status_code=503, detail="Service not ready")
|
| 527 |
+
|
| 528 |
+
try:
|
| 529 |
+
nearest_idx = find_nearest_point(lat, lon, spatial_index)
|
| 530 |
+
|
| 531 |
+
return {
|
| 532 |
+
"requested": {"lat": lat, "lon": lon},
|
| 533 |
+
"nearest_grid": {
|
| 534 |
+
"lat": float(spatial_index['lats'][nearest_idx]),
|
| 535 |
+
"lon": float(spatial_index['lons'][nearest_idx]),
|
| 536 |
+
"index": nearest_idx
|
| 537 |
+
}
|
| 538 |
+
}
|
| 539 |
+
|
| 540 |
+
except Exception as e:
|
| 541 |
+
raise HTTPException(status_code=500, detail=str(e))
|
| 542 |
+
|
| 543 |
+
if __name__ == "__main__":
|
| 544 |
+
uvicorn.run(app, host="0.0.0.0", port=8000)
|
| 545 |
+
```
|
| 546 |
+
|
| 547 |
+
## API Endpoints
|
| 548 |
+
|
| 549 |
+
### Base URL
|
| 550 |
+
```
|
| 551 |
+
https://your-domain.com/api/weather/
|
| 552 |
+
```
|
| 553 |
+
|
| 554 |
+
### Endpoints
|
| 555 |
+
|
| 556 |
+
#### GET /health
|
| 557 |
+
Health check and service status
|
| 558 |
+
```json
|
| 559 |
+
{
|
| 560 |
+
"status": "healthy",
|
| 561 |
+
"latest_run": "20250924_12",
|
| 562 |
+
"data_points": 2949120
|
| 563 |
+
}
|
| 564 |
+
```
|
| 565 |
+
|
| 566 |
+
#### POST /forecast
|
| 567 |
+
Get weather forecast for location
|
| 568 |
+
```json
|
| 569 |
+
// Request
|
| 570 |
+
{
|
| 571 |
+
"latitude": 52.5200,
|
| 572 |
+
"longitude": 13.4050
|
| 573 |
+
}
|
| 574 |
+
|
| 575 |
+
// Response
|
| 576 |
+
{
|
| 577 |
+
"location": {"lat": 52.52, "lon": 13.405},
|
| 578 |
+
"grid_point": {"lat": 52.520, "lon": 13.336},
|
| 579 |
+
"forecast_run": "20250924_12",
|
| 580 |
+
"forecast_data": {
|
| 581 |
+
"t_2m": [287.15, 286.8, 285.5, ...],
|
| 582 |
+
"u_10m": [2.1, 2.3, 1.8, ...],
|
| 583 |
+
"v_10m": [-1.2, -0.8, -1.5, ...],
|
| 584 |
+
"tot_prec": [0.0, 0.1, 0.3, ...],
|
| 585 |
+
"snow_gsp": [0.0, 0.0, 0.0, ...],
|
| 586 |
+
"clct": [0.65, 0.72, 0.58, ...],
|
| 587 |
+
"cape_con": [0, 150, 320, ...],
|
| 588 |
+
"vmax_10m": [3.2, 3.8, 4.1, ...]
|
| 589 |
+
}
|
| 590 |
+
}
|
| 591 |
+
```
|
| 592 |
+
|
| 593 |
+
#### GET /locations/nearest?lat=52.52&lon=13.405
|
| 594 |
+
Get nearest grid point information
|
| 595 |
+
```json
|
| 596 |
+
{
|
| 597 |
+
"requested": {"lat": 52.52, "lon": 13.405},
|
| 598 |
+
"nearest_grid": {
|
| 599 |
+
"lat": 52.520,
|
| 600 |
+
"lon": 13.336,
|
| 601 |
+
"index": [1247, 856]
|
| 602 |
+
}
|
| 603 |
+
}
|
| 604 |
+
```
|
| 605 |
+
|
| 606 |
+
## Monitoring & Reliability
|
| 607 |
+
|
| 608 |
+
### Key Metrics to Monitor
|
| 609 |
+
- **Download success rate**: >95%
|
| 610 |
+
- **API response time**: <100ms
|
| 611 |
+
- **Data freshness**: <6 hours old
|
| 612 |
+
- **Storage usage**: Monitor disk space
|
| 613 |
+
- **Memory usage**: Monitor spatial index memory
|
| 614 |
+
|
| 615 |
+
### Alerting Thresholds
|
| 616 |
+
```yaml
|
| 617 |
+
# Example monitoring config
|
| 618 |
+
alerts:
|
| 619 |
+
- name: "DWD Download Failed"
|
| 620 |
+
condition: "download_success_rate < 0.95"
|
| 621 |
+
severity: "critical"
|
| 622 |
+
|
| 623 |
+
- name: "API Slow Response"
|
| 624 |
+
condition: "api_response_time_p95 > 200ms"
|
| 625 |
+
severity: "warning"
|
| 626 |
+
|
| 627 |
+
- name: "Stale Data"
|
| 628 |
+
condition: "data_age > 8h"
|
| 629 |
+
severity: "critical"
|
| 630 |
+
|
| 631 |
+
- name: "Disk Space Low"
|
| 632 |
+
condition: "disk_usage > 80%"
|
| 633 |
+
severity: "warning"
|
| 634 |
+
```
|
| 635 |
+
|
| 636 |
+
### Log Files
|
| 637 |
+
- **Downloader**: `/var/log/dwd-downloader.log`
|
| 638 |
+
- **Processor**: `/var/log/dwd-processor.log`
|
| 639 |
+
- **API Server**: `/var/log/dwd-api.log`
|
| 640 |
+
|
| 641 |
+
### Systemd Services
|
| 642 |
+
```ini
|
| 643 |
+
# /etc/systemd/system/dwd-downloader.service
|
| 644 |
+
[Unit]
|
| 645 |
+
Description=DWD ICON Data Downloader
|
| 646 |
+
After=network.target
|
| 647 |
+
|
| 648 |
+
[Service]
|
| 649 |
+
Type=oneshot
|
| 650 |
+
ExecStart=/usr/local/bin/dwd-downloader
|
| 651 |
+
User=weather
|
| 652 |
+
Group=weather
|
| 653 |
+
|
| 654 |
+
# /etc/systemd/system/dwd-downloader.timer
|
| 655 |
+
[Unit]
|
| 656 |
+
Description=Run DWD downloader every 6 hours
|
| 657 |
+
Requires=dwd-downloader.service
|
| 658 |
+
|
| 659 |
+
[Timer]
|
| 660 |
+
OnCalendar=*-*-* 04,10,16,22:30:00
|
| 661 |
+
Persistent=true
|
| 662 |
+
|
| 663 |
+
[Install]
|
| 664 |
+
WantedBy=timers.target
|
| 665 |
+
|
| 666 |
+
# /etc/systemd/system/dwd-api.service
|
| 667 |
+
[Unit]
|
| 668 |
+
Description=DWD Weather API Server
|
| 669 |
+
After=network.target
|
| 670 |
+
|
| 671 |
+
[Service]
|
| 672 |
+
Type=simple
|
| 673 |
+
ExecStart=/usr/local/bin/dwd-api
|
| 674 |
+
Restart=always
|
| 675 |
+
User=weather
|
| 676 |
+
Group=weather
|
| 677 |
+
|
| 678 |
+
[Install]
|
| 679 |
+
WantedBy=multi-user.target
|
| 680 |
+
```
|
| 681 |
+
|
| 682 |
+
## Performance Optimization
|
| 683 |
+
|
| 684 |
+
### Storage Optimization
|
| 685 |
+
```bash
|
| 686 |
+
# Compressed storage (optional)
|
| 687 |
+
# Store processed data in compressed format
|
| 688 |
+
STORAGE_FORMAT="zarr" # or "parquet", "hdf5"
|
| 689 |
+
|
| 690 |
+
# Partition by date for faster queries
|
| 691 |
+
DATA_STRUCTURE="
|
| 692 |
+
/var/lib/weather-data/
|
| 693 |
+
βββ coordinates/
|
| 694 |
+
β βββ 20250924/
|
| 695 |
+
β βββ CLAT.grib2.bz2
|
| 696 |
+
β βββ CLON.grib2.bz2
|
| 697 |
+
βββ weather/
|
| 698 |
+
β βββ 20250924_12/
|
| 699 |
+
β βββ t_2m/
|
| 700 |
+
β βββ u_10m/
|
| 701 |
+
β βββ ...
|
| 702 |
+
βββ processed/
|
| 703 |
+
βββ 20250924_12/
|
| 704 |
+
βββ spatial_index.pkl
|
| 705 |
+
βββ weather_data.zarr
|
| 706 |
+
"
|
| 707 |
+
```
|
| 708 |
+
|
| 709 |
+
### Memory Optimization
|
| 710 |
+
```python
|
| 711 |
+
# Load only required regions for specific queries
|
| 712 |
+
def load_regional_data(bounds):
|
| 713 |
+
"""Load data only for specific geographic bounds"""
|
| 714 |
+
# Implementation for regional data loading
|
| 715 |
+
pass
|
| 716 |
+
|
| 717 |
+
# Use memory mapping for large datasets
|
| 718 |
+
def memory_map_data(file_path):
|
| 719 |
+
"""Memory map data files for efficient access"""
|
| 720 |
+
return np.memmap(file_path, mode='r')
|
| 721 |
+
```
|
| 722 |
+
|
| 723 |
+
### Caching Strategy
|
| 724 |
+
```python
|
| 725 |
+
# Redis/Memcached for frequently requested locations
|
| 726 |
+
CACHE_CONFIG = {
|
| 727 |
+
'redis_url': 'redis://localhost:6379',
|
| 728 |
+
'cache_ttl': 3600, # 1 hour
|
| 729 |
+
'max_cached_locations': 10000
|
| 730 |
+
}
|
| 731 |
+
|
| 732 |
+
# Pre-compute forecasts for major cities
|
| 733 |
+
PRECOMPUTE_LOCATIONS = [
|
| 734 |
+
(52.5200, 13.4050), # Berlin
|
| 735 |
+
(48.8566, 2.3522), # Paris
|
| 736 |
+
(51.5074, -0.1278), # London
|
| 737 |
+
# ... add more major cities
|
| 738 |
+
]
|
| 739 |
+
```
|
| 740 |
+
|
| 741 |
+
## Legal & Attribution
|
| 742 |
+
|
| 743 |
+
### License Requirements
|
| 744 |
+
- **Data Source**: DWD Open Government Data
|
| 745 |
+
- **Attribution**: "Weather data provided by German Weather Service (DWD)"
|
| 746 |
+
- **Commercial Use**: β
Permitted
|
| 747 |
+
- **Redistribution**: β
Allowed with attribution
|
| 748 |
+
|
| 749 |
+
### Required Attribution Text
|
| 750 |
+
```
|
| 751 |
+
Weather data provided by:
|
| 752 |
+
German Weather Service (Deutscher Wetterdienst - DWD)
|
| 753 |
+
ICON Global Weather Model
|
| 754 |
+
https://opendata.dwd.de/
|
| 755 |
+
|
| 756 |
+
This product uses data from the DWD ICON model.
|
| 757 |
+
DWD bears no responsibility for the correctness,
|
| 758 |
+
accuracy or completeness of the data provided.
|
| 759 |
+
```
|
| 760 |
+
|
| 761 |
+
### Terms of Use
|
| 762 |
+
- No warranty on data accuracy
|
| 763 |
+
- Users responsible for verification
|
| 764 |
+
- Commercial use permitted
|
| 765 |
+
- Must maintain attribution
|
| 766 |
+
- Cannot claim data as proprietary
|
| 767 |
+
|
| 768 |
+
## Deployment Checklist
|
| 769 |
+
|
| 770 |
+
### Pre-Production
|
| 771 |
+
- [ ] Set up monitoring and alerting
|
| 772 |
+
- [ ] Configure log rotation
|
| 773 |
+
- [ ] Set up automated backups
|
| 774 |
+
- [ ] Test failover scenarios
|
| 775 |
+
- [ ] Load test API endpoints
|
| 776 |
+
- [ ] Validate data quality
|
| 777 |
+
- [ ] Set up SSL certificates
|
| 778 |
+
|
| 779 |
+
### Production Deployment
|
| 780 |
+
- [ ] Deploy downloader service
|
| 781 |
+
- [ ] Deploy API server
|
| 782 |
+
- [ ] Configure reverse proxy (nginx)
|
| 783 |
+
- [ ] Set up monitoring dashboards
|
| 784 |
+
- [ ] Configure automated scaling
|
| 785 |
+
- [ ] Test end-to-end workflow
|
| 786 |
+
- [ ] Document operational procedures
|
| 787 |
+
|
| 788 |
+
### Post-Deployment
|
| 789 |
+
- [ ] Monitor for 48 hours
|
| 790 |
+
- [ ] Verify data accuracy
|
| 791 |
+
- [ ] Check performance metrics
|
| 792 |
+
- [ ] Test backup/restore
|
| 793 |
+
- [ ] Update documentation
|
| 794 |
+
- [ ] Train operations team
|
| 795 |
+
|
| 796 |
+
## Support & Maintenance
|
| 797 |
+
|
| 798 |
+
### Regular Maintenance Tasks
|
| 799 |
+
- **Daily**: Monitor system health, check logs
|
| 800 |
+
- **Weekly**: Verify data quality, check storage usage
|
| 801 |
+
- **Monthly**: Review performance metrics, update documentation
|
| 802 |
+
- **Quarterly**: Security updates, capacity planning
|
| 803 |
+
|
| 804 |
+
### Troubleshooting Common Issues
|
| 805 |
+
|
| 806 |
+
#### Download Failures
|
| 807 |
+
```bash
|
| 808 |
+
# Check DWD server status
|
| 809 |
+
curl -I https://opendata.dwd.de/weather/nwp/icon/grib/
|
| 810 |
+
|
| 811 |
+
# Verify network connectivity
|
| 812 |
+
nslookup opendata.dwd.de
|
| 813 |
+
|
| 814 |
+
# Check disk space
|
| 815 |
+
df -h /var/lib/weather-data/
|
| 816 |
+
|
| 817 |
+
# Review download logs
|
| 818 |
+
tail -f /var/log/dwd-downloader.log
|
| 819 |
+
```
|
| 820 |
+
|
| 821 |
+
#### API Performance Issues
|
| 822 |
+
```bash
|
| 823 |
+
# Check API server status
|
| 824 |
+
curl http://localhost:8000/health
|
| 825 |
+
|
| 826 |
+
# Monitor response times
|
| 827 |
+
curl -w "@curl-format.txt" http://localhost:8000/forecast
|
| 828 |
+
|
| 829 |
+
# Check memory usage
|
| 830 |
+
ps aux | grep dwd-api
|
| 831 |
+
```
|
| 832 |
+
|
| 833 |
+
## Contact & Support
|
| 834 |
+
- **Issues**: Create GitHub issue with system details
|
| 835 |
+
- **Documentation**: Keep this guide updated with changes
|
| 836 |
+
- **Monitoring**: Set up alerts for critical failures
|
| 837 |
+
|
| 838 |
+
---
|
| 839 |
+
|
| 840 |
+
**Version**: 1.0.0
|
| 841 |
+
**Last Updated**: 2025-09-24
|
| 842 |
+
**Maintainer**: Weather API Team
|