nakas Claude commited on
Commit
aaf72a1
Β·
1 Parent(s): e960572

Add comprehensive production implementation guide

Browse files

- Complete DWD ICON deployment guide for production systems
- Background downloader service with 6-hour update schedule
- Fast API server for instant global forecast queries
- Monitoring, alerting, and reliability best practices
- Performance optimization strategies and caching
- Legal requirements and proper attribution
- Deployment checklist and maintenance procedures
- Production-ready architecture for scaling weather services

πŸ€– Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

Files changed (1) hide show
  1. PRODUCTION_GUIDE.md +842 -0
PRODUCTION_GUIDE.md ADDED
@@ -0,0 +1,842 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # DWD ICON Weather Data - Production Implementation Guide
2
+
3
+ ## Overview
4
+ This guide covers implementing a production weather forecasting system using real-time DWD ICON global model data from the German Weather Service.
5
+
6
+ ## Table of Contents
7
+ - [Data Source Information](#data-source-information)
8
+ - [Update Schedule](#update-schedule)
9
+ - [Architecture Overview](#architecture-overview)
10
+ - [Production Implementation](#production-implementation)
11
+ - [API Endpoints](#api-endpoints)
12
+ - [Monitoring & Reliability](#monitoring--reliability)
13
+ - [Performance Optimization](#performance-optimization)
14
+ - [Legal & Attribution](#legal--attribution)
15
+
16
+ ## Data Source Information
17
+
18
+ ### Source Details
19
+ - **Provider**: German Weather Service (Deutscher Wetterdienst - DWD)
20
+ - **Model**: ICON Global Weather Model
21
+ - **Data Server**: https://opendata.dwd.de/weather/nwp/icon/grib/
22
+ - **License**: Open Government Data (commercial use permitted)
23
+ - **Format**: GRIB2 compressed with bzip2
24
+ - **Grid**: Icosahedral unstructured grid (global coverage)
25
+ - **Resolution**: ~13km globally
26
+
27
+ ### Available Parameters
28
+ **Essential Parameters (recommended for production):**
29
+ - `t_2m`: Temperature at 2m (Kelvin β†’ Celsius)
30
+ - `u_10m`: U-component wind at 10m (m/s)
31
+ - `v_10m`: V-component wind at 10m (m/s)
32
+ - `tot_prec`: Total precipitation (kg/mΒ²/s β†’ mm/h)
33
+ - `snow_gsp`: Grid-scale snow (kg/mΒ²/s β†’ mm/h)
34
+ - `clct`: Total cloud cover (fraction β†’ percentage)
35
+ - `cape_con`: Convective Available Potential Energy (J/kg)
36
+ - `vmax_10m`: Wind gusts at 10m (m/s)
37
+
38
+ **Additional Parameters Available:**
39
+ - `relhum_2m`: Relative humidity at 2m
40
+ - `pmsl`: Pressure at mean sea level
41
+ - `rain_con`: Convective rain
42
+ - `rain_gsp`: Grid-scale rain
43
+ - `snow_con`: Convective snow
44
+ - `asob_s`: Net shortwave radiation
45
+ - Pressure level data (850, 700, 500, 300 hPa)
46
+
47
+ ## Update Schedule
48
+
49
+ ### Model Run Times (UTC)
50
+ - **00:00 UTC** - Available ~03:30 UTC
51
+ - **06:00 UTC** - Available ~09:30 UTC
52
+ - **12:00 UTC** - Available ~15:30 UTC
53
+ - **18:00 UTC** - Available ~21:30 UTC
54
+
55
+ ### Data Availability Delay
56
+ - **Typical delay**: 3-4 hours after model run time
57
+ - **Coordinate files**: Only available from 00Z run (time-invariant)
58
+ - **Forecast range**: 0-180 hours (7.5 days)
59
+
60
+ ### Recommended Update Strategy
61
+ ```cron
62
+ # Download every 6 hours at 30 minutes past availability
63
+ 30 4,10,16,22 * * * /path/to/download_dwd_data.py
64
+ ```
65
+
66
+ ## Architecture Overview
67
+
68
+ ### Optimal Production Architecture
69
+
70
+ ```
71
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
72
+ β”‚ PRODUCTION SYSTEM β”‚
73
+ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
74
+ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”‚
75
+ β”‚ β”‚ Background β”‚ β”‚ Data Storage β”‚ β”‚ API Server β”‚
76
+ β”‚ β”‚ Downloader │───▢│ & Processing │───▢│ (Instant β”‚
77
+ β”‚ β”‚ (Every 6hrs) β”‚ β”‚ β”‚ β”‚ Response) β”‚
78
+ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ └─────────────── β”‚
79
+ β”‚ β”‚ β”‚ β”‚ β”‚
80
+ β”‚ β–Ό β–Ό β–Ό β”‚
81
+ β”‚ β€’ Download GRIBs β€’ Parse & Store β€’ Extract β”‚
82
+ β”‚ β€’ Validate data β€’ Index by location β€’ Generate β”‚
83
+ β”‚ β€’ Handle failures β€’ Cache coordinates β€’ Serve JSON β”‚
84
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
85
+ ```
86
+
87
+ ### File URL Structure
88
+ ```
89
+ # Coordinate files (time-invariant, only from 00Z run)
90
+ https://opendata.dwd.de/weather/nwp/icon/grib/00/clat/icon_global_icosahedral_time-invariant_YYYYMMDD00_CLAT.grib2.bz2
91
+ https://opendata.dwd.de/weather/nwp/icon/grib/00/clon/icon_global_icosahedral_time-invariant_YYYYMMDD00_CLON.grib2.bz2
92
+
93
+ # Weather data files
94
+ https://opendata.dwd.de/weather/nwp/icon/grib/{RUN_HOUR}/{PARAMETER}/icon_global_icosahedral_single-level_{YYYYMMDD}{RUN_HOUR}_{FORECAST_HOUR:03d}_{PARAMETER}.grib2.bz2
95
+ ```
96
+
97
+ ### Example URLs
98
+ ```
99
+ # Temperature at 2m, 12Z run, +006 forecast hour
100
+ https://opendata.dwd.de/weather/nwp/icon/grib/12/t_2m/icon_global_icosahedral_single-level_2025092412_006_T_2M.grib2.bz2
101
+
102
+ # Wind gusts, 00Z run, +024 forecast hour
103
+ https://opendata.dwd.de/weather/nwp/icon/grib/00/vmax_10m/icon_global_icosahedral_single-level_2025092400_024_VMAX_10M.grib2.bz2
104
+ ```
105
+
106
+ ## Production Implementation
107
+
108
+ ### 1. Background Data Downloader
109
+
110
+ ```python
111
+ #!/usr/bin/env python3
112
+ """
113
+ DWD ICON Data Downloader - Production Service
114
+ Downloads global weather data every 6 hours
115
+ """
116
+
117
+ import requests
118
+ import tempfile
119
+ import logging
120
+ from datetime import datetime, timedelta, timezone
121
+ from pathlib import Path
122
+ import os
123
+ import bz2
124
+
125
+ # Configuration
126
+ DATA_DIR = Path("/var/lib/weather-data")
127
+ LOG_FILE = "/var/log/dwd-downloader.log"
128
+ MAX_RETRIES = 3
129
+ TIMEOUT = 300 # 5 minutes per file
130
+
131
+ # Essential parameters for production
132
+ PARAMETERS = {
133
+ 't_2m': 'T_2M',
134
+ 'u_10m': 'U_10M',
135
+ 'v_10m': 'V_10M',
136
+ 'tot_prec': 'TOT_PREC',
137
+ 'snow_gsp': 'SNOW_GSP',
138
+ 'clct': 'CLCT',
139
+ 'cape_con': 'CAPE_CON',
140
+ 'vmax_10m': 'VMAX_10M'
141
+ }
142
+
143
+ # Optimized forecast hours: every 3hrs for 48hrs, then 24hr intervals
144
+ FORECAST_HOURS = [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 72, 96]
145
+
146
+ def get_latest_dwd_run():
147
+ """Get the latest available DWD ICON model run"""
148
+ now = datetime.now(timezone.utc)
149
+ available_time = now - timedelta(hours=4) # 4-hour delay
150
+
151
+ run_hours = [0, 6, 12, 18]
152
+ current_hour = available_time.hour
153
+ latest_run = max([h for h in run_hours if h <= current_hour], default=18)
154
+
155
+ if latest_run > current_hour:
156
+ available_time = available_time - timedelta(days=1)
157
+ latest_run = 18
158
+
159
+ return available_time.replace(hour=latest_run, minute=0, second=0, microsecond=0)
160
+
161
+ def download_coordinate_files(run_date, data_dir):
162
+ """Download coordinate files (only from 00Z run)"""
163
+ base_url = "https://opendata.dwd.de/weather/nwp/icon/grib"
164
+ date_str = run_date.strftime("%Y%m%d")
165
+
166
+ coord_dir = data_dir / "coordinates" / date_str
167
+ coord_dir.mkdir(parents=True, exist_ok=True)
168
+
169
+ files = {
170
+ 'clat': f"icon_global_icosahedral_time-invariant_{date_str}00_CLAT.grib2.bz2",
171
+ 'clon': f"icon_global_icosahedral_time-invariant_{date_str}00_CLON.grib2.bz2"
172
+ }
173
+
174
+ for coord_type, filename in files.items():
175
+ url = f"{base_url}/00/{coord_type}/{filename}"
176
+ output_path = coord_dir / filename
177
+
178
+ if output_path.exists():
179
+ logging.info(f"Coordinate file exists: {output_path}")
180
+ continue
181
+
182
+ logging.info(f"Downloading coordinate file: {url}")
183
+ download_file(url, output_path)
184
+
185
+ return coord_dir
186
+
187
+ def download_weather_data(run_date, data_dir):
188
+ """Download weather parameter files"""
189
+ base_url = "https://opendata.dwd.de/weather/nwp/icon/grib"
190
+ date_str = run_date.strftime("%Y%m%d")
191
+ run_hour = f"{run_date.hour:02d}"
192
+
193
+ weather_dir = data_dir / "weather" / f"{date_str}_{run_hour}"
194
+ weather_dir.mkdir(parents=True, exist_ok=True)
195
+
196
+ total_files = len(PARAMETERS) * len(FORECAST_HOURS)
197
+ downloaded = 0
198
+
199
+ for param_key, param_dwd in PARAMETERS.items():
200
+ param_dir = weather_dir / param_key
201
+ param_dir.mkdir(exist_ok=True)
202
+
203
+ for forecast_hour in FORECAST_HOURS:
204
+ filename = f"icon_global_icosahedral_single-level_{date_str}{run_hour}_{forecast_hour:03d}_{param_dwd}.grib2.bz2"
205
+ url = f"{base_url}/{run_hour}/{param_key}/{filename}"
206
+ output_path = param_dir / filename
207
+
208
+ if output_path.exists():
209
+ logging.info(f"File exists: {output_path}")
210
+ downloaded += 1
211
+ continue
212
+
213
+ logging.info(f"Downloading [{downloaded+1}/{total_files}]: {param_key} +{forecast_hour:03d}h")
214
+
215
+ if download_file(url, output_path):
216
+ downloaded += 1
217
+ else:
218
+ logging.error(f"Failed to download: {url}")
219
+
220
+ logging.info(f"Downloaded {downloaded}/{total_files} files")
221
+ return weather_dir
222
+
223
+ def download_file(url, output_path):
224
+ """Download a single file with retries"""
225
+ for attempt in range(MAX_RETRIES):
226
+ try:
227
+ response = requests.get(url, timeout=TIMEOUT, stream=True)
228
+ response.raise_for_status()
229
+
230
+ # Stream download to handle large files
231
+ with open(output_path, 'wb') as f:
232
+ for chunk in response.iter_content(chunk_size=8192):
233
+ f.write(chunk)
234
+
235
+ file_size = output_path.stat().st_size
236
+ logging.info(f"Downloaded: {output_path.name} ({file_size / 1024 / 1024:.1f} MB)")
237
+ return True
238
+
239
+ except Exception as e:
240
+ logging.warning(f"Download attempt {attempt + 1} failed: {e}")
241
+ if output_path.exists():
242
+ output_path.unlink()
243
+
244
+ if attempt == MAX_RETRIES - 1:
245
+ logging.error(f"Failed to download after {MAX_RETRIES} attempts: {url}")
246
+ return False
247
+
248
+ return False
249
+
250
+ def cleanup_old_data(data_dir, keep_days=3):
251
+ """Remove data older than keep_days"""
252
+ cutoff_date = datetime.now() - timedelta(days=keep_days)
253
+
254
+ for data_type in ['coordinates', 'weather']:
255
+ type_dir = data_dir / data_type
256
+ if not type_dir.exists():
257
+ continue
258
+
259
+ for item in type_dir.iterdir():
260
+ if item.is_dir():
261
+ try:
262
+ # Parse date from directory name
263
+ if data_type == 'coordinates':
264
+ item_date = datetime.strptime(item.name, '%Y%m%d')
265
+ else: # weather
266
+ item_date = datetime.strptime(item.name[:8], '%Y%m%d')
267
+
268
+ if item_date < cutoff_date:
269
+ logging.info(f"Removing old data: {item}")
270
+ import shutil
271
+ shutil.rmtree(item)
272
+
273
+ except ValueError:
274
+ continue # Skip items that don't match date pattern
275
+
276
+ def main():
277
+ """Main download process"""
278
+ logging.basicConfig(
279
+ level=logging.INFO,
280
+ format='%(asctime)s - %(levelname)s - %(message)s',
281
+ handlers=[
282
+ logging.FileHandler(LOG_FILE),
283
+ logging.StreamHandler()
284
+ ]
285
+ )
286
+
287
+ try:
288
+ DATA_DIR.mkdir(parents=True, exist_ok=True)
289
+
290
+ run_date = get_latest_dwd_run()
291
+ logging.info(f"Downloading DWD ICON data for run: {run_date.strftime('%Y-%m-%d %H:%M UTC')}")
292
+
293
+ # Download coordinate files
294
+ coord_dir = download_coordinate_files(run_date, DATA_DIR)
295
+
296
+ # Download weather data
297
+ weather_dir = download_weather_data(run_date, DATA_DIR)
298
+
299
+ # Cleanup old data
300
+ cleanup_old_data(DATA_DIR)
301
+
302
+ logging.info("Download process completed successfully")
303
+
304
+ except Exception as e:
305
+ logging.error(f"Download process failed: {e}")
306
+ raise
307
+
308
+ if __name__ == "__main__":
309
+ main()
310
+ ```
311
+
312
+ ### 2. Data Processing Service
313
+
314
+ ```python
315
+ #!/usr/bin/env python3
316
+ """
317
+ DWD ICON Data Processor - Production Service
318
+ Processes GRIB files into queryable format
319
+ """
320
+
321
+ import xarray as xr
322
+ import numpy as np
323
+ from pathlib import Path
324
+ import sqlite3
325
+ import json
326
+ import logging
327
+ from scipy.spatial import cKDTree
328
+ import pickle
329
+
330
+ def process_coordinates(coord_dir):
331
+ """Process coordinate files and build spatial index"""
332
+ clat_file = next(coord_dir.glob("*_CLAT.grib2.bz2"))
333
+ clon_file = next(coord_dir.glob("*_CLON.grib2.bz2"))
334
+
335
+ # Load coordinate data
336
+ clat_ds = xr.open_dataset(clat_file, engine='cfgrib')
337
+ clon_ds = xr.open_dataset(clon_file, engine='cfgrib')
338
+
339
+ # Extract coordinates (handle different variable names)
340
+ if 'clat' in clat_ds:
341
+ lats = clat_ds.clat.values
342
+ else:
343
+ lats = clat_ds[list(clat_ds.data_vars.keys())[0]].values
344
+
345
+ if 'clon' in clon_ds:
346
+ lons = clon_ds.clon.values
347
+ else:
348
+ lons = clon_ds[list(clon_ds.data_vars.keys())[0]].values
349
+
350
+ # Build spatial index for fast lookups
351
+ coords = np.column_stack([lats.ravel(), lons.ravel()])
352
+ tree = cKDTree(np.radians(coords))
353
+
354
+ return {
355
+ 'lats': lats,
356
+ 'lons': lons,
357
+ 'tree': tree,
358
+ 'coords': coords
359
+ }
360
+
361
+ def find_nearest_point(lat, lon, spatial_index):
362
+ """Find nearest grid point using spatial index"""
363
+ target = np.radians([lat, lon])
364
+ distance, index = spatial_index['tree'].query(target)
365
+
366
+ grid_shape = spatial_index['lats'].shape
367
+ return np.unravel_index(index, grid_shape)
368
+
369
+ def extract_forecast_data(weather_dir, spatial_index, lat, lon):
370
+ """Extract forecast data for specific location"""
371
+ nearest_idx = find_nearest_point(lat, lon, spatial_index)
372
+
373
+ forecast_data = {
374
+ 'location': {'lat': lat, 'lon': lon},
375
+ 'grid_point': {
376
+ 'lat': float(spatial_index['lats'][nearest_idx]),
377
+ 'lon': float(spatial_index['lons'][nearest_idx])
378
+ },
379
+ 'forecast': []
380
+ }
381
+
382
+ # Process each parameter
383
+ for param_key in PARAMETERS.keys():
384
+ param_dir = weather_dir / param_key
385
+ if not param_dir.exists():
386
+ continue
387
+
388
+ param_data = []
389
+
390
+ for forecast_hour in FORECAST_HOURS:
391
+ grib_files = list(param_dir.glob(f"*_{forecast_hour:03d}_*.grib2.bz2"))
392
+ if not grib_files:
393
+ param_data.append(None)
394
+ continue
395
+
396
+ try:
397
+ ds = xr.open_dataset(grib_files[0], engine='cfgrib')
398
+ var_name = list(ds.data_vars.keys())[0]
399
+ value = ds[var_name].values[nearest_idx]
400
+ param_data.append(float(value))
401
+
402
+ except Exception as e:
403
+ logging.warning(f"Error processing {param_key} +{forecast_hour:03d}h: {e}")
404
+ param_data.append(None)
405
+
406
+ forecast_data[param_key] = param_data
407
+
408
+ return forecast_data
409
+ ```
410
+
411
+ ### 3. Fast API Server
412
+
413
+ ```python
414
+ #!/usr/bin/env python3
415
+ """
416
+ DWD Weather API - Production Server
417
+ Serves instant forecasts from processed data
418
+ """
419
+
420
+ from fastapi import FastAPI, HTTPException
421
+ from fastapi.responses import JSONResponse
422
+ from pydantic import BaseModel
423
+ import uvicorn
424
+ from pathlib import Path
425
+ import pickle
426
+ import json
427
+ from datetime import datetime, timedelta
428
+ import logging
429
+
430
+ app = FastAPI(
431
+ title="DWD ICON Weather API",
432
+ description="Real-time weather forecasts from German Weather Service",
433
+ version="1.0.0"
434
+ )
435
+
436
+ # Global variables for cached data
437
+ spatial_index = None
438
+ latest_run_date = None
439
+ data_cache = {}
440
+
441
+ class ForecastRequest(BaseModel):
442
+ latitude: float
443
+ longitude: float
444
+
445
+ class ForecastResponse(BaseModel):
446
+ location: dict
447
+ grid_point: dict
448
+ forecast_run: str
449
+ forecast_data: dict
450
+
451
+ @app.on_event("startup")
452
+ async def startup_event():
453
+ """Load latest data on startup"""
454
+ global spatial_index, latest_run_date
455
+
456
+ try:
457
+ # Load spatial index
458
+ index_file = Path("/var/lib/weather-data/spatial_index.pkl")
459
+ if index_file.exists():
460
+ with open(index_file, 'rb') as f:
461
+ spatial_index = pickle.load(f)
462
+ logging.info("Loaded spatial index")
463
+
464
+ # Determine latest run
465
+ weather_dir = Path("/var/lib/weather-data/weather")
466
+ if weather_dir.exists():
467
+ run_dirs = sorted([d for d in weather_dir.iterdir() if d.is_dir()])
468
+ if run_dirs:
469
+ latest_run_date = run_dirs[-1].name
470
+ logging.info(f"Latest data run: {latest_run_date}")
471
+
472
+ except Exception as e:
473
+ logging.error(f"Startup failed: {e}")
474
+
475
+ @app.get("/")
476
+ async def root():
477
+ return {"message": "DWD ICON Weather API", "status": "operational"}
478
+
479
+ @app.get("/health")
480
+ async def health_check():
481
+ """Health check endpoint"""
482
+ if spatial_index is None:
483
+ raise HTTPException(status_code=503, detail="Spatial index not loaded")
484
+
485
+ if latest_run_date is None:
486
+ raise HTTPException(status_code=503, detail="No weather data available")
487
+
488
+ return {
489
+ "status": "healthy",
490
+ "latest_run": latest_run_date,
491
+ "data_points": len(spatial_index['coords']) if spatial_index else 0
492
+ }
493
+
494
+ @app.post("/forecast", response_model=ForecastResponse)
495
+ async def get_forecast(request: ForecastRequest):
496
+ """Get weather forecast for specific location"""
497
+ if spatial_index is None:
498
+ raise HTTPException(status_code=503, detail="Service not ready")
499
+
500
+ try:
501
+ # Extract forecast data
502
+ weather_dir = Path(f"/var/lib/weather-data/weather/{latest_run_date}")
503
+ forecast_data = extract_forecast_data(
504
+ weather_dir,
505
+ spatial_index,
506
+ request.latitude,
507
+ request.longitude
508
+ )
509
+
510
+ return ForecastResponse(
511
+ location=forecast_data['location'],
512
+ grid_point=forecast_data['grid_point'],
513
+ forecast_run=latest_run_date,
514
+ forecast_data={k: v for k, v in forecast_data.items()
515
+ if k not in ['location', 'grid_point']}
516
+ )
517
+
518
+ except Exception as e:
519
+ logging.error(f"Forecast generation failed: {e}")
520
+ raise HTTPException(status_code=500, detail="Forecast generation failed")
521
+
522
+ @app.get("/locations/nearest")
523
+ async def get_nearest_grid_point(lat: float, lon: float):
524
+ """Get nearest grid point information"""
525
+ if spatial_index is None:
526
+ raise HTTPException(status_code=503, detail="Service not ready")
527
+
528
+ try:
529
+ nearest_idx = find_nearest_point(lat, lon, spatial_index)
530
+
531
+ return {
532
+ "requested": {"lat": lat, "lon": lon},
533
+ "nearest_grid": {
534
+ "lat": float(spatial_index['lats'][nearest_idx]),
535
+ "lon": float(spatial_index['lons'][nearest_idx]),
536
+ "index": nearest_idx
537
+ }
538
+ }
539
+
540
+ except Exception as e:
541
+ raise HTTPException(status_code=500, detail=str(e))
542
+
543
+ if __name__ == "__main__":
544
+ uvicorn.run(app, host="0.0.0.0", port=8000)
545
+ ```
546
+
547
+ ## API Endpoints
548
+
549
+ ### Base URL
550
+ ```
551
+ https://your-domain.com/api/weather/
552
+ ```
553
+
554
+ ### Endpoints
555
+
556
+ #### GET /health
557
+ Health check and service status
558
+ ```json
559
+ {
560
+ "status": "healthy",
561
+ "latest_run": "20250924_12",
562
+ "data_points": 2949120
563
+ }
564
+ ```
565
+
566
+ #### POST /forecast
567
+ Get weather forecast for location
568
+ ```json
569
+ // Request
570
+ {
571
+ "latitude": 52.5200,
572
+ "longitude": 13.4050
573
+ }
574
+
575
+ // Response
576
+ {
577
+ "location": {"lat": 52.52, "lon": 13.405},
578
+ "grid_point": {"lat": 52.520, "lon": 13.336},
579
+ "forecast_run": "20250924_12",
580
+ "forecast_data": {
581
+ "t_2m": [287.15, 286.8, 285.5, ...],
582
+ "u_10m": [2.1, 2.3, 1.8, ...],
583
+ "v_10m": [-1.2, -0.8, -1.5, ...],
584
+ "tot_prec": [0.0, 0.1, 0.3, ...],
585
+ "snow_gsp": [0.0, 0.0, 0.0, ...],
586
+ "clct": [0.65, 0.72, 0.58, ...],
587
+ "cape_con": [0, 150, 320, ...],
588
+ "vmax_10m": [3.2, 3.8, 4.1, ...]
589
+ }
590
+ }
591
+ ```
592
+
593
+ #### GET /locations/nearest?lat=52.52&lon=13.405
594
+ Get nearest grid point information
595
+ ```json
596
+ {
597
+ "requested": {"lat": 52.52, "lon": 13.405},
598
+ "nearest_grid": {
599
+ "lat": 52.520,
600
+ "lon": 13.336,
601
+ "index": [1247, 856]
602
+ }
603
+ }
604
+ ```
605
+
606
+ ## Monitoring & Reliability
607
+
608
+ ### Key Metrics to Monitor
609
+ - **Download success rate**: >95%
610
+ - **API response time**: <100ms
611
+ - **Data freshness**: <6 hours old
612
+ - **Storage usage**: Monitor disk space
613
+ - **Memory usage**: Monitor spatial index memory
614
+
615
+ ### Alerting Thresholds
616
+ ```yaml
617
+ # Example monitoring config
618
+ alerts:
619
+ - name: "DWD Download Failed"
620
+ condition: "download_success_rate < 0.95"
621
+ severity: "critical"
622
+
623
+ - name: "API Slow Response"
624
+ condition: "api_response_time_p95 > 200ms"
625
+ severity: "warning"
626
+
627
+ - name: "Stale Data"
628
+ condition: "data_age > 8h"
629
+ severity: "critical"
630
+
631
+ - name: "Disk Space Low"
632
+ condition: "disk_usage > 80%"
633
+ severity: "warning"
634
+ ```
635
+
636
+ ### Log Files
637
+ - **Downloader**: `/var/log/dwd-downloader.log`
638
+ - **Processor**: `/var/log/dwd-processor.log`
639
+ - **API Server**: `/var/log/dwd-api.log`
640
+
641
+ ### Systemd Services
642
+ ```ini
643
+ # /etc/systemd/system/dwd-downloader.service
644
+ [Unit]
645
+ Description=DWD ICON Data Downloader
646
+ After=network.target
647
+
648
+ [Service]
649
+ Type=oneshot
650
+ ExecStart=/usr/local/bin/dwd-downloader
651
+ User=weather
652
+ Group=weather
653
+
654
+ # /etc/systemd/system/dwd-downloader.timer
655
+ [Unit]
656
+ Description=Run DWD downloader every 6 hours
657
+ Requires=dwd-downloader.service
658
+
659
+ [Timer]
660
+ OnCalendar=*-*-* 04,10,16,22:30:00
661
+ Persistent=true
662
+
663
+ [Install]
664
+ WantedBy=timers.target
665
+
666
+ # /etc/systemd/system/dwd-api.service
667
+ [Unit]
668
+ Description=DWD Weather API Server
669
+ After=network.target
670
+
671
+ [Service]
672
+ Type=simple
673
+ ExecStart=/usr/local/bin/dwd-api
674
+ Restart=always
675
+ User=weather
676
+ Group=weather
677
+
678
+ [Install]
679
+ WantedBy=multi-user.target
680
+ ```
681
+
682
+ ## Performance Optimization
683
+
684
+ ### Storage Optimization
685
+ ```bash
686
+ # Compressed storage (optional)
687
+ # Store processed data in compressed format
688
+ STORAGE_FORMAT="zarr" # or "parquet", "hdf5"
689
+
690
+ # Partition by date for faster queries
691
+ DATA_STRUCTURE="
692
+ /var/lib/weather-data/
693
+ β”œβ”€β”€ coordinates/
694
+ β”‚ └── 20250924/
695
+ β”‚ β”œβ”€β”€ CLAT.grib2.bz2
696
+ β”‚ └── CLON.grib2.bz2
697
+ β”œβ”€β”€ weather/
698
+ β”‚ └── 20250924_12/
699
+ β”‚ β”œβ”€β”€ t_2m/
700
+ β”‚ β”œβ”€β”€ u_10m/
701
+ β”‚ └── ...
702
+ └── processed/
703
+ └── 20250924_12/
704
+ β”œβ”€β”€ spatial_index.pkl
705
+ └── weather_data.zarr
706
+ "
707
+ ```
708
+
709
+ ### Memory Optimization
710
+ ```python
711
+ # Load only required regions for specific queries
712
+ def load_regional_data(bounds):
713
+ """Load data only for specific geographic bounds"""
714
+ # Implementation for regional data loading
715
+ pass
716
+
717
+ # Use memory mapping for large datasets
718
+ def memory_map_data(file_path):
719
+ """Memory map data files for efficient access"""
720
+ return np.memmap(file_path, mode='r')
721
+ ```
722
+
723
+ ### Caching Strategy
724
+ ```python
725
+ # Redis/Memcached for frequently requested locations
726
+ CACHE_CONFIG = {
727
+ 'redis_url': 'redis://localhost:6379',
728
+ 'cache_ttl': 3600, # 1 hour
729
+ 'max_cached_locations': 10000
730
+ }
731
+
732
+ # Pre-compute forecasts for major cities
733
+ PRECOMPUTE_LOCATIONS = [
734
+ (52.5200, 13.4050), # Berlin
735
+ (48.8566, 2.3522), # Paris
736
+ (51.5074, -0.1278), # London
737
+ # ... add more major cities
738
+ ]
739
+ ```
740
+
741
+ ## Legal & Attribution
742
+
743
+ ### License Requirements
744
+ - **Data Source**: DWD Open Government Data
745
+ - **Attribution**: "Weather data provided by German Weather Service (DWD)"
746
+ - **Commercial Use**: βœ… Permitted
747
+ - **Redistribution**: βœ… Allowed with attribution
748
+
749
+ ### Required Attribution Text
750
+ ```
751
+ Weather data provided by:
752
+ German Weather Service (Deutscher Wetterdienst - DWD)
753
+ ICON Global Weather Model
754
+ https://opendata.dwd.de/
755
+
756
+ This product uses data from the DWD ICON model.
757
+ DWD bears no responsibility for the correctness,
758
+ accuracy or completeness of the data provided.
759
+ ```
760
+
761
+ ### Terms of Use
762
+ - No warranty on data accuracy
763
+ - Users responsible for verification
764
+ - Commercial use permitted
765
+ - Must maintain attribution
766
+ - Cannot claim data as proprietary
767
+
768
+ ## Deployment Checklist
769
+
770
+ ### Pre-Production
771
+ - [ ] Set up monitoring and alerting
772
+ - [ ] Configure log rotation
773
+ - [ ] Set up automated backups
774
+ - [ ] Test failover scenarios
775
+ - [ ] Load test API endpoints
776
+ - [ ] Validate data quality
777
+ - [ ] Set up SSL certificates
778
+
779
+ ### Production Deployment
780
+ - [ ] Deploy downloader service
781
+ - [ ] Deploy API server
782
+ - [ ] Configure reverse proxy (nginx)
783
+ - [ ] Set up monitoring dashboards
784
+ - [ ] Configure automated scaling
785
+ - [ ] Test end-to-end workflow
786
+ - [ ] Document operational procedures
787
+
788
+ ### Post-Deployment
789
+ - [ ] Monitor for 48 hours
790
+ - [ ] Verify data accuracy
791
+ - [ ] Check performance metrics
792
+ - [ ] Test backup/restore
793
+ - [ ] Update documentation
794
+ - [ ] Train operations team
795
+
796
+ ## Support & Maintenance
797
+
798
+ ### Regular Maintenance Tasks
799
+ - **Daily**: Monitor system health, check logs
800
+ - **Weekly**: Verify data quality, check storage usage
801
+ - **Monthly**: Review performance metrics, update documentation
802
+ - **Quarterly**: Security updates, capacity planning
803
+
804
+ ### Troubleshooting Common Issues
805
+
806
+ #### Download Failures
807
+ ```bash
808
+ # Check DWD server status
809
+ curl -I https://opendata.dwd.de/weather/nwp/icon/grib/
810
+
811
+ # Verify network connectivity
812
+ nslookup opendata.dwd.de
813
+
814
+ # Check disk space
815
+ df -h /var/lib/weather-data/
816
+
817
+ # Review download logs
818
+ tail -f /var/log/dwd-downloader.log
819
+ ```
820
+
821
+ #### API Performance Issues
822
+ ```bash
823
+ # Check API server status
824
+ curl http://localhost:8000/health
825
+
826
+ # Monitor response times
827
+ curl -w "@curl-format.txt" http://localhost:8000/forecast
828
+
829
+ # Check memory usage
830
+ ps aux | grep dwd-api
831
+ ```
832
+
833
+ ## Contact & Support
834
+ - **Issues**: Create GitHub issue with system details
835
+ - **Documentation**: Keep this guide updated with changes
836
+ - **Monitoring**: Set up alerts for critical failures
837
+
838
+ ---
839
+
840
+ **Version**: 1.0.0
841
+ **Last Updated**: 2025-09-24
842
+ **Maintainer**: Weather API Team