docs: add design spec for CDSE openEO EO product upgrade

Fundamental upgrade from scene-level metadata proxies to pixel-level
raster products. Replaces Open-Meteo/STAC-metadata approach with
server-side processing via CDSE openEO for Sentinel-1/2, adds SAR
change detection and built-up area monitoring, upgrades map rendering
from blank cartopy grids to raster overlays on true-color composites.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>

docs/superpowers/specs/2026-03-31-openeo-eo-upgrade-design.md

@@ -0,0 +1,457 @@
+# Aperture EO Product Upgrade — CDSE openEO
+
+**Date:** 2026-03-31
+**Status:** Draft
+**Scope:** Fundamental upgrade from scene-level metadata proxies to pixel-level raster products via CDSE openEO
+
+## Problem
+
+Aperture's current EO products are not credible to technical EO specialists:
+
+1. **No real spectral indices** — vegetation and water use Sentinel-2 scene-level metadata percentages (`s2:vegetation_percentage`, `s2:water_percentage`) instead of computing NDVI/MNDWI from pixels.
+2. **No satellite imagery in reports** — maps render as bounding boxes on blank grids. No true-color composites, no visual context.
+3. **Spatial analysis is too coarse** — rainfall, LST, and NO2 query a single centroid point via Open-Meteo. Large AOIs lose all spatial variation.
+4. **Maps and charts look amateur** — cartopy-based maps with no basemap (shapefiles fail in Docker), sparse 2-point charts, empty x-axes.
+5. **Some indicators aren't real observations** — NO2 comes from a CAMS forecast model, not satellite data. Cropland duplicates vegetation without land cover classification.
+
+## Solution
+
+Replace the data processing backbone with **CDSE openEO** (Copernicus Data Space Ecosystem). Processing graphs are defined in Python and executed server-side on ESA infrastructure. Aperture receives computed raster products (GeoTIFFs) and handles post-processing, status classification, and report rendering locally.
+
+Active fires remain on NASA FIRMS API (near-real-time, already working well).
+
+## Architecture
+
+```
+User defines AOI + period
+        │
+        ▼
+┌─────────────────────┐
+│  Aperture App       │  (HF Space, free tier)
+│  - Gradio frontend  │
+│  - Job orchestrator  │
+│  - Report renderer   │
+└────────┬────────────┘
+         │ openEO process graphs + FIRMS HTTP
+         ▼
+┌─────────────────────┐     ┌──────────────┐
+│  CDSE openEO        │     │ NASA FIRMS   │
+│  - Sentinel-1 ARD   │     │ - Fire points│
+│  - Sentinel-2 NDVI  │     └──────────────┘
+│  - CHIRPS rainfall  │
+│  - Composites       │
+│  - LST, nightlights │
+└────────┬────────────┘
+         │ GeoTIFF / JSON results
+         ▼
+┌─────────────────────┐
+│  Post-processing    │  (on Aperture app, lightweight)
+│  - Clip to AOI      │
+│  - Compute zonal stats │
+│  - Classify status  │
+│  - Render maps/PDF  │
+└─────────────────────┘
+```
+
+### openEO Processing Model
+
+Processing graphs are defined in Python using the `openeo` library. Nothing executes locally — the graph is serialized and sent to CDSE, which runs it on their infrastructure and returns a GeoTIFF.
+
+```python
+import openeo
+
+conn = openeo.connect("openeo.dataspace.copernicus.eu")
+conn.authenticate_oidc()
+
+cube = conn.load_collection(
+    "SENTINEL2_L2A",
+    spatial_extent={"west": 32.45, "south": 15.65, "east": 32.65, "north": 15.8},
+    temporal_extent=["2025-03-01", "2026-03-01"],
+    bands=["B04", "B08"]
+)
+ndvi = (cube.band("B08") - cube.band("B04")) / (cube.band("B08") + cube.band("B04"))
+monthly = ndvi.aggregate_temporal_period("month", reducer="median")
+result = monthly.download("ndvi_monthly.tif")
+```
+
+### Key Design Decisions
+
+1. **Synchronous for small jobs** — openEO supports sync (< 5 min) and async batch. For typical AOIs (< 500 km²) at 100m resolution, sync should work. Async batch with polling for larger jobs.
+
+2. **Results as GeoTIFF** — openEO returns multi-band rasters. Aperture downloads, clips, computes zonal stats, classifies status, renders maps. The existing indicator → status → report pipeline stays intact.
+
+3. **One openEO connection per job** — authenticate once, submit indicator processing graphs sequentially (memory constraint), collect results.
+
+4. **Resolution as config** — default 100m for free-tier HF Space (2GB RAM). Configurable to 10m when deployed on larger hardware. This is a spatial resampling parameter passed to openEO, not a code change.
+
+5. **Graceful fallback** — if CDSE is down or quota exhausted, fall back to current metadata-based approach with a "degraded data quality" flag in the report.
+
+6. **Caching** — downloaded GeoTIFFs stored in `results/{job_id}/`. Re-running a report for the same AOI/period skips openEO calls.
+
+## Configuration
+
+New environment variables / config constants:
+
+| Variable | Default | Description |
+|---|---|---|
+| `APERTURE_RESOLUTION_M` | `100` | Spatial resolution in meters. 100m for free tier, 10m on upgraded hardware |
+| `APERTURE_MAX_AOI_KM2` | `500` | Soft limit for AOI size on free tier |
+| `OPENEO_BACKEND` | `openeo.dataspace.copernicus.eu` | openEO backend URL |
+| `OPENEO_CLIENT_ID` | (secret) | CDSE OAuth2 client ID |
+| `OPENEO_CLIENT_SECRET` | (secret) | CDSE OAuth2 client secret |
+
+Existing config stays: `APERTURE_CORS_ORIGINS`, `APERTURE_DEMO`.
+
+## Indicator Set
+
+### Upgraded Indicators (openEO-processed)
+
+#### 1. Vegetation — NDVI (replaces vegetation + cropland)
+
+| Field | Value |
+|---|---|
+| **ID** | `ndvi` |
+| **Source** | Sentinel-2 L2A via CDSE |
+| **Method** | Pixel-level NDVI = (B08 - B04) / (B08 + B04), monthly median composites, cloud-masked (SCL band) |
+| **Baseline** | 5-year monthly NDVI medians |
+| **Status** | Anomaly = current median - baseline median. GREEN: ≥ -0.05, AMBER: -0.05 to -0.15, RED: < -0.15 |
+| **Outputs** | Monthly NDVI raster, anomaly map (current vs baseline), zonal stats |
+| **Resolution** | Configurable (default 100m, native 10m) |
+
+#### 2. Water / Flood Extent
+
+| Field | Value |
+|---|---|
+| **ID** | `water` |
+| **Source** | Sentinel-2 L2A (MNDWI) + Sentinel-1 GRD (flood backup) |
+| **Method** | MNDWI = (B03 - B11) / (B03 + B11), threshold > 0.0 for water classification. Sentinel-1 VH backscatter drop for cloud-free flood detection |
+| **Baseline** | 3-year water extent frequency map |
+| **Status** | Change in water pixel fraction. GREEN: within ±10pp, AMBER: 10-25pp change, RED: > 25pp change |
+| **Outputs** | Water mask raster, flood extent change map, area statistics |
+| **Resolution** | Configurable (default 100m, optical native 10m, SAR native 10m) |
+
+#### 3. Land Surface Temperature
+
+| Field | Value |
+|---|---|
+| **ID** | `lst` |
+| **Source** | Sentinel-3 SLSTR LST via CDSE (preferred), or MODIS MOD11A2 via NASA AppEEARS (fallback — MODIS may not be on CDSE) |
+| **Method** | Daytime LST (Band 31), quality-filtered, monthly mean composites |
+| **Baseline** | 5-year monthly means |
+| **Status** | Z-score anomaly. GREEN: |z| < 1.0, AMBER: 1.0-2.0, RED: > 2.0 |
+| **Outputs** | Monthly LST raster, anomaly map, zonal stats |
+| **Resolution** | 1 km (MODIS native) |
+
+#### 4. Rainfall — SPI
+
+| Field | Value |
+|---|---|
+| **ID** | `rainfall` |
+| **Source** | CHIRPS v2.0 pentadal/monthly. Primary: direct UCSB/IRI download (CHIRPS is not a Copernicus product and may not be on CDSE). Alternative: ERA5-Land precipitation via CDSE openEO |
+| **Method** | Monthly precipitation totals, Standardized Precipitation Index (SPI-3) against 30-year climatology |
+| **Baseline** | CHIRPS 30-year climatology (1991-2020) |
+| **Status** | SPI-3 classification. GREEN: SPI > -1.0, AMBER: -1.0 to -1.5, RED: < -1.5 (severe drought) |
+| **Outputs** | Monthly precipitation raster, SPI anomaly map, zonal stats |
+| **Resolution** | ~5 km (CHIRPS native) |
+
+#### 5. SAR Change Detection
+
+| Field | Value |
+|---|---|
+| **ID** | `sar_change` |
+| **Source** | Sentinel-1 GRD (VV + VH polarization) via CDSE |
+| **Method** | Log-ratio change detection: dB_change = 10·log10(current / baseline). Temporal composites (monthly median) to reduce speckle. VV for surface change, VH for vegetation structure |
+| **Baseline** | 12-month pre-period median backscatter |
+| **Status** | Based on area fraction with significant change (|dB_change| > 2.0). GREEN: < 5%, AMBER: 5-15%, RED: > 15% |
+| **Outputs** | Backscatter change raster (dB), classified change map, zonal stats |
+| **Resolution** | Configurable (default 100m, native 10m) |
+
+#### 6. Nightlights
+
+| Field | Value |
+|---|---|
+| **ID** | `nightlights` |
+| **Source** | VIIRS DNB monthly composites via Colorado School of Mines EOG (direct download — VIIRS is not a Copernicus product). CDSE fallback if EOG adds openEO support |
+| **Method** | Monthly mean radiance (nW·cm⁻²·sr⁻¹), stray-light corrected |
+| **Baseline** | 3-year same-month mean radiance |
+| **Status** | Percentage change from baseline. GREEN: > -15%, AMBER: -15% to -40%, RED: < -40% |
+| **Outputs** | Radiance raster, change map, zonal stats |
+| **Resolution** | ~500m (VIIRS native) |
+
+#### 7. Built-up Area Change
+
+| Field | Value |
+|---|---|
+| **ID** | `built_up` |
+| **Source** | Sentinel-1 GRD coherence pairs via CDSE |
+| **Method** | InSAR coherence between consecutive passes. Coherence drop indicates structural change (destruction). Coherence increase can indicate new construction. Monthly coherence composites |
+| **Baseline** | 12-month pre-period mean coherence |
+| **Status** | Area fraction with coherence loss > 0.3. GREEN: < 2%, AMBER: 2-10%, RED: > 10% |
+| **Outputs** | Coherence change raster, classified damage map, zonal stats |
+| **Resolution** | Configurable (default 100m, native ~20m) |
+
+### Kept Outside openEO
+
+#### 8. Active Fires
+
+| Field | Value |
+|---|---|
+| **ID** | `fires` |
+| **Source** | NASA FIRMS API (VIIRS SNPP NRT, 375m) |
+| **Method** | Unchanged — point-based fire detections, confidence filtering, 10-day chunked queries |
+| **Outputs** | Fire point GeoJSON (upgraded: rendered on true-color composite base) |
+
+### Dropped
+
+| Indicator | Reason |
+|---|---|
+| **NO2** (`no2.py`) | CAMS forecast model data, not satellite observation. Not credible as EO product |
+| **Cropland** (`cropland.py`) | Merged into NDVI vegetation — redundant without land cover classification |
+| **Food Security composite** (`food_security.py`) | Becomes a narrative section in the report, not a pseudo-indicator with traffic-light status |
+
+### Added to Every Report
+
+| Product | Source | Purpose |
+|---|---|---|
+| **True-color composite** | Sentinel-2 B04/B03/B02 via openEO | Visual context — the first thing any EO analyst looks for |
+| **False-color composite** | Sentinel-2 B08/B04/B03 via openEO | Vegetation health at a glance — red = healthy vegetation |
+
+These are rendered as full-page images in the report (before/current period side by side) and used as base layers for all indicator maps.
+
+## Map Rendering Upgrade
+
+### Current → New
+
+Replace cartopy with **rasterio + matplotlib** for raster-native rendering.
+
+| Aspect | Current | New |
+|---|---|---|
+| **Base layer** | Blank grid (cartopy fails in Docker) | True-color Sentinel-2 composite |
+| **Indicator overlay** | Colored bounding box or point scatter | Raster overlay with transparency (alpha blend on true-color base) |
+| **Colormaps** | Generic matplotlib defaults | Indicator-specific diverging colormaps (RdYlGn for NDVI, RdBu for temperature, etc.) |
+| **AOI boundary** | Orange rectangle | Subtle white/black outline on satellite imagery |
+| **Annotations** | Lat/lon gridlines only | Scale bar, north arrow, coordinate labels, colorbar with units |
+| **Legend** | None or basic | Proper legend with units, data source, date range |
+| **Output size** | 4"×3" at 150 DPI | 6"×5" at 200 DPI (higher quality for PDF) |
+
+### Map Types Produced Per Indicator
+
+Each indicator produces up to 3 maps:
+
+1. **Current-period map** — indicator raster on true-color base (e.g., NDVI values overlaid on RGB)
+2. **Anomaly/change map** — diverging colormap showing deviation from baseline (e.g., NDVI loss in red, gain in green)
+3. **Classification map** — binary/categorical overlay (e.g., flood extent, damaged areas)
+
+The summary map becomes a **multi-indicator change detection overview** — a single map showing the spatial distribution of where indicators flag RED/AMBER.
+
+### Rendering Implementation
+
+```python
+import rasterio
+import matplotlib.pyplot as plt
+import numpy as np
+
+def render_indicator_map(
+    true_color_tif: str,      # Sentinel-2 RGB composite GeoTIFF
+    indicator_tif: str,        # Indicator raster (NDVI, LST, etc.)
+    aoi_geojson: dict,         # AOI boundary
+    output_path: str,
+    cmap: str = "RdYlGn",     # Diverging colormap
+    vmin: float = None,
+    vmax: float = None,
+    alpha: float = 0.6,        # Overlay transparency
+    label: str = "",
+):
+    fig, ax = plt.subplots(figsize=(6, 5), dpi=200)
+
+    # Render true-color base
+    with rasterio.open(true_color_tif) as src:
+        rgb = src.read([1, 2, 3])
+        extent = [src.bounds.left, src.bounds.right, src.bounds.bottom, src.bounds.top]
+    rgb_normalized = np.clip(rgb / 3000, 0, 1)  # Sentinel-2 reflectance scaling
+    ax.imshow(rgb_normalized.transpose(1, 2, 0), extent=extent)
+
+    # Overlay indicator raster
+    with rasterio.open(indicator_tif) as src:
+        data = src.read(1)
+        nodata = src.nodata
+    masked = np.ma.masked_where(data == nodata, data)
+    im = ax.imshow(masked, extent=extent, cmap=cmap, alpha=alpha, vmin=vmin, vmax=vmax)
+
+    # Colorbar, AOI outline, annotations
+    plt.colorbar(im, ax=ax, label=label, shrink=0.8)
+    # ... scale bar, north arrow, title
+    fig.savefig(output_path, bbox_inches="tight")
+    plt.close(fig)
+```
+
+## Chart Rendering Upgrade
+
+### Changes
+
+| Aspect | Current | New |
+|---|---|---|
+| **Data density** | 2-point charts (baseline, current) for most indicators | 12+ monthly data points from openEO temporal aggregation |
+| **Baseline overlay** | None | Shaded min-max band + dashed mean line (from per-year monthly stats) |
+| **Anomaly mode** | Not available | For LST, rainfall: chart shows deviation from baseline, not absolute values |
+| **X-axis** | Often too wide, empty space | Tight to analysis period, monthly ticks |
+| **Before/after context** | None | Vertical line marking baseline/current boundary |
+
+The existing chart rendering code in `app/outputs/charts.py` is mostly retained — it already supports baseline overlays (from the Phase 1 plan). The main change is that every indicator now supplies dense monthly data instead of 2-point summaries.
+
+## Report Structure
+
+### Current Report Sections
+
+1. Title + metadata
+2. How to Read This Report
+3. Executive Summary (traffic-light counts)
+4. Summary map (AOI bounding box)
+5. Per-indicator: status badge, map, chart, summary, limitations
+6. Status summary table
+7. Data Sources & Methodology
+8. Disclaimer
+
+### New Report Sections
+
+1. **Title + metadata** — unchanged, add resolution and processing backend info
+2. **Visual Overview** (NEW) — full-page before/current true-color composite, side by side. This is the first thing an EO analyst wants to see.
+3. **False-color Overview** (NEW) — NIR-R-G composite for vegetation health context
+4. **Executive Summary** — traffic-light counts, unchanged
+5. **Multi-indicator Change Map** (NEW) — single spatial overview showing where indicators flag concern
+6. **Per-indicator sections** (UPGRADED):
+   - Status badge + headline
+   - Current-period map (indicator on true-color base)
+   - Anomaly/change map (diverging colormap)
+   - Monthly time-series chart with baseline band
+   - Zonal statistics table (mean, std, % area affected)
+   - Summary text + limitations
+7. **Food Security Narrative** (CHANGED) — prose section synthesizing rainfall, NDVI, LST, and fires into a food security assessment. Not a traffic-light indicator.
+8. **Status summary table** — unchanged
+9. **Data Sources & Methodology** (UPGRADED) — actual processing chain descriptions (e.g., "Sentinel-2 L2A → SCL cloud mask → NDVI = (B08-B04)/(B08+B04) → monthly median composite → 5-year anomaly")
+10. **Disclaimer** — unchanged
+
+## Files Changed
+
+### New Files
+
+| File | Purpose |
+|---|---|
+| `app/openeo_client.py` | openEO connection management, authentication, graph builders for each indicator |
+| `app/indicators/ndvi.py` | Replaces vegetation.py + cropland.py |
+| `app/indicators/sar_change.py` | New: SAR backscatter change detection |
+| `app/indicators/built_up.py` | New: coherence-based structural change |
+| `app/config.py` | Centralized configuration (resolution, AOI limits, backend URL) |
+
+### Modified Files
+
+| File | Change |
+|---|---|
+| `app/indicators/water.py` | Rewrite: MNDWI pixel-level water classification via openEO |
+| `app/indicators/lst.py` | Rewrite: MODIS LST via openEO instead of Open-Meteo |
+| `app/indicators/rainfall.py` | Rewrite: CHIRPS SPI via openEO instead of Open-Meteo |
+| `app/indicators/nightlights.py` | Rewrite: VIIRS DNB via openEO/EOG instead of Planetary Computer |
+| `app/indicators/fires.py` | Minor: map rendering upgrade (points on true-color base) |
+| `app/indicators/base.py` | Add openEO connection parameter, raster result handling |
+| `app/indicators/__init__.py` | Update registry: remove cropland/no2/food_security, add ndvi/sar_change/built_up |
+| `app/outputs/maps.py` | Rewrite: rasterio-based rendering replacing cartopy |
+| `app/outputs/charts.py` | Extend: all indicators now supply monthly data, anomaly chart mode |
+| `app/outputs/report.py` | Extend: new sections (visual overview, change map, food security narrative) |
+| `app/models.py` | Add resolution config to AOI, remove food_security from indicator list |
+| `Dockerfile` | Remove cartopy + Natural Earth. Add openeo, keep rasterio/rioxarray |
+| `pyproject.toml` | Add `openeo` dependency. Remove `cartopy`, `planetary-computer`, `stackstac` |
+
+### Deleted Files
+
+| File | Reason |
+|---|---|
+| `app/indicators/vegetation.py` | Replaced by `ndvi.py` |
+| `app/indicators/cropland.py` | Merged into `ndvi.py` |
+| `app/indicators/no2.py` | Dropped — CAMS model data, not EO |
+| `app/indicators/food_security.py` | Replaced by narrative section in report |
+
+## Deployment
+
+### Hardware
+
+Free HF Space (2GB RAM). Resolution defaults to 100m to fit memory constraints.
+
+At 100m resolution, a 500 km² AOI produces rasters of ~220×220 pixels per band per month — roughly 0.5MB per indicator product. Comfortable on 2GB.
+
+### Resolution Scaling
+
+| Deployment | `APERTURE_RESOLUTION_M` | Pixel grid (500 km² AOI) | RAM per indicator |
+|---|---|---|---|
+| Free HF Space (2GB) | `100` | ~220×220 | ~0.5 MB |
+| CPU Upgrade HF ($9/mo, 16GB) | `20` | ~1100×1100 | ~12 MB |
+| CPU Upgrade HF ($9/mo, 16GB) | `10` | ~2200×2200 | ~48 MB |
+
+Upgrading resolution requires only changing the environment variable. No code changes.
+
+### Memory Management
+
+Process indicators sequentially (one at a time). After each indicator:
+1. Download GeoTIFF from openEO
+2. Compute zonal stats
+3. Render maps
+4. Write results to disk
+5. Release raster arrays from memory
+
+### CDSE Authentication
+
+CDSE openEO supports OAuth2 client credentials:
+1. Register free CDSE account at dataspace.copernicus.eu
+2. Create client credentials in the dashboard
+3. Store as HF Space secrets: `OPENEO_CLIENT_ID`, `OPENEO_CLIENT_SECRET`
+4. `openeo` library picks them up via `conn.authenticate_oidc()`
+
+### Docker Image Changes
+
+| Remove | Add |
+|---|---|
+| `cartopy ≥0.22.0` | `openeo ≥0.28.0` |
+| `planetary-computer ≥1.0.0` | — |
+| `stackstac ≥0.5.0` | — |
+| `pystac-client ≥0.7.0` | — |
+| Natural Earth shapefile download | — |
+| `libproj-dev`, `proj-data` (builder) | — |
+
+Net effect: Docker image gets smaller (~400MB vs current ~500MB).
+
+## Testing Strategy
+
+### Unit Tests
+
+- **openEO graph builders** — mock the openEO connection, verify correct process graph structure (bands, temporal extent, spatial extent, resolution)
+- **Raster post-processing** — test zonal stats computation, status classification from raster values
+- **Map rendering** — test that rasterio-based renderer produces valid PNGs from sample GeoTIFFs
+- **Chart rendering** — test monthly data with baseline bands (extends existing tests)
+
+### Integration Tests
+
+- **openEO round-trip** — submit a minimal NDVI graph to CDSE sandbox, verify GeoTIFF returned (requires CDSE credentials, run manually or in CI with secrets)
+- **End-to-end report** — run full job with test AOI, verify PDF contains all new sections
+
+### Test Data
+
+Create sample GeoTIFFs (small, synthetic rasters) for unit tests so they run without network access or CDSE credentials.
+
+## Migration Path
+
+This is a breaking change to the indicator set. Recommended phased approach:
+
+1. **Phase A** — openEO client + NDVI indicator + new map renderer. Proves the full pipeline works end-to-end.
+2. **Phase B** — Migrate remaining indicators (water, LST, rainfall, nightlights) from current sources to openEO.
+3. **Phase C** — Add new indicators (SAR change, built-up) and visual overview pages.
+4. **Phase D** — Remove deprecated code (cartopy, Open-Meteo, Planetary Computer, old indicators).
+
+Each phase is independently deployable and testable.
+
+## Out of Scope
+
+- Interactive web maps (Leaflet/MapLibre) for indicator results — keep static PNG for now
+- Land cover classification (distinguishing crop from forest from grassland)
+- Sub-daily temporal resolution (hourly fire tracking, etc.)
+- Multi-AOI comparison in a single report
+- User-uploaded vector boundaries (shapefile/GeoJSON AOI upload)
+- Automated scheduled monitoring (cron-based re-runs)