# Co-Study4Grid Backend FastAPI service that orchestrates contingency analysis on top of [`expert_op4grid_recommender`](https://github.com/marota/Expert_op4grid_recommender) and serves the React frontend. This document is the **backend overview** — architecture, data flow, service patterns, key conventions. For deeper dives, see the linked specialised docs. --- ## At a glance ``` expert_backend/ ├── main.py # FastAPI app + endpoints ├── services/ # Domain services │ ├── network_service.py # pypowsybl network loading & queries │ ├── recommender_service.py # Analysis orchestration (singleton) │ ├── diagram_mixin.py + diagram/ # NAD / SLD rendering (mixin + 7 helpers) │ ├── analysis_mixin.py + analysis/ # Two-step analysis (mixin + 5 helpers) │ ├── simulation_mixin.py # Manual + combined actions │ ├── model_selection_mixin.py # Pluggable recommender state + getters │ ├── overflow_overlay.py # Action-pin overlay for interactive overflow viewer │ └── sanitize.py # NumPy → native-Python JSON coercion └── recommenders/ # Pluggable recommendation models ├── registry.py # register / build / list_models ├── random_basic.py # RandomRecommender (canonical example) ├── random_overflow.py # RandomOverflowRecommender (3-layer chain) ├── overflow_path_filter.py # Layer 2 of the sampling filter chain ├── network_existence.py # Layer 3 of the sampling filter chain └── synthetic_actions.py # Synthetic reco / shed / curtail builders # (model dispatch is explicit composition on # RecommenderService — no patching module) ``` --- ## Architecture ### Singletons Two top-level singletons drive every request: - **`network_service`** — owns the pypowsybl `Network` instance. Reset on every `POST /api/config`. Exposes high-level read queries (`get_disconnectable_elements`, `get_voltage_levels`, `get_element_voltage_levels`, `get_generator_type`, ...). On load it transparently resolves and decompresses a **zipped network** (`_resolve_network_file` / `_extract_network_zip`): an explicit `*.zip` path, a missing `foo.xiidm` whose sibling `foo.xiidm.zip` exists, or a directory holding only a `.zip` all Just Work (the large France 225/400 kV grid ships as `network.xiidm.zip`). Consumers point the network path at the `.xiidm` (or the `.zip`) regardless. - **`recommender_service`** — owns analysis state (the `_analysis_context` dict built by step-1, the `_dict_action` enriched action dictionary, the `_last_result`, layout caches). Composed of mixins so each concern lives in a focused file: - `DiagramMixin` → NAD / SLD generation + patch endpoints - `AnalysisMixin` → `run_analysis`, `run_analysis_step1`, `run_analysis_step2`, action enrichment - `SimulationMixin` → `simulate_manual_action`, `compute_superposition` - `ModelSelectionMixin` → recommender name + `compute_overflow_graph` toggle, inherited directly by `RecommenderService` (explicit composition — the former import-time patching module `_service_integration.py` was removed in the 2026-07 D1 revision) The mixin pattern keeps each concern ≤ 500 lines and unit-testable in isolation (see `expert_backend/tests/`). ### Pluggable recommenders The analysis pipeline does NOT hardcode the expert system. It dispatches to any class implementing the `RecommenderModel` ABC from [`expert_op4grid_recommender.models.base`](https://github.com/marota/Expert_op4grid_recommender/blob/main/expert_op4grid_recommender/models/base.py). Three models ship out of the box: `expert` (default), `random`, `random_overflow`. Third-party packages can register additional models via `@register` at import time. The **full reference** — contract, three-layer filter chain, backend / frontend wiring, step-by-step guide for plugging in a new model, troubleshooting — is in [**`docs/backend/recommender_models.md`**](recommender_models.md). ### Data flow ``` /api/config → network_service.load_network() loads pypowsybl Network │ builds dict_action │ stores recommender model name v /api/run-analysis-step1 │ run_analysis_step1(context, ...) simulates N-K contingency │ detects overloads │ picks subset that keeps │ the graph connected v /api/run-analysis-step2 (NDJSON stream) │ run_analysis_step2_graph(context) (skipped when the chosen model │ doesn't require the overflow │ graph AND the operator did │ not opt in) │ builds alphaDeesp graph │ + visualisation HTML │ run_analysis_step2_discovery(context, recommender, params) │ runs expert rule filter │ (whenever graph is available) │ calls recommender.recommend(inputs, params) │ reassesses every action │ (simulation → rho-before / rho-after │ / non-convergence / │ combined-pair superposition) │ ├→ yield { type: "pdf", pdf_path } first event so the UI can │ paint the overflow tab early └→ yield { type: "result", actions, action_scores, lines_overloaded, combined_actions, active_model, ← echoed for the saved session compute_overflow_graph, ... } ``` The two-step flow exists so the operator can pick **which** overloads to resolve before step-2 runs (the expensive part). The legacy single-shot `POST /api/run-analysis` is kept for backward compatibility. --- ## Conventions ### Per-endpoint gzip (no global middleware) Large SVG diagrams compress ~10× with gzip, but the streaming `run-analysis-step2` endpoint MUST NOT be wrapped in `GZipMiddleware` — it buffers NDJSON events and delays the early-`pdf` event the UI relies on. Instead, `main.py` exposes `_maybe_gzip_json` and `_maybe_gzip_svg_text` and the relevant non-streaming endpoints opt in per-call. See the comment at the top of `expert_backend/main.py` for the full rationale. ### NumPy → JSON coercion Everything yielded by `run_analysis_step2` and returned by `simulate_manual_action` goes through `expert_backend/services/sanitize.py::sanitize_for_json` to coerce numpy scalars / arrays into native Python types. Without this the FastAPI JSON encoder either crashes (`numpy.int64` is not JSON serialisable) or emits `NaN` / `Infinity` literals the React parser rejects. ### Mixin-based service composition `recommender_service.py` doesn't put every method in one class — it composes specialised mixins: ```python class RecommenderService( DiagramMixin, AnalysisMixin, SimulationMixin, ... ): def __init__(self): ... ``` Each mixin owns a few attributes (`_dict_action`, `_analysis_context`, `_last_result`, ...) and a small surface area. `ModelSelectionMixin` is a regular base class of `RecommenderService`; `update_config` / `reset` call `_apply_model_settings` / `_reset_model_settings` explicitly and the model-aware `run_analysis_step2` lives on `AnalysisMixin` — see [`recommender_models.md`](recommender_models.md) §4. ### Pre-extraction + idempotent helpers Where possible, step-1 outputs are propagated to downstream phases through the `context` dict instead of being recomputed. Examples: - `lines_overloaded_ids_kept` — island-prevention-guard result - `pre_existing_rho` — N-state rho of pre-existing overloads - `filtered_candidate_actions` — expert rule-filter result; available to every model on `inputs.filtered_candidate_actions` whenever the overflow graph is in context (so non-expert models that opt in via `compute_overflow_graph=True` also see it). Idempotent helper `_run_expert_action_filter(context)` returns immediately when the field is already populated. ### Defensive filters The random-recommender sampling chain (layers 1–3 in [`recommender_models.md`](recommender_models.md) §3) is **conservative on internal failure**: every layer returns the input list unchanged when its internal logic raises. A bug in one filter cannot silently empty the pool. The two non-trivial layers also handle both shapes the distribution graph may return (integer indices into `obs.name_sub` legacy build, plus `numpy.str_` / `str` names current build) — see `_resolve_node_to_name` in `overflow_path_filter.py`. --- ## Endpoints Full table lives in the top-level [`README.md`](../../README.md#api-reference). The groups, by responsibility: - **Configuration & session**: `/api/config`, `/api/user-config`, `/api/config-file-path`, `/api/models`, `/api/pick-path`, `/api/save-session`, `/api/list-sessions`, `/api/load-session`, `/api/restore-analysis-context`. - **Network introspection**: `/api/branches`, `/api/voltage-levels`, `/api/nominal-voltages`, `/api/element-voltage-levels`, `/api/voltage-level-substations`, `/api/actions`. - **Analysis**: `/api/run-analysis`, `/api/run-analysis-step1`, `/api/run-analysis-step2` (NDJSON stream), `/api/simulate-manual-action`, `/api/compute-superposition`. - **Diagrams**: `/api/network-diagram`, `/api/contingency-diagram`, `/api/contingency-diagram-patch`, `/api/action-variant-diagram`, `/api/action-variant-diagram-patch`, `/api/focused-diagram`, `/api/action-variant-focused-diagram`, `/api/n-sld`, `/api/contingency-sld`, `/api/action-variant-sld`, `/api/simulate-and-variant-diagram`, `/api/regenerate-overflow-graph`, `/results/pdf/{filename}`. ### Optional same-origin SPA hosting (0.8.0) When `COSTUDY4GRID_FRONTEND_DIST` (default `frontend/dist/`) holds an `index.html`, the built React app is mounted at `/` via `StaticFiles(html=True)`. The mount is added **last**, after every `/api/*` and `/results/*` route, so those keep priority over the catch-all; it is inert when the dist is absent, so local dev and the test suite are unaffected. This is what lets the HuggingFace Docker Space serve UI + API from a single uvicorn process on port 7860 — see [`deploy/huggingface/`](../../deploy/huggingface/) and the root `Dockerfile`. --- ## Session persistence `POST /api/save-session` writes a `session.json` snapshot of the entire analysis state to disk, plus an `interaction_log.json` and a copy of the overflow viewer HTML. The shape captures both **what was configured** (`configuration.model`, `configuration.compute_overflow_graph`) and **what was actually executed** (`analysis.active_model`, `analysis.compute_overflow_graph`), so reloaded sessions show which recommender produced the suggestions — useful when an unknown model name silently fell back to the default. Full reference: [`docs/features/save-results.md`](../features/save-results.md). The interaction log is replay-ready (every chip toggle, click, simulation, save, reload carries enough data to reproduce the gesture). Full reference: [`docs/features/interaction-logging.md`](../features/interaction-logging.md). --- ## Testing The recommender-subsystem tests live in the canonical `expert_backend/tests/` suite (they were rescued from an orphaned root `tests/` package — which no pytest config collected — in the 2026-07 D1 revision) and run in CI with the rest of the backend suite. They need the real `expert_op4grid_recommender` package (the registry imports the concrete model classes) and skip under the conftest mock layer: - `test_recommenders_registry.py` — register / unregister / build / list_models / canonical-three + `params_spec()` failure resilience. - `test_random_recommenders.py` — Random + RandomOverflow metadata, sampling cardinality, three-layer filter chain, None-vs-`[]` fallback semantics, drop-on-unknown-VL regression. - `test_overflow_path_filter.py` — `_resolve_node_to_name` covering every shape the distribution graph may return, including the `numpy.str_` regression and the underscore-in-substation-name segment-scan fix. - `test_network_existence.py` — `filter_to_existing_network_elements`, conservative fallback on introspection failure. - `test_action_enrichment.py` — `extract_action_topology` with numpy-array attribute tolerance + 4-way `set_bus` backfill + `voltage_level_id` surfacing. - `test_model_selection_mixin.py` — state defaults, settings parsing. - `test_model_composition.py` — explicit composition wiring (mixin inherited, `update_config` / `reset` delegate to the mixin, single model-aware `run_analysis_step2`, overflow-graph cache fast path, `antenna_meta` pass-through regression). - `test_models_api.py` — `ConfigRequest` schema + `GET /api/models`. Run: `pytest expert_backend/tests` (or plain `pytest`). --- ## Related docs - [Pluggable Recommendation Models](recommender_models.md) — the full plug-in reference (this folder). - [Save Results](../features/save-results.md) — session JSON shape, reload behaviour, model persistence. - [Interaction Logging](../features/interaction-logging.md) — every user event captured for replay (settings tab includes model selection). - [Interactive Overflow Analysis](../features/interactive-overflow-analysis.md) — the HTML viewer that replaced the static PDF. - [Combined Actions](../features/combined-actions.md) — superposition estimation + full pair simulation modal. - Top-level [README](../../README.md) — stack, getting started, full API reference, performance highlights. - Library-side contract: [`marota/expert_op4grid_recommender` — docs/recommender_models.md](https://github.com/marota/expert_op4grid_recommender/blob/main/docs/recommender_models.md).