Add backend initialization plan

.claude/plans/backend_init.md

@@ -0,0 +1,283 @@
+# Backend Initialization Plan
+
+## Context
+
+Build the Django backend foundation: project scaffolding, dataset/entity/relation loading, health and discovery endpoints. NO inference/generation logic — that comes later. This covers all non-prediction endpoints from `docs/api.yaml`.
+
+## Project Structure
+
+```
+src/backend/
+  manage.py
+  requirements.txt
+  research_api/
+    __init__.py
+    settings.py
+    urls.py
+    wsgi.py
+  api/
+    __init__.py
+    apps.py
+    urls.py
+    views/
+      __init__.py
+      health.py
+      coins.py
+      graph_generation.py
+      kg_anomaly.py
+    services/
+      __init__.py
+      registry.py          # Singleton: loads datasets at startup, scans checkpoints
+      coins_loader.py      # Lightweight KG dataset reader (NOT reusing research Loader)
+      constants.py         # All static/hardcoded data (models, query structures, etc.)
+    pagination.py          # Custom page/page_size pagination helper
+    exceptions.py          # Standardized error envelope
+```
+
+## Key Design Decision: Reuse Research Loaders
+
+Use the prebuilt `Loader` class from `graph_completion/graphs/load_graph.py` for COINs datasets and the equivalent loaders for generation. This ensures data loading matches the research code exactly (same entity/relation mappings, same splits). The Django service layer wraps these loaders with a thin adapter for search/pagination.
+
+Requires adding `src/research/COINs-KGGeneration` and `src/research/MultiProxAn` to `sys.path` at startup.
+
+### Dependencies for Loader
+
+The research `Loader` needs: `igraph`, `pandas`, `numpy`, `torch` (for device handling). These must be in `requirements.txt` even for the init phase.
+
+## Model Registry (singleton)
+
+`registry.py` — loaded once via `AppConfig.ready()`:
+
+1. **Downloads checkpoints from Google Drive** if not already present locally
+2. **Loads all 3 COINs datasets** using the research `Loader`
+3. **Scans checkpoint directories** for `.tar` / `.ckpt` files to determine availability flags (does NOT load model weights)
+4. **Generates sample subgraphs** for KG anomaly using the existing research code:
+   - `Loader.get_context_subgraph_dataset(max_graph_size, device)` — DFS-based partitioning of the full graph into sized subgraphs
+   - Internally calls `Sampler.get_context_subgraph_samples_dfs()` in `graph_completion/graphs/preprocess.py` (line 742)
+   - Returns `List[ContextSubgraphData]` (torch_geometric Data objects with `x`, `x_index`, `edge_index`, `edge_attr`, `subgraph_row`, `subgraph_col`)
+   - `context_subgraphs_to_tensors()` (preprocess.py line 809) converts raw DFS samples to tensors
+   - For the API sample-subgraphs endpoint: call this once per dataset at startup, store a subset, convert to the API response format (entity names + relation names resolved from the Loader's name maps)
+
+### Checkpoint Download (on each boot)
+
+Per CLAUDE.md: "backend downloads all files and places them in the `checkpoints` folders in `src/research` appropriately on each system boot".
+
+Source: Google Drive folder `https://drive.google.com/drive/u/0/folders/14Bf8fi4KJn0rDdh9y8EFyA5b8OpQyXWi`
+
+Download logic in `registry.py`:
+1. Use `gdown` library to list and download files from the shared Google Drive folder
+2. For each file, check if it already exists locally (by filename + size) — skip if present
+3. Place `.tar` files into `{COINS_COMPLETION_DIR}/checkpoints/`
+4. Place `.ckpt` files into the appropriate dir based on naming convention:
+   - `{dataset}_c.ckpt` / `{dataset}.ckpt` (MultiProxAn) -> `{MULTIPROXAN_DIR}/checkpoints/`
+   - `{dataset}_correct.ckpt` / `{dataset}.ckpt` (DiGress KG) -> `{DIGRESS_KG_DIR}/checkpoints/`
+5. Log download progress to console
+6. If download fails (no internet, quota exceeded), log warning and continue — models_loaded flags will be false
+
+Add `gdown` to `requirements.txt`.
+
+### Checkpoint Scanning (after download)
+
+- COINs (link prediction): glob `{COINS_COMPLETION_DIR}/checkpoints/{dataset}_{algorithm}.tar`
+- MultiProxAn (graph generation): glob `{MULTIPROXAN_DIR}/checkpoints/{dataset}.ckpt` and `{dataset}_c.ckpt`
+- DiGress KG (anomaly correction): glob `{DIGRESS_KG_DIR}/checkpoints/{dataset}.ckpt` and `{dataset}_correct.ckpt`
+
+## Constants (hardcoded from API spec)
+
+`constants.py` contains all static data:
+- `METHODS` — 3 research methods (id, name, thesis_section, description)
+- `COINS_DATASET_META` — dataset id -> {name, description, data_dir}
+- `COINS_MODELS` — 6 algorithms with supported_query_structures and available_datasets
+  - TransE, DistMult, ComplEx, RotatE, KBGAT: 1p only
+  - Q2B: all query structures (1p, 2p, 3p, 2i, 3i, ip, pi)
+- `QUERY_STRUCTURES` — 7 query graph templates (nodes + edges arrays)
+- `GRAPHGEN_DATASETS` — QM9 (molecular) + Community20 (synthetic)
+- `GRAPHGEN_SAMPLING_MODES` — standard (T, chain_frames) + multiprox (T, m, t, t_prime)
+- `KG_ANOMALY_DATASET_META` — freebase, wordnet, nell
+
+## Endpoints (12 total)
+
+| Method | Path | View Class | Data Source |
+|--------|------|-----------|-------------|
+| `GET` | `/health` | `HealthView` | Registry availability flags |
+| `GET` | `/methods` | `MethodsView` | `METHODS` constant |
+| `GET` | `/coins/datasets` | `CoinsDatasetsView` | Registry + `COINS_DATASET_META` |
+| `GET` | `/coins/datasets/{id}/entities` | `CoinsEntitiesView` | Registry dataset `search_entities()` |
+| `GET` | `/coins/datasets/{id}/relations` | `CoinsRelationsView` | Registry dataset `search_relations()` |
+| `GET` | `/coins/datasets/{id}/sample-triples` | `CoinsSampleTriplesView` | Registry dataset `sample_triples()` |
+| `GET` | `/coins/models` | `CoinsModelsView` | `COINS_MODELS` filtered by available checkpoints |
+| `GET` | `/coins/query-structures` | `CoinsQueryStructuresView` | `QUERY_STRUCTURES` constant |
+| `GET` | `/graph-generation/datasets` | `GraphGenDatasetsView` | `GRAPHGEN_DATASETS` filtered by checkpoints |
+| `GET` | `/graph-generation/sampling-modes` | `GraphGenSamplingModesView` | `GRAPHGEN_SAMPLING_MODES` constant |
+| `GET` | `/kg-anomaly/datasets` | `KgAnomalyDatasetsView` | `KG_ANOMALY_DATASET_META` |
+| `GET` | `/kg-anomaly/datasets/{id}/sample-subgraphs` | `KgAnomalySampleSubgraphsView` | Registry pre-computed subgraphs |
+
+## Django Settings
+
+- `DATABASES = {}` — no database
+- `INSTALLED_APPS`: `rest_framework`, `corsheaders`, `api`
+- `CORS_ALLOWED_ORIGINS`: `["https://bani57.pythonanywhere.com"]`
+- `REST_FRAMEWORK`: JSON renderer only, no default pagination, custom exception handler
+- Custom research path settings:
+  - `COINS_DATA_DIR` -> `src/research/COINs-KGGeneration/data/`
+  - `COINS_COMPLETION_DIR` -> `src/research/COINs-KGGeneration/graph_completion/`
+  - `DIGRESS_KG_DIR` -> `src/research/COINs-KGGeneration/graph_generation/` (DiGress model applied to KG data)
+  - `MULTIPROXAN_DIR` -> `src/research/MultiProxAn/`
+- Each has its own checkpoints subdirectory:
+  - COINs (link prediction): `{COINS_COMPLETION_DIR}/checkpoints/`
+  - DiGress KG (anomaly correction): `{DIGRESS_KG_DIR}/checkpoints/`
+  - MultiProxAn (graph generation): `{MULTIPROXAN_DIR}/checkpoints/`
+
+## Error Handling
+
+Custom DRF exception handler wrapping all errors in:
+```json
+{"error": {"code": "NOT_FOUND", "message": "...", "details": {}}}
+```
+
+`ApiError` base class with `NotFoundError`, `InvalidRequestError` subclasses.
+
+## Dependencies
+
+The research repos have **two incompatible Python environments**:
+
+### COINs (graph_completion) — Python 3.6.13
+From `src/research/COINs-KGGeneration/requirements.txt`:
+```
+-f https://download.pytorch.org/whl/cu113/torch_stable.html
+-f https://data.pyg.org/whl/torch-1.10.2+cu113.html
+torch==1.10.2+cu113
+torch-geometric==2.0.3
+torch-cluster==1.5.9
+torch-scatter==2.0.9
+torch-sparse==0.6.12
+torch-spline-conv==1.2.1
+igraph==0.9.11
+numpy==1.19.5
+pandas==1.1.5
+scipy==1.5.4
+PyYAML==6.0
+```
+Install: `conda create --name coins python=3.6.13 && pip install -r requirements.txt`
+
+### DiGress KG (graph_generation) + MultiProxAn — Python 3.9
+From `src/research/COINs-KGGeneration/graph_generation/requirements.txt` and `src/research/MultiProxAn/requirements.txt` (identical):
+```
+torch_geometric==2.3.1
+pytorch_lightning==2.0.4
+hydra-core==1.3.2
+omegaconf==2.3.0
+numpy==1.23
+pandas==1.4
+scipy==1.11.0
+torchmetrics==0.11.4
+wandb==0.15.4
+imageio==2.31.1
+matplotlib==3.7.1
+networkx==2.8.7
+pyemd==1.0.0
+PyGSP==0.5.1
+```
+Install requires conda for rdkit and graph-tool, plus specific torch index URL:
+```bash
+conda create -c conda-forge -n digress rdkit=2023.03.2 python=3.9
+conda activate digress
+conda install -c conda-forge graph-tool=2.45
+conda install -c "nvidia/label/cuda-11.8.0" cuda
+pip install torch==2.0.1 --index-url https://download.pytorch.org/whl/cu118
+pip install -r requirements.txt
+pip install -e .
+# Compile orca:
+cd src/analysis/orca && g++ -O2 -std=c++11 -o orca orca.cpp
+```
+
+### Backend requirements.txt
+
+The backend must unify both into one Python 3.9 environment. Since the COINs `Loader` is the only part reused for init, we pin to the DiGress/MultiProxAn versions and ensure COINs code runs under them. For PythonAnywhere (CPU only), use CPU torch wheels.
+
+```
+# Django
+django==4.2.*
+djangorestframework==3.14.*
+django-cors-headers==4.*
+
+# Checkpoint download from Google Drive
+gdown>=4.7
+
+# PyTorch (CPU only for PythonAnywhere)
+-f https://download.pytorch.org/whl/cpu/torch_stable.html
+torch==2.0.1+cpu
+torchvision==0.15.2+cpu
+torchaudio==2.0.2+cpu
+
+# PyTorch Geometric (must match torch version)
+torch-geometric==2.3.1
+
+# Research shared deps
+pytorch-lightning==2.0.4
+hydra-core==1.3.2
+omegaconf==2.3.0
+numpy==1.23
+pandas==1.4
+scipy==1.11.0
+igraph==0.9.11
+PyYAML==6.0
+networkx==2.8.7
+matplotlib==3.7.1
+imageio==2.31.1
+torchmetrics==0.11.4
+tqdm==4.65.0
+
+# Conda-only deps (must be pre-installed, not in pip requirements):
+# rdkit==2023.03.2 (conda create -c conda-forge)
+# graph-tool==2.45 (conda install -c conda-forge)
+```
+
+**Note**: rdkit and graph-tool are conda-only packages. On PythonAnywhere (pip-only), these must either be pre-compiled or the code paths that use them guarded with try/except imports.
+
+## Implementation Sequence
+
+1. Write `requirements.txt` and install dependencies (django, DRF, cors-headers, torch, igraph, scipy, pandas, numpy)
+2. Create Django project scaffolding (`manage.py`, `research_api/` settings/urls/wsgi)
+3. Configure settings — CORS, DRF, research paths (`COINS_DATA_DIR`, `COINS_COMPLETION_DIR`, `DIGRESS_KG_DIR`, `MULTIPROXAN_DIR`), add research dirs to `sys.path`
+4. Create `api` app skeleton (`apps.py`, `urls.py`)
+5. Write `constants.py` — all static data from API spec (methods, models, query structures, dataset metadata, sampling modes)
+6. Write `registry.py` — `ModelRegistry` singleton: Google Drive checkpoint download, research `Loader` for COINs datasets, checkpoint scanning across all 3 dirs
+7. Write `pagination.py` — `paginate_list()` helper
+8. Write `exceptions.py` — custom error envelope formatting
+9. Write views — one class per endpoint, using registry + constants
+10. Wire URL routing (`research_api/urls.py` -> `api/urls.py`)
+11. Hook `ModelRegistry.initialize()` in `AppConfig.ready()`
+
+## Verification
+
+1. `pip install -r requirements.txt` — all dependencies install cleanly
+2. `python manage.py check` — Django system check passes (no DB warnings expected since `DATABASES = {}`)
+3. `python manage.py runserver` — registry downloads checkpoints from Google Drive (or skips if present), loads 3 COINs datasets via research `Loader`, scans all 3 checkpoint dirs, prints entity/relation counts
+4. `GET /api/v1/health` — returns `{"status": "ok", "models_loaded": {"coins": true/false, "multiproxan": true/false, "kg_anomaly": true/false}}` based on checkpoint scan
+5. `GET /api/v1/methods` — returns 3 methods
+6. `GET /api/v1/coins/datasets` — returns 3 datasets with correct entity/relation counts (FB15k-237: ~14541 entities, 237 relations; WN18RR: ~40943 entities, 11 relations; NELL-995: from entity2id.txt/relation2id.txt)
+7. `GET /api/v1/coins/datasets/freebase/entities?q=location&page=1&page_size=10` — filtered, paginated results
+8. `GET /api/v1/coins/datasets/freebase/relations?q=country` — substring search on relation names
+9. `GET /api/v1/coins/datasets/freebase/sample-triples?count=5` — 5 random triples with resolved entity/relation names
+10. `GET /api/v1/coins/datasets/unknown/entities` — `{"error": {"code": "NOT_FOUND", ...}}`
+11. `GET /api/v1/coins/models` — 6 algorithms, filtered by actually available checkpoints
+12. `GET /api/v1/coins/query-structures` — 7 templates (1p, 2p, 3p, 2i, 3i, ip, pi) matching API spec
+13. `GET /api/v1/graph-generation/datasets` — QM9 + Community20 with model type availability from MultiProxAn checkpoint scan
+14. `GET /api/v1/graph-generation/sampling-modes` — standard + multiprox with parameter specs
+15. `GET /api/v1/kg-anomaly/datasets` — 3 datasets with availability from DiGress KG checkpoint scan
+16. `GET /api/v1/kg-anomaly/datasets/freebase/sample-subgraphs?count=3` — 3 pre-computed subgraphs with entity/relation names
+17. Import `docs/postman/collection.json` into Postman, set environment, run all GET requests against running server
+
+## Critical Source Files
+
+| File | Why |
+|------|-----|
+| `docs/api.yaml` | Authoritative API contract for all response schemas |
+| `src/research/COINs-KGGeneration/graph_completion/graphs/load_graph.py` | Research `Loader` class reused for dataset loading |
+| `src/research/COINs-KGGeneration/graph_data/load_data.py` | Dataset classes (FreeBase, WordNet, NELL) with file formats |
+| `src/research/COINs-KGGeneration/data/` | Actual data files (train/valid/test.txt, entity2id.txt, relation2id.txt) |
+| `src/research/MultiProxAn/checkpoints/` | MultiProxAn checkpoint dir (scanned for availability) |
+| `src/research/COINs-KGGeneration/graph_generation/checkpoints/` | DiGress KG checkpoint dir (scanned for availability) |
+| `src/research/COINs-KGGeneration/graph_completion/checkpoints/` | COINs checkpoint dir (scanned for availability) |
+| `CLAUDE.md` | Line length 120, Ruff rules, deployment target |