# OmniDiag Architecture (Mermaid) Modular monolith: one FastAPI process, feature modules, config-driven model-family registry. ## ١. الصورة الكبرى: Modular Monolith عملية FastAPI واحدة (deployable واحد) مقسّمة داخلياً إلى وحدات (modules) بحدود واضحة؛ الأمراض تُضاف بملف YAML دون لمس باقي الكود. ```mermaid flowchart TB subgraph CL["Clients"] direction LR SPA["React SPA - Vite, Tailwind, PWA
(Vercel)"] EMR["EMR / API clients
JWT cookie or X-API-Key"] end subgraph MW["Cross-cutting middleware (order of wrapping)"] direction LR CORS["CORS"] --> SEC["SecurityHeaders
HTTPS enforce"] --> AUD["AuditMiddleware
writes audit_logs"] --> MET["MetricsMiddleware
Prometheus"] --> RL["slowapi rate limit
per-endpoint"] end subgraph APP["FastAPI process = the monolith (backend/main.py)"] direction TB subgraph MODS["Feature modules (APIRouter each)"] direction LR AUTH["auth
/auth"] ADM["admin
/admin"] PAT["patients
/api/v4/patients"] CLIN["clinical diagnosis
/api/v4/{disease}/..."] AL["active_learning (HITL)
/api/v4/review"] MON["monitoring
/api/v4/admin/drift, mlflow"] NLP["nlp
notes_parser"] LLM["llm
report_generator"] FED["federated
SUPERSEDED prototype"] end subgraph CORE["Shared core services"] direction LR ROUTER["OmniDiagRouter
scans configs/*.yaml"] PS["probability_scale +
prevalence_correction"] SHAP["shap_service"] CF["counterfactual_generator +
diabetes_what_if_levers"] CACHE["cache.py
Redis or in-memory"] end subgraph REG["Model-family registry (backend/model_backends)"] direction LR B1["glm_ivap_conformal
HEART"] B3["ebm_platt_conformal
DIABETES NHANES"] B4["sklearn_pipeline /
sklearn_generic"] end end subgraph DATA["State and artifacts"] direction LR DB[("SQL DB
SQLite dev / PostgreSQL prod
SQLAlchemy async + Alembic")] RED[("Redis cache")] ART[("models/ bundles
built at image build")] MLF[("MLflow
runs and registry")] CFG["configs/*.yaml
single source of truth per disease"] end subgraph OBS["Observability"] PROM["Prometheus"] --> GRAF["Grafana"] end subgraph EXT["External"] direction LR DS["DeepSeek API
(OpenAI-compatible)"] BERT["BioBERT NER
HuggingFace, optional"] end %% Original Exact Connections CL --> MW MW --> MODS CLIN --> ROUTER ROUTER --> REG CLIN --> PS CLIN --> SHAP CLIN --> CF CLIN --> CACHE CACHE -.-> RED CFG --> ROUTER REG --> ART AUTH --> DB PAT --> DB ADM --> DB AL --> DB AUD -.-> DB MON --> MLF AL --> MLF MET --> PROM LLM -.-> DS NLP -.-> BERT ``` ## ٢. خريطة الوحدات (Package Map) كل مجلد = وحدة بمسؤولية واحدة. الوحدات تتحدث عبر استدعاءات داخلية (in-process) لا عبر شبكة، وهذا ما يجعله Monolith، والحدود النظيفة بينها هي ما يجعله Modular. ```mermaid flowchart LR ROOT(["OmniDiag repo"]) ROOT --> BE["backend/"] ROOT --> FE["frontend/src"] ROOT --> CF["configs/ + features/"] ROOT --> SC["scripts/ + models/ + experiment_files/"] ROOT --> OPS["Dockerfile, docker-compose, k8s, helm, deploy, .github"] ROOT --> TST["tests/ (about 60 files)"] ROOT --> DOC["docs/, evaluation_evidence/, alembic/"] BE --> m1["main.py - app wiring, lifespan,
predict/explain/batch/report/parse-notes"] BE --> m2["router.py - disease dispatcher"] BE --> m3["model_backends/ - 4 families,
auto-registered by file"] BE --> m4["heart_glm/ - stack.py, candidate.py,
reference_scores.json"] BE --> m5["auth/ - jwt, hashing, rbac, api_key"] BE --> m6["admin/ - users, audit, cache flush,
api-keys, stats, retrain"] BE --> m7["patients/ - EMR: patients, visits,
predictions, export, notes"] BE --> m8["active_learning/ - sampler, routes,
retrain, nhanes candidate"] BE --> m9["monitoring/ - drift, drift_stats,
metrics, mlflow_tracker"] BE --> m10["nlp/ + llm/"] BE --> m11["middleware/ - audit, security"] BE --> m12["db_models/ - 8 ORM tables"] BE --> m13["federated/ - superseded"] CF --> c1["heart_disease.yaml
diabetes_nhanes.yaml"] CF --> c2["base_features.py
(extension point; no live module uses it)"] SC --> s1["train_heart_glm.py
build_drift_reference.py
retrain.py, seed_db.py"] ``` ## ٣. رحلة طلب التنبؤ (Predict Request) ما يحدث من الضغط على Predict حتى ظهور النتيجة: المصادقة، الكاش، النموذج، التسجيل، وإضافة الحالات غير المؤكدة لطابور المراجعة البشرية. ```mermaid sequenceDiagram autonumber participant U as Clinician (React) participant MW as Middleware participant API as "POST /api/v4/{disease}/predict" participant AU as Auth + RBAC participant C as Cache (Redis or memory) participant R as OmniDiagRouter participant B as ModelBackend (per family) participant DB as SQL DB participant AL as Active-learning sampler U->>MW: JSON features MW->>API: rate limit, security headers, start timer API->>AU: JWT cookie or API key AU-->>API: user, roles must be doctor, nurse or super_admin API->>API: validate against schema from YAML API->>C: get(hash of disease + features) alt cache hit C-->>API: stored result (scale checked) else cache miss API->>R: predict in threadpool R->>B: engineer features, model, calibrate B-->>R: probability, decision, bounds R-->>API: result API->>C: set(result, ttl) end API->>API: record Prometheus counters (hit or miss) API->>DB: INSERT predictions (scale and threshold stored) API->>AL: uncertain? alt conformal decision is uncertain, or entropy near threshold AL->>DB: INSERT review_queue (pending) end API-->>MW: PredictResponse + Cache-Hit header MW->>DB: INSERT audit_logs (user, endpoint, status, ms) MW-->>U: result, SHAP available via /explain ``` ## ٤. الإضافة بالإعدادات: Config-driven Registry إضافة مرض = ملف YAML (وملف features فقط إن احتاجته العائلة). الـ Router لا يعرف أسماء الأمراض ولا العائلات. كل عائلة نموذج تسجّل نفسها بمجرد وجود ملفها في model_backends. ```mermaid flowchart LR subgraph FS["Filesystem"] Y1["configs/heart_disease.yaml
family: glm_ivap_conformal"] Y3["configs/diabetes_nhanes.yaml
family: ebm_platt_conformal"] end subgraph STARTUP["At startup"] SCAN["OmniDiagRouter._load_all_configs()
one entry per YAML"] IMP["model_backends/__init__.py
pkgutil imports every module"] REGI[("registry
@register_backend('name')")] IMP --> REGI end subgraph IFACE["ModelBackend interface"] I1["predict_proba, feature_names, shap_values"] I2["predict, predict_batch, explain"] I3["BackendCapabilities:
tree_shap, vectorized_batch,
counterfactuals, explainer"] end Y1 --> SCAN Y3 --> SCAN SCAN -->|family key| GET["get_backend(family)"] REGI --> GET GET -->|unknown family fails at startup| ERR["UnknownModelFamilyError"] GET --> L1["HeartGlmConformal"] GET --> L3["EbmPlattConformal"] GET -.-> L4["sklearn_pipeline / sklearn_generic"] L1 -.->|lazy load on first request| W1[("heart_l3_glm_stack.pkl")] L3 -.-> W3[("diabetes_nhanes_ebm.joblib")] L1 --- I1 L3 --- I1 ``` ## ٥. نموذج القلب: Spline-GLM + Venn-Abers + Conformal القرار ليس احتمالاً مقابل عتبة واحدة، بل مجموعة conformal تعطي referral / no_referral / uncertain، مع فاصل [p_lower, p_upper]. 7 مدخلات فقط، مدرَّب على 920 مريضاً من 4 مستشفيات. ```mermaid flowchart LR IN["7 inputs (L3)
Age, Sex, ChestPainType, RestingBP,
Cholesterol, FastingBS, RestingECG"] PRE["Frozen preprocessing
IterativeImputer, scaler,
SplineTransformer, one-hot"] GLM["Spline-GLM
logistic, ridge"] S["raw score
(log-odds, exact additive SHAP)"] IVAP["IVAP
inductive Venn-Abers"] P["calibrated p
+ interval [p_lower, p_upper]"] MC["Mondrian conformal
groups: Sex x class
alpha = 0.10"] DEC{"decision"} R1["referral"] R2["no_referral"] R3["uncertain"] Q["review_queue
(human in the loop)"] IN --> PRE --> GLM --> S S --> IVAP --> P S --> MC --> DEC DEC --> R1 DEC --> R2 DEC --> R3 --> Q subgraph LIM["Declared limits shown in config"] L1["Scope: patients already referred for catheterisation"] L2["No HIGH/MODERATE/LOW bands (risk_bands: null)"] L3["LOHO AUC 0.802, calibration does not transport between hospitals"] end DEC -.-> LIM ``` ## ٦. الإنسان في الحلقة والتعلّم النشط (HITL + Active Learning) النظام لا يتعلّم من نفسه تلقائياً: الحالات غير المؤكدة تذهب لطبيب، تسميته تُخزَّن، والإعادة التدريبية تنتج مرشَّحاً (candidate) يقرّر إنسان ترقيته. ```mermaid flowchart TB P["Prediction"] --> Q1{"module type"} Q1 -->|"conformal (heart, NHANES)"| C1["decision == uncertain"] Q1 -->|"threshold-only module
(none live; the retired BRFSS was one)"| C2["threshold-centred entropy >= 0.88
(prior-shift map sends threshold to 0.5)"] C1 --> QUEUE C2 --> QUEUE QUEUE[("review_queue
status = pending
uncertainty_score, scale, threshold")] QUEUE --> UI["ReviewQueuePanel (doctor)
GET /api/v4/review/queue"] UI --> ACT{"doctor action"} ACT -->|"annotate: label 0/1 + notes"| REV["status = reviewed"] ACT -->|"skip"| SKP["status = skipped"] REV --> RT["super_admin: POST /admin/retrain
min-samples gate"] RT --> FAM{"model.family"} FAM -->|"heart: glm_ivap_conformal"| H1["build_bundle() on 920 UCI rows
+ reviewed rows
(same code as shipped build)"] H1 --> H2["models/heart_disease/candidates/RUN_ID"] H2 --> H3["compare to SHIPPED model:
conformal coverage +
decision transition matrix"] H3 --> H4["log to MLflow"] H4 --> H5{{"HUMAN decides promotion
(no code path promotes)"}} FAM -->|"NHANES: ebm_platt_conformal"| N1["status: unsupported (not wired yet);
diabetes_nhanes_candidate.py exists:
candidate only, HbA1c provenance required"] FAM -->|"any other family"| D1["status: unsupported
(nothing read or written)"] FAM -->|"retired disease (diabetes)"| D2["410 DISEASE_RETIRED"] SKP -.-> STATS["GET /review/stats"] REV -.-> STATS ``` ## ٧. معالجة اللغة (NLP) والتقرير بالـ LLM ملاحظة سريرية نصية تتحول إلى حقول جاهزة للنموذج؛ وبعد التنبؤ يُولَّد تقرير سريري بالـ LLM مع حارس يمنع المحتوى الممنوع وبديل قاعدي عند الفشل. ```mermaid flowchart TB subgraph NLPX["NLP: POST /api/v4/parse-notes (20/min, no auth) or /patients/ID/notes"] N0["free-text note"] --> N1{"detect_script"} N1 -->|"Arabic"| N1a["language_support warning:
patterns are English only"] N1 -->|"Latin"| N2["1. Regex baseline
spelled numbers to digits, negation"] N2 --> N3["2. spaCy dependency parse
numeric mentions, linking words,
overrides regex (optional dep)"] N3 --> N4["3. BioBERT NER via HuggingFace
lazy load, overrides on overlap"] N4 --> N5["map_to_disease_schema
(heart only; NHANES has no map)"] N5 --> N6["pre-filled form
missing fields omitted, user confirms"] end N6 --> PRED["/predict"] subgraph LLMX["LLM: POST /api/v4/generate-report (10/min)"] PRED --> L0["result: probability, decision, bounds, SHAP top 5"] L0 --> L1["band_for_report
one band value used everywhere"] L1 --> L2{"DEEPSEEK_API_KEY set?"} L2 -->|"yes"| L3["DeepSeek chat via OpenAI SDK"] L3 --> L4{"forbidden_content(text)?"} L4 -->|"clean"| L5["source = llm"] L4 -->|"violations"| L6["rule-based template
fallback_reason recorded"] L2 -->|"no"| L6 L6 --> L7["source = rule_based"] end L5 --> OUT["ClinicalReportModal / PDFReport"] L7 --> OUT ``` ## ٨. الأمن والصلاحيات والتدقيق JWT في كوكي + Refresh، مفتاح API بديل للأنظمة الخارجية، صلاحيات RBAC، وسجل تدقيق لكل طلب. ```mermaid flowchart TB subgraph AUTHF["Authentication"] REG["POST /auth/register"] --> HASH["password hashing"] LOG["POST /auth/login"] --> JWT["access JWT + refresh cookie"] REF["POST /auth/refresh"] --> JWT OUT["POST /auth/logout"] KEY["X-API-Key (hash stored)
POST /admin/api-keys"] end subgraph AUTHZ["Authorisation (RBAC)"] DEP["get_current_active_user"] RR["require_role(...)"] ROLES["roles: doctor, nurse, super_admin
CLINICAL_ROLES = doctor, nurse, super_admin
ADMIN_ROLES = super_admin"] DEP --> RR --> ROLES end JWT --> DEP KEY --> DEP subgraph PROT["What each role reaches"] P1["predict, explain, patients, review: clinical roles"] P2["users, audit-logs, cache flush, retrain, drift run, mlflow: super_admin"] end RR --> P1 RR --> P2 subgraph DEF["Defence in depth"] D1["SecurityHeadersMiddleware + HTTPS enforcement"] D2["slowapi limits: LIMIT_CLINICAL, LIMIT_ADMIN, 10/min report, 20/min notes"] D3["AuditMiddleware: user, endpoint, method, status, IP, duration_ms"] D4["patient soft-delete (deleted_at)"] end D3 --> AL[("audit_logs")] ``` ## ٩. المراقبة والـ Drift و MLflow مراقبة تشغيلية (Prometheus/Grafana) ومراقبة انحراف المدخلات بإحصاءات KS و chi-square و PSI مقابل مرجع مجمَّد. الانحراف في المدخلات تنبيه للفحص وليس دليلاً على تدهور النموذج. ```mermaid flowchart LR subgraph RUNTIME["Runtime metrics"] M1["MetricsMiddleware
request duration, active requests"] M2["record_prediction
predictions_total, confidence histogram"] M3["cache hits / misses"] M4["conformal decisions, batch rows"] M5["drift_share gauge"] end M1 --> ME["GET /metrics"] M2 --> ME M3 --> ME M4 --> ME M5 --> ME ME --> PR["Prometheus
deploy/prometheus.yml"] --> GR["Grafana dashboards"] subgraph DRIFT["Input drift (Gate 8.7c)"] REFP[("drift_reference.json
frozen profile
built by build_drift_reference.py")] RECENT["recent predictions from DB"] PM["ProfileDriftMonitor
KS, chi-square, PSI on scipy"] REFP --> PM RECENT --> PM PM --> ST["GET /drift/DISEASE/status"] PM --> RUN["POST /drift/DISEASE/run"] PM --> REP["GET /drift/DISEASE/report"] RUN --> M5 end subgraph ML["MLflow"] TR["mlflow_tracker
experiment OmniDiag"] TR --> RUNS["GET /mlflow/runs"] TR --> RG["POST /mlflow/register"] end RUN --> TR RET["retrain candidates"] --> TR ``` ## ١٠. نموذج البيانات (Database) ثمانية جداول؛ كل تنبؤ يُسجَّل مع مقياسه وعتبته، وطابور المراجعة مرتبط بالتنبؤ بعلاقة واحد لواحد. ```mermaid erDiagram USERS ||--o{ USER_ROLES : has ROLES ||--o{ USER_ROLES : grants USERS ||--o{ PATIENTS : created_by PATIENTS ||--o{ PATIENT_VISITS : has PATIENTS ||--o{ PREDICTIONS : linked USERS ||--o{ PREDICTIONS : created_by PREDICTIONS ||--o| REVIEW_QUEUE : flagged USERS ||--o{ REVIEW_QUEUE : reviewer USERS ||--o{ AUDIT_LOGS : actor USERS { string id PK string email string hashed_password string api_key_hash bool is_active } ROLES { int id PK string name } PATIENTS { string id PK string mrn UK string full_name date date_of_birth datetime deleted_at } PATIENT_VISITS { string id PK string patient_id FK } PREDICTIONS { string id PK string disease float confidence string probability_scale string decision float probability_lower float probability_upper } REVIEW_QUEUE { string id PK string prediction_id UK float uncertainty_score float decision_threshold string status int label string notes } AUDIT_LOGS { string id PK string endpoint string method int status_code string ip_address int duration_ms } ``` ## ١١. الواجهة الأمامية (React SPA) الفورم يُبنى ديناميكياً من الـ schema القادم من الـ API (لا حقول مكتوبة يدوياً لكل مرض)، ولذلك تظهر الأمراض الجديدة بدون تعديل الواجهة. ```mermaid flowchart TB MAIN["main.jsx"] --> PROV["Providers: Auth, Disease, Theme"] PROV --> APP["App.jsx - DiseaseSelector, theme toggle, view tabs"] APP --> V1["Engineering Mode"] APP --> V2["Clinical EMR Mode"] APP --> V3["Batch Prediction (CSV, max 500 rows)"] APP --> V4["Patient Comparison"] APP --> V5["Review Queue (clinical roles)"] APP --> V6["AdminDashboard"] subgraph SCH["Schema-driven form"] H1["useDiseaseSchema
GET /api/v4/DISEASE/schema"] H2["schemaFieldParser + schemaToZod"] H3["DynamicClinicalForm + SchemaFieldFactory"] H4["ClinicalNotesInput (NLP prefill)"] H1 --> H2 --> H3 H4 --> H3 end V1 --> SCH V2 --> SCH subgraph RES["Result views"] R1["ThresholdBar + decision"] R2["ShapBarChart (useShapSnapshot)"] R3["WhatIfScenarioCard
counterfactuals.js + /counterfactuals"] R4["ClinicalActionPlan"] R5["ClinicalReportModal + PDFReport"] R6["PatientRiskTimeline / PatientTimeline"] end SCH --> RES V5 --> RL["ReviewLabelBox
annotate or skip"] APP --> PWA["PWA: InstallPrompt, OfflineBanner"] API["api.js - fetch wrapper, cookies"] --- SCH API --- RES API --- V5 API --- V6 ``` ## ١٢. النشر والبنية التحتية (Deployment / CI) نفس الصورة (image) تعمل محلياً وعلى Hugging Face Spaces وKubernetes. النماذج تُبنى أو تُحمَّل وقت بناء الصورة، فلا ملفات ثنائية في الريبو. ```mermaid flowchart LR DEV["Developer"] --> GH["GitHub repo"] GH --> CI["CI: .github/workflows/ci.yml
1. pip install, import backend
2. start API check
3. npm ci, npm run build"] GH --> DF["Dockerfile
python 3.13-slim"] DF --> BUILD["Build time:
train_heart_glm.py (CSV sha256 verified)
download NHANES bundle from HF Hub
(sha256-checked)"] BUILD --> IMG["Image: model baked in
port 7860, instant start"] IMG --> HF["Hugging Face Space
yahyoha/omnidiag"] IMG --> K8["Kubernetes
k8s/*.yaml, helm chart,
HPA, ingress, secrets"] IMG --> DC["docker-compose"] subgraph COMPOSE["docker-compose services"] PG[("postgres 15")] RD[("redis 7")] BE["backend"] MF["mlflow"] PRM["prometheus"] GRF["grafana"] RTN["retrain (profile: retrain)
scripts/retrain.py"] end DC --> COMPOSE BE --> PG BE --> RD BE --> MF PRM --> BE GRF --> PRM RTN --> PG RTN --> MF FE["frontend build (vite dist)"] --> VC["Vercel
SPA rewrite to index.html"] VC -->|"HTTPS API calls"| HF ``` ## ١٣. التعلّم الاتحادي (Federated Learning): الحالة الفعلية غير مُشغَّل في المنتج. النموذج الأولي (Flower + FedAvg + XGBoost) لا يستطيع التجميع، والبديل GLORE مُنفَّذ ومقاس في مستودع التجارب فقط. ```mermaid flowchart LR subgraph OLD["backend/federated/ (SUPERSEDED, not used)"] O1["client.py
pickle.dumps(XGBoost)"] O2["aggregator.py
Flower FedAvg"] O3["add_dp_noise()
never called, no epsilon"] O1 -->|"opaque bytes, cannot be averaged"| O2 O2 -.-x O3 end subgraph NEW["omnidiag_experiments (research only, not merged)"] G0["4 sites: Cleveland, Hungarian,
Switzerland, Long Beach VA"] G1["each site computes locally
gradient g and Hessian H"] G2["coordinator sums g, H
arrow-block Newton step
per-site intercept, shared slopes"] G3["7 iterations =
centralised fit (error 2e-14)"] G0 --> G1 --> G2 --> G3 G2 -->|"updated beta"| G1 end NEW --> RES["Measured: LOSO AUC 0.779 vs 0.802 shipped (-0.023)
no differential privacy, simulation on one machine"] ```