| --- |
| title: BP Monitoring Pipeline |
| emoji: 🩺 |
| colorFrom: red |
| colorTo: blue |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| --- |
| <div align="center"> |
|
|
| # BP Monitoring Pipeline |
|
|
| ### Clean Architecture & System Design Documentation |
|
|
|  |
|  |
|  |
|  |
|  |
|
|
| *Dokumen ini menjelaskan arsitektur, design patterns, dan keputusan teknis yang digunakan dalam pipeline data monitoring tekanan darah berbasis sinyal PPG (Photoplethysmography) dari perangkat IoT.* |
|
|
| --- |
|
|
| **Status**: `Production-Ready Design` · **Versi**: `1.0.0` · **Terakhir Diperbarui**: `30 Mei 2026` |
|
|
| </div> |
|
|
| --- |
|
|
| ## 📑 Daftar Isi |
|
|
| - [1. Executive Summary](#1-executive-summary) |
| - [2. Architecture Overview](#2-architecture-overview) |
| - [2.1. High-Level System Architecture](#21-high-level-system-architecture) |
| - [2.2. Clean Architecture Layers](#22-clean-architecture-layers) |
| - [2.3. Dependency Rule](#23-dependency-rule) |
| - [3. Layer-by-Layer Breakdown](#3-layer-by-layer-breakdown) |
| - [3.1. Domain Layer](#31-domain-layer) |
| - [3.2. Application Layer](#32-application-layer) |
| - [3.3. Infrastructure Layer](#33-infrastructure-layer) |
| - [3.4. Interface Layer](#34-interface-layer) |
| - [3.5. Shared Module](#35-shared-module) |
| - [4. Design Principles](#4-design-principles) |
| - [4.1. SOLID Principles](#41-solid-principles) |
| - [4.2. DRY (Don't Repeat Yourself)](#42-dry-dont-repeat-yourself) |
| - [5. Design Patterns](#5-design-patterns) |
| - [6. Data Flow & ETL Pipeline](#6-data-flow--etl-pipeline) |
| - [6.1. ETL #1 — Data Ingestion](#61-etl-1--data-ingestion) |
| - [6.2. ETL #2 — Processing & Inference](#62-etl-2--processing--inference) |
| - [6.3. End-to-End Data Flow](#63-end-to-end-data-flow) |
| - [7. Project Structure](#7-project-structure) |
| - [8. Deployment Architecture](#8-deployment-architecture) |
| - [8.1. Deployment Topology](#81-deployment-topology) |
| - [8.2. Technology Stack](#82-technology-stack) |
| - [8.3. Entry Points](#83-entry-points) |
| - [9. Configuration Management](#9-configuration-management) |
| - [10. Testing Strategy](#10-testing-strategy) |
| - [11. Glossary](#11-glossary) |
| - [12. References](#12-references) |
|
|
| --- |
|
|
| ## 1. Executive Summary |
|
|
| **BP Monitoring Pipeline** adalah sistem data pipeline yang dirancang untuk memproses sinyal **PPG (Photoplethysmography)** dari perangkat IoT melalui smartphone, kemudian melakukan inferensi menggunakan model deep learning (**GAN** dan **VGTL-Net**) untuk memprediksi **ABP (Arterial Blood Pressure)** secara non-invasif. |
|
|
| ### Tujuan Utama |
|
|
| | # | Tujuan | Deskripsi | |
| |---|--------|-----------| |
| | 1 | **Reliability** | Menjamin data PPG dari sensor IoT tidak hilang selama proses transmisi dan penyimpanan | |
| | 2 | **Scalability** | Arsitektur yang dapat berkembang seiring bertambahnya jumlah sensor dan pengguna | |
| | 3 | **Maintainability** | Codebase yang mudah dipahami, dimodifikasi, dan di-debug oleh seluruh anggota tim | |
| | 4 | **Replaceability** | Setiap komponen eksternal (database, message broker, model AI) dapat diganti tanpa mengubah business logic | |
| | 5 | **Testability** | Setiap layer dapat di-test secara independen melalui unit test dan integration test | |
|
|
| ### Keputusan Arsitektural Utama |
|
|
| | Keputusan | Pilihan | Alasan | |
| |-----------|---------|--------| |
| | Architecture Style | Clean Architecture | Separation of concerns, dependency inversion, framework independence | |
| | Communication Pattern | Asynchronous Messaging (Message Queue) | Decoupling antara Data Node dan Model Node | |
| | Data Pipeline Pattern | ETL (Extract-Transform-Load) | Data harus di-transform sebelum masuk ke downstream (model dan database) | |
| | Deployment Strategy | Multi-platform (HF Spaces + Kaggle) | Cost optimization ($0) dengan memanfaatkan free tier | |
|
|
| --- |
|
|
| ## 2. Architecture Overview |
|
|
| ### 2.1. High-Level System Architecture |
|
|
| Sistem terdiri dari **tiga node** utama yang berkomunikasi secara asinkron: |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────────────────┐ |
| │ SYSTEM ARCHITECTURE │ |
| │ │ |
| │ ┌──────────┐ ┌──────────────┐ ┌──────────────────┐ │ |
| │ │ IoT Node │───────→│ Data Node │───────→│ Model Node │ │ |
| │ │ (Sensor │ PPG │ (ETL #1) │ Queue │ (ETL #2) │ │ |
| │ │ + HP) │ Signal │ │ │ │ │ |
| │ └──────────┘ │ ┌────────┐ │ │ ┌────────────┐ │ │ |
| │ │ │FastAPI │ │ │ │ GAN │ │ │ |
| │ │ │+ ETL │ │ │ │ + VGTL-Net │ │ │ |
| │ │ └────────┘ │ │ └────────────┘ │ │ |
| │ └──────┬───────┘ └────────┬─────────┘ │ |
| │ │ │ │ |
| │ ▼ ▼ │ |
| │ ┌──────────────────────────────────┐ │ |
| │ │ Database │ │ |
| │ │ (Raw PPG + Predictions) │ │ |
| │ └──────────────────────────────────┘ │ |
| │ ▲ │ |
| │ │ │ |
| │ ┌──────┴───────┐ │ |
| │ │ Application │ │ |
| │ │ Node │ │ |
| │ │ (Frontend + │ │ |
| │ │ Backend) │ │ |
| │ └──────────────┘ │ |
| └─────────────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| ### 2.2. Clean Architecture Layers |
|
|
| Seluruh codebase diorganisasi ke dalam **empat layer konsentris** sesuai prinsip Clean Architecture oleh Robert C. Martin: |
|
|
| ``` |
| ╔═══════════════════════════════════════════════════════╗ |
| ║ ║ |
| ║ ┌───────────────────────────────────────────┐ ║ |
| ║ │ INTERFACE LAYER │ ║ |
| ║ │ (FastAPI Routes, CLI, Consumer) │ ║ |
| ║ │ │ ║ |
| ║ │ ┌───────────────────────────────────┐ │ ║ |
| ║ │ │ INFRASTRUCTURE LAYER │ │ ║ |
| ║ │ │ (PostgreSQL, RabbitMQ, SciPy, │ │ ║ |
| ║ │ │ Model Service Implementations) │ │ ║ |
| ║ │ │ │ │ ║ |
| ║ │ │ ┌───────────────────────────┐ │ │ ║ |
| ║ │ │ │ APPLICATION LAYER │ │ │ ║ |
| ║ │ │ │ (Use Cases, DTOs) │ │ │ ║ |
| ║ │ │ │ │ │ │ ║ |
| ║ │ │ │ ┌─────────────────┐ │ │ │ ║ |
| ║ │ │ │ │ DOMAIN LAYER │ │ │ │ ║ |
| ║ │ │ │ │ (Entities, │ │ │ │ ║ |
| ║ │ │ │ │ Interfaces, │ │ │ │ ║ |
| ║ │ │ │ │ Value Objects)│ │ │ │ ║ |
| ║ │ │ │ └─────────────────┘ │ │ │ ║ |
| ║ │ │ └───────────────────────────┘ │ │ ║ |
| ║ │ └───────────────────────────────────┘ │ ║ |
| ║ └───────────────────────────────────────────┘ ║ |
| ║ ║ |
| ╚═══════════════════════════════════════════════════════╝ |
| ``` |
|
|
| ### 2.3. Dependency Rule |
|
|
| > **"Source code dependencies harus selalu mengarah KE DALAM (ke arah Domain Layer)."** |
| > — Robert C. Martin, *Clean Architecture* (2017) |
|
|
| ``` |
| Interface ──depends on──→ Application ──depends on──→ Domain |
| ▲ |
| Infrastructure ──────────────implements────────────────┘ |
| ``` |
|
|
| **Implikasi penting:** |
|
|
| - Domain Layer **tidak mengetahui** keberadaan PostgreSQL, RabbitMQ, FastAPI, atau framework apapun. |
| - Application Layer (Use Cases) bergantung pada **interfaces** yang didefinisikan di Domain, bukan pada implementasi konkret. |
| - Infrastructure Layer **mengimplementasikan** interfaces Domain, sehingga bisa di-swap tanpa mengubah business logic. |
| - Interface Layer bertindak sebagai **entry point** — hanya melakukan wiring/composition. |
|
|
| --- |
|
|
| ## 3. Layer-by-Layer Breakdown |
|
|
| ### 3.1. Domain Layer |
|
|
| **Lokasi**: `src/domain/` |
|
|
| Domain Layer adalah **inti dari sistem** — berisi business rules murni yang tidak bergantung pada teknologi apapun. Layer ini harus bisa berjalan **tanpa framework, tanpa database, tanpa network**. |
|
|
| #### 3.1.1. Entities |
|
|
| Entity merepresentasikan objek bisnis utama dengan identitas dan lifecycle. |
|
|
| | Entity | File | Deskripsi | Key Properties | |
| |--------|------|-----------|----------------| |
| | `PPGSignal` | `entities/ppg_signal.py` | Representasi sinyal PPG yang masuk dari sensor | `device_id`, `user_id`, `timestamp`, `sampling_rate`, `ppg_values`, `duration_seconds` | |
| | `BPPrediction` | `entities/prediction.py` | Hasil prediksi tekanan darah dari model AI | `predicted_sbp`, `predicted_dbp`, `model_version`, `inference_time_ms` | |
|
|
| **Karakteristik Entity:** |
| - Menggunakan `@dataclass` murni Python — **bukan** SQLAlchemy model, **bukan** Pydantic model. |
| - Berisi **domain validation** (`validate()`) yang merepresentasikan business rules. |
| - Berisi **computed properties** (e.g., `num_samples`, `mean_arterial_pressure`). |
|
|
| #### 3.1.2. Value Objects |
|
|
| Value Object adalah objek immutable yang didefinisikan oleh atributnya, bukan identitasnya. |
|
|
| | Value Object | File | Deskripsi | |
| |-------------|------|-----------| |
| | `DeviceInfo` | `value_objects/device_info.py` | Informasi perangkat sensor IoT | |
| | `SignalMetadata` | `value_objects/signal_metadata.py` | Metadata sinyal (sampling rate, durasi, jumlah sampel) | |
|
|
| #### 3.1.3. Interfaces (Contracts) |
|
|
| Interfaces mendefinisikan **kontrak** yang harus dipenuhi oleh Infrastructure Layer. Menggunakan Abstract Base Class (ABC). |
|
|
| | Interface | File | Kontrak | |
| |-----------|------|---------| |
| | `BaseRepository[T]` | `interfaces/repositories/base_repository.py` | Generic CRUD: `add()`, `get_by_id()`, `get_all()`, `delete()` | |
| | `PPGRepository` | `interfaces/repositories/ppg_repository.py` | Extends `BaseRepository[PPGSignal]` + `get_by_user()`, `get_by_device()` | |
| | `PredictionRepository` | `interfaces/repositories/prediction_repository.py` | Extends `BaseRepository[BPPrediction]` + query spesifik prediksi | |
| | `MessageBroker` | `interfaces/services/message_broker.py` | `publish()`, `consume()`, `connect()`, `disconnect()` | |
| | `SignalProcessor` | `interfaces/services/signal_processor.py` | `filter_signal()`, `normalize()`, `segment()`, `process()` | |
| | `ModelService` | `interfaces/services/model_service.py` | `load_model()`, `predict()` | |
|
|
| > **📌 Catatan Penting**: Interface `MessageBroker` tidak menyebut "RabbitMQ" dimanapun. Jika di kemudian hari ingin berpindah ke **Apache Kafka** atau **Redis Streams**, cukup buat implementasi baru tanpa mengubah satu baris pun di Domain atau Application Layer. |
|
|
| #### 3.1.4. Domain Exceptions |
|
|
| ``` |
| DomainException (base) |
| ├── InvalidSignalError |
| ├── PredictionOutOfRangeError |
| └── EntityNotFoundError |
| ``` |
|
|
| --- |
|
|
| ### 3.2. Application Layer |
|
|
| **Lokasi**: `src/application/` |
|
|
| Application Layer berisi **use cases** — orkestrasi langkah-langkah business logic untuk memenuhi satu kebutuhan pengguna yang spesifik. |
|
|
| #### 3.2.1. Use Cases |
|
|
| | Use Case | File | Deskripsi | Input → Output | |
| |----------|------|-----------|----------------| |
| | `IngestPPGUseCase` | `use_cases/ingest_ppg.py` | ETL #1: Terima PPG → Store → Publish | `PPGIngestRequest` → `PPGIngestResponse` | |
| | `ProcessAndPredictUseCase` | `use_cases/process_and_predict.py` | ETL #2: Consume → Preprocess → Predict → Store | `dict (message)` → `BPPrediction` | |
| | `GetPredictionHistoryUseCase` | `use_cases/get_prediction_history.py` | Query riwayat prediksi per user | `user_id, date_range` → `list[BPPrediction]` | |
|
|
| **Prinsip Use Case:** |
| - Setiap use case memiliki **satu method publik**: `execute()`. |
| - Dependencies di-inject melalui **constructor** (Dependency Inversion). |
| - Use case **tidak mengetahui** dari mana dia dipanggil (API? CLI? Consumer?). |
|
|
| ``` |
| ┌──────────────────────────────────────────────────────────────┐ |
| │ IngestPPGUseCase │ |
| │ │ |
| │ __init__(ppg_repo: PPGRepository, broker: MessageBroker) │ |
| │ ▲ ▲ │ |
| │ │ interface │ interface │ |
| │ │ │ │ |
| │ execute(request: PPGIngestRequest) → PPGIngestResponse │ |
| │ 1. Create PPGSignal entity │ |
| │ 2. entity.validate() │ |
| │ 3. ppg_repo.add(entity) │ |
| │ 4. broker.publish("ppg_queue", message) │ |
| │ 5. Return response DTO │ |
| └──────────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| #### 3.2.2. Data Transfer Objects (DTOs) |
|
|
| DTO digunakan untuk transfer data antar layer boundary. Menggunakan **Pydantic** untuk validasi otomatis. |
|
|
| | DTO | Tipe | Deskripsi | |
| |-----|------|-----------| |
| | `PPGIngestRequest` | Input | Data yang diterima dari HP via API | |
| | `PPGIngestResponse` | Output | Response setelah ingest berhasil | |
| | `PredictionResponse` | Output | Hasil prediksi untuk ditampilkan ke frontend | |
|
|
| > **Kenapa bukan Entity langsung?** |
| > Entity adalah representasi domain murni. DTO adalah representasi "bentuk data" yang masuk/keluar dari sistem. Memisahkan keduanya mencegah domain entity terkontaminasi oleh kebutuhan serialization/validation framework. |
|
|
| --- |
|
|
| ### 3.3. Infrastructure Layer |
|
|
| **Lokasi**: `src/infrastructure/` |
|
|
| Infrastructure Layer berisi **implementasi konkret** dari interfaces yang didefinisikan di Domain Layer. Layer ini berhubungan langsung dengan teknologi pihak ketiga. |
|
|
| #### 3.3.1. Database |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `connection.py` | `database/connection.py` | SQLAlchemy async engine + session factory | |
| | `Base` | `database/models/base.py` | Declarative base dengan `id` dan `created_at` otomatis | |
| | `PPGModel` | `database/models/ppg_model.py` | ORM model untuk tabel `raw_ppg` | |
| | `PredictionModel` | `database/models/prediction_model.py` | ORM model untuk tabel `predictions` | |
| | `SQLAlchemyBaseRepository` | `database/repositories/sqlalchemy_base.py` | Generic CRUD implementation (DRY) | |
| | `SQLAlchemyPPGRepository` | `database/repositories/ppg_repository.py` | Implementasi `PPGRepository` | |
| | `SQLAlchemyPredictionRepository` | `database/repositories/prediction_repository.py` | Implementasi `PredictionRepository` | |
|
|
| **Mapping Pattern:** |
|
|
| ``` |
| Domain Entity (PPGSignal) ←─── _to_entity() ─── ORM Model (PPGModel) |
| ─── _to_model() ───→ |
| ``` |
|
|
| Setiap repository konkret wajib mengimplementasikan dua method mapping: |
| - `_to_entity(model) → entity` — Mengkonversi ORM model ke domain entity. |
| - `_to_model(entity) → model` — Mengkonversi domain entity ke ORM model. |
|
|
| #### 3.3.2. Messaging |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `RabbitMQBroker` | `messaging/rabbitmq_broker.py` | Implementasi `MessageBroker` menggunakan `aio-pika` | |
|
|
| Mendukung koneksi **AMQP** (lokal) dan **AMQPS** (CloudAMQP — SSL) melalui konfigurasi URL. |
|
|
| #### 3.3.3. Signal Processing |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `ScipySignalProcessor` | `processing/scipy_signal_processor.py` | Implementasi `SignalProcessor` menggunakan SciPy | |
|
|
| Pipeline pemrosesan sinyal: |
|
|
| ``` |
| Raw PPG Signal |
| │ |
| ▼ |
| ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ |
| │ Bandpass Filter │────→│ Normalization │────→│ Segmentation │ |
| │ (0.5 – 8.0 Hz) │ │ (Z-score) │ │ (8s windows) │ |
| │ Butterworth 4th │ │ │ │ │ |
| └─────────────────┘ └─────────────────┘ └─────────────────┘ |
| │ |
| ▼ |
| Processed Segments |
| shape: (N, window_size) |
| ``` |
|
|
| #### 3.3.4. Model Service |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `MockModelService` | `model/mock_model_service.py` | Mock implementation untuk testing tanpa model asli | |
| | `GANVGTLNetService` | `model/gan_vgtlnet_service.py` | Implementasi production: GAN + VGTL-Net inference | |
|
|
| --- |
|
|
| ### 3.4. Interface Layer |
|
|
| **Lokasi**: `src/interface/` |
|
|
| Interface Layer adalah **entry point** ke sistem. Layer ini bertanggung jawab atas: |
| - Menerima input dari dunia luar (HTTP request, message queue, CLI command). |
| - Melakukan **dependency wiring** (composition root). |
| - Memanggil use case yang sesuai. |
| - Mengembalikan output dalam format yang diharapkan. |
|
|
| #### 3.4.1. API (Entry Point 1 — HF Spaces) |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `create_app()` | `api/app.py` | FastAPI application factory | |
| | `dependencies.py` | `api/dependencies.py` | Dependency injection wiring menggunakan `Depends()` | |
| | `ppg_routes.py` | `api/routes/ppg_routes.py` | REST endpoint untuk ingest PPG | |
| | `prediction_routes.py` | `api/routes/prediction_routes.py` | REST endpoint untuk query predictions | |
|
|
| #### 3.4.2. Consumer (Entry Point 2 — Kaggle) |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `run_consumer.py` | `consumer/run_consumer.py` | Standalone consumer service | |
| | `colab_notebook.ipynb` | `consumer/colab_notebook.ipynb` | Jupyter notebook untuk menjalankan consumer di Colab | |
|
|
| > **📌 Kedua entry point menggunakan use case, repository, dan interface yang PERSIS SAMA.** Perbedaannya hanya pada cara bootstrapping dan dependency wiring. |
|
|
| --- |
|
|
| ### 3.5. Shared Module |
|
|
| **Lokasi**: `src/shared/` |
|
|
| Cross-cutting concerns yang digunakan oleh seluruh layer. |
|
|
| | Komponen | File | Deskripsi | |
| |----------|------|-----------| |
| | `Settings` | `config.py` | Konfigurasi aplikasi dari environment variables (Pydantic Settings) | |
| | `get_logger()` | `logger.py` | Logging factory dengan format konsisten | |
| | `constants.py` | `constants.py` | Named constants (menghilangkan magic numbers) | |
|
|
| --- |
|
|
| ## 4. Design Principles |
|
|
| ### 4.1. SOLID Principles |
|
|
| #### **S — Single Responsibility Principle** |
|
|
| > *"Sebuah class harus memiliki satu, dan hanya satu, alasan untuk berubah."* |
|
|
| | Komponen | Responsibility Tunggal | |
| |----------|----------------------| |
| | `PPGSignal` | Merepresentasikan sinyal PPG dan validasi domain-nya | |
| | `IngestPPGUseCase` | Mengorkestrasikan alur ingest data PPG | |
| | `ScipySignalProcessor` | Memproses sinyal menggunakan SciPy | |
| | `SQLAlchemyPPGRepository` | Mempersist PPGSignal ke PostgreSQL | |
| | `RabbitMQBroker` | Mengirim/menerima pesan via RabbitMQ | |
| | `ppg_routes.py` | Menangani HTTP routing untuk endpoint PPG | |
|
|
| #### **O — Open/Closed Principle** |
|
|
| > *"Entitas software harus terbuka untuk ekstensi, tertutup untuk modifikasi."* |
|
|
| | Skenario | Solusi (Tanpa Modifikasi) | |
| |----------|--------------------------| |
| | Ganti PostgreSQL → MongoDB | Buat `MongoDBPPGRepository` yang implement `PPGRepository`. Use case **tidak berubah**. | |
| | Ganti RabbitMQ → Kafka | Buat `KafkaBroker` yang implement `MessageBroker`. Use case **tidak berubah**. | |
| | Ganti SciPy → PyTorch preprocessing | Buat `TorchSignalProcessor` yang implement `SignalProcessor`. Use case **tidak berubah**. | |
| | Tambah model baru selain GAN+VGTL-Net | Buat `NewModelService` yang implement `ModelService`. Use case **tidak berubah**. | |
|
|
| #### **L — Liskov Substitution Principle** |
|
|
| > *"Objek dari superclass harus bisa digantikan oleh objek subclass tanpa merusak program."* |
|
|
| ```python |
| # Keduanya bisa digunakan di mana pun PPGRepository dibutuhkan |
| repo: PPGRepository = SQLAlchemyPPGRepository(session) # ✅ Production |
| repo: PPGRepository = InMemoryPPGRepository() # ✅ Testing |
| |
| # Keduanya bisa digunakan di mana pun MessageBroker dibutuhkan |
| broker: MessageBroker = RabbitMQBroker() # ✅ Production |
| broker: MessageBroker = InMemoryBroker() # ✅ Testing |
| ``` |
|
|
| #### **I — Interface Segregation Principle** |
|
|
| > *"Client tidak boleh dipaksa bergantung pada interface yang tidak mereka gunakan."* |
|
|
| ``` |
| ┌──── PPGRepository ────┐ |
| │ get_by_user() │ |
| BaseRepository[T] │ get_by_device() │ |
| add() └──────────────────────┘ |
| get_by_id() |
| get_all() ┌── PredictionRepository ──┐ |
| delete() │ get_by_user_latest() │ |
| │ get_by_date_range() │ |
| └──────────────────────────┘ |
| ``` |
|
|
| - `IngestPPGUseCase` hanya bergantung pada `PPGRepository` dan `MessageBroker` — **bukan** pada `PredictionRepository` atau `SignalProcessor`. |
| - `ProcessAndPredictUseCase` bergantung pada `PPGRepository`, `PredictionRepository`, `SignalProcessor`, dan `ModelService` — masing-masing interface kecil dan fokus. |
|
|
| #### **D — Dependency Inversion Principle** |
|
|
| > *"Module high-level tidak boleh bergantung pada module low-level. Keduanya harus bergantung pada abstraksi."* |
|
|
| ``` |
| ┌──────────────────┐ ┌──────────────────────┐ |
| │ Use Case │ │ Interface (ABC) │ |
| │ (High-Level) │──────→│ PPGRepository │ |
| │ │ │ MessageBroker │ |
| └──────────────────┘ └──────────┬───────────┘ |
| │ |
| │ implements |
| │ |
| ┌──────────▼───────────┐ |
| │ Concrete Class │ |
| │ (Low-Level) │ |
| │ SQLAlchemyPPGRepo │ |
| │ RabbitMQBroker │ |
| └──────────────────────┘ |
| ``` |
|
|
| Use case menerima dependency melalui **constructor injection**: |
|
|
| ```python |
| class IngestPPGUseCase: |
| def __init__( |
| self, |
| ppg_repo: PPGRepository, # ← interface, bukan SQLAlchemyPPGRepository |
| broker: MessageBroker, # ← interface, bukan RabbitMQBroker |
| ): |
| self._ppg_repo = ppg_repo |
| self._broker = broker |
| ``` |
|
|
| --- |
|
|
| ### 4.2. DRY (Don't Repeat Yourself) |
|
|
| | Penerapan | Lokasi | Sebelum (Redundant) | Sesudah (DRY) | |
| |-----------|--------|---------------------|---------------| |
| | Generic CRUD | `SQLAlchemyBaseRepository` | Setiap repository menulis `add()`, `get_by_id()`, `delete()` sendiri-sendiri | Tulis sekali di base class, semua repo inherit | |
| | ORM Base Model | `database/models/base.py` | Setiap ORM model mendefinisikan `id` dan `created_at` | Definisi di `Base`, semua model inherit | |
| | Magic Numbers | `shared/constants.py` | `0.5` dan `8.0` tersebar di banyak file | `PPG_BANDPASS_LOW = 0.5` didefinisikan sekali | |
| | Signal Processing Pipeline | `SignalProcessor.process()` | Urutan `filter → normalize → segment` ditulis ulang di setiap pemanggil | Template Method di base class: `process()` memanggil tiga langkah secara berurutan | |
| | Logger Creation | `shared/logger.py` | `logging.getLogger()` + handler setup di setiap file | `get_logger(__name__)` — factory function | |
|
|
| --- |
|
|
| ## 5. Design Patterns |
|
|
| | Pattern | Lokasi | Penggunaan | |
| |---------|--------|------------| |
| | **Repository Pattern** | `domain/interfaces/repositories/` | Abstraksi akses data, memisahkan domain dari persistence layer | |
| | **Factory Pattern** | `interface/api/app.py` → `create_app()` | Membuat instance FastAPI application dengan konfigurasi yang konsisten | |
| | **Dependency Injection** | `interface/api/dependencies.py` | Wiring dependencies menggunakan FastAPI `Depends()` | |
| | **Template Method** | `SignalProcessor.process()` | Mendefinisikan skeleton algoritma; subclass override langkah-langkah spesifik | |
| | **Data Transfer Object (DTO)** | `application/dto/` | Transfer data antar layer boundary tanpa mengekspos domain entity | |
| | **Data Mapper** | `_to_entity()` / `_to_model()` | Mapping antara domain entity dan ORM model | |
| | **Observer (Pub/Sub)** | `MessageBroker` + RabbitMQ | Publisher (ETL #1) dan Consumer (ETL #2) loosely coupled melalui message queue | |
| | **Strategy Pattern** | `SignalProcessor`, `ModelService` | Algoritma preprocessing dan model inference dapat di-swap melalui interface | |
| | **Singleton** | `_broker` di `dependencies.py` | Satu instance broker di-share ke seluruh request | |
|
|
| --- |
|
|
| ## 6. Data Flow & ETL Pipeline |
|
|
| ### 6.1. ETL #1 — Data Ingestion |
|
|
| **Dijalankan di**: 🤗 Hugging Face Spaces |
| **Trigger**: HTTP POST request dari mobile app |
|
|
| ``` |
| ┌─────────┐ ┌────────────────────────────────────────────────┐ |
| │ Mobile │ │ ETL #1 (HF Spaces) │ |
| │ App │ │ │ |
| │ (HP) │─────→│ ┌───────────┐ ┌──────────────┐ ┌────────┐ │ |
| │ │ POST │ │ EXTRACT │→ │ TRANSFORM │→ │ LOAD │ │ |
| └─────────┘ │ │ │ │ │ │ │ │ |
| │ │ Parse │ │ Create │ │ ┌────┐ │ │ |
| │ │ JSON body │ │ PPGSignal │ │ │ DB │ │ │ |
| │ │ Validate │ │ entity │ │ └────┘ │ │ |
| │ │ DTO │ │ domain │ │ ┌────┐ │ │ |
| │ │ │ │ validation │ │ │ MQ │ │ │ |
| │ └───────────┘ └──────────────┘ │ └────┘ │ │ |
| │ └────────┘ │ |
| └────────────────────────────────────────────────┘ |
| ``` |
|
|
| | Tahap | Aksi | Output | |
| |-------|------|--------| |
| | **Extract** | Parse JSON request body, validasi melalui Pydantic DTO | `PPGIngestRequest` object | |
| | **Transform** | Buat `PPGSignal` entity, jalankan `entity.validate()` | Validated `PPGSignal` entity | |
| | **Load** | `ppg_repo.add()` → Supabase; `broker.publish()` → CloudAMQP | Stored record + queued message | |
|
|
| ### 6.2. ETL #2 — Processing & Inference |
|
|
| **Dijalankan di**: 🧪 Kaggle (GPU) |
| **Trigger**: Message baru di RabbitMQ queue |
|
|
| ``` |
| ┌───────────┐ ┌─────────────────────────────────────────────────────┐ |
| │ CloudAMQP │ │ ETL #2 (Kaggle) │ |
| │ (Queue) │ │ │ |
| │ │─────→│ ┌───────────┐ ┌──────────────┐ ┌──────────────┐ │ |
| │ │ msg │ │ EXTRACT │→ │ TRANSFORM │→ │ LOAD │ │ |
| └───────────┘ │ │ │ │ │ │ │ │ |
| │ │ Consume │ │ Bandpass │ │ GAN predict │ │ |
| │ │ message │ │ filter │ │ VGTL-Net │ │ |
| │ │ Fetch PPG │ │ Normalize │ │ predict │ │ |
| │ │ from DB │ │ Segment │ │ Store to DB │ │ |
| │ └───────────┘ └──────────────┘ └──────────────┘ │ |
| └─────────────────────────────────────────────────────┘ |
| ``` |
|
|
| | Tahap | Aksi | Output | |
| |-------|------|--------| |
| | **Extract** | Consume message dari queue, fetch full PPG signal dari database | `PPGSignal` entity | |
| | **Transform** | Bandpass filter → Z-score normalization → 8-second segmentation | `np.ndarray` shape `(N, window_size)` | |
| | **Load** | GAN inference → VGTL-Net inference → `prediction_repo.add()` | `BPPrediction` stored in database | |
|
|
| ### 6.3. End-to-End Data Flow |
|
|
| ``` |
| 📱 HP 🤗 HF Spaces 🐰 CloudAMQP 🧪 Kaggle ☁️ Supabase |
| │ │ │ │ │ |
| │ POST /ppg/ingest │ │ │ │ |
| ├───────────────────────→│ │ │ │ |
| │ │ │ │ │ |
| │ │── Store raw PPG ──────────────────────────────────────────────→│ |
| │ │ │ │ │ |
| │ │── Publish message ───→│ │ │ |
| │ │ │ │ │ |
| │ ← 200 OK (response) │ │ │ │ |
| │←───────────────────────│ │ │ │ |
| │ │ │ │ │ |
| │ │ │── Consume ─────────→│ │ |
| │ │ │ │ │ |
| │ │ │ │── Fetch raw PPG ──→│ |
| │ │ │ │←── PPG data ───────│ |
| │ │ │ │ │ |
| │ │ │ │ [ETL #2] │ |
| │ │ │ │ Filter → Norm │ |
| │ │ │ │ → Segment │ |
| │ │ │ │ │ |
| │ │ │ │ [Model] │ |
| │ │ │ │ GAN → VGTL-Net │ |
| │ │ │ │ │ |
| │ │ │ │── Store prediction→│ |
| │ │ │ │ │ |
| ``` |
|
|
| --- |
|
|
| ## 7. Project Structure |
|
|
| ``` |
| bp-monitoring-pipeline/ |
| │ |
| ├── README.md # Project overview |
| ├── ARCHITECTURE.md # ⬅ Dokumen ini |
| ├── docker-compose.yml # Local development stack |
| ├── docker-compose.prod.yml # Production stack |
| ├── Dockerfile # HF Spaces deployment |
| ├── pyproject.toml # Project metadata & dependencies |
| ├── alembic.ini # Database migration config |
| ├── .env.example # Environment variables template |
| │ |
| ├── src/ |
| │ ├── __init__.py |
| │ │ |
| │ ├── domain/ # 🟢 LAYER 1: Domain (Innermost) |
| │ │ ├── __init__.py # |
| │ │ ├── entities/ # Core business objects |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── ppg_signal.py # PPG signal entity |
| │ │ │ └── prediction.py # Blood pressure prediction entity |
| │ │ ├── value_objects/ # Immutable value types |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── device_info.py # IoT device information |
| │ │ │ └── signal_metadata.py # Signal metadata (rate, duration) |
| │ │ ├── interfaces/ # Abstract contracts (ABC) |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── repositories/ # Data persistence contracts |
| │ │ │ │ ├── __init__.py # |
| │ │ │ │ ├── base_repository.py # Generic CRUD interface |
| │ │ │ │ ├── ppg_repository.py # PPG-specific queries |
| │ │ │ │ └── prediction_repository.py # Prediction-specific queries |
| │ │ │ └── services/ # External service contracts |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── message_broker.py # Pub/Sub messaging |
| │ │ │ ├── signal_processor.py # Signal preprocessing |
| │ │ │ └── model_service.py # AI model inference |
| │ │ └── exceptions/ # Domain-specific errors |
| │ │ ├── __init__.py # |
| │ │ └── domain_exceptions.py # InvalidSignalError, etc. |
| │ │ |
| │ ├── application/ # 🔵 LAYER 2: Application |
| │ │ ├── __init__.py # |
| │ │ ├── use_cases/ # Business use case orchestrators |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── ingest_ppg.py # ETL #1: Ingest from mobile |
| │ │ │ ├── process_and_predict.py # ETL #2: Process + AI predict |
| │ │ │ └── get_prediction_history.py # Query historical predictions |
| │ │ └── dto/ # Data Transfer Objects |
| │ │ ├── __init__.py # |
| │ │ ├── ppg_dto.py # Request/Response for PPG |
| │ │ └── prediction_dto.py # Request/Response for predictions |
| │ │ |
| │ ├── infrastructure/ # 🟠 LAYER 3: Infrastructure |
| │ │ ├── __init__.py # |
| │ │ ├── database/ # PostgreSQL implementation |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── connection.py # Engine & session factory |
| │ │ │ ├── models/ # SQLAlchemy ORM models |
| │ │ │ │ ├── __init__.py # |
| │ │ │ │ ├── base.py # Shared base (id, created_at) |
| │ │ │ │ ├── ppg_model.py # raw_ppg table |
| │ │ │ │ └── prediction_model.py # predictions table |
| │ │ │ └── repositories/ # Concrete repository implementations |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── sqlalchemy_base.py # Generic CRUD impl (DRY) |
| │ │ │ ├── ppg_repository.py # PPGRepository impl |
| │ │ │ └── prediction_repository.py # PredictionRepository impl |
| │ │ ├── messaging/ # RabbitMQ implementation |
| │ │ │ ├── __init__.py # |
| │ │ │ └── rabbitmq_broker.py # MessageBroker impl |
| │ │ ├── processing/ # Signal processing implementation |
| │ │ │ ├── __init__.py # |
| │ │ │ └── scipy_signal_processor.py # SignalProcessor impl (SciPy) |
| │ │ └── model/ # AI model implementation |
| │ │ ├── __init__.py # |
| │ │ ├── mock_model_service.py # Mock for testing |
| │ │ └── gan_vgtlnet_service.py # Production GAN + VGTL-Net |
| │ │ |
| │ ├── interface/ # 🔴 LAYER 4: Interface (Outermost) |
| │ │ ├── __init__.py # |
| │ │ ├── api/ # Entry Point 1: REST API (HF Spaces) |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── app.py # FastAPI application factory |
| │ │ │ ├── dependencies.py # Dependency injection wiring |
| │ │ │ └── routes/ # HTTP route handlers |
| │ │ │ ├── __init__.py # |
| │ │ │ ├── ppg_routes.py # POST /api/v1/ppg/ingest |
| │ │ │ └── prediction_routes.py # GET /api/v1/predictions |
| │ │ └── consumer/ # Entry Point 2: MQ Consumer (Colab) |
| │ │ ├── __init__.py # |
| │ │ ├── run_consumer.py # Standalone async consumer |
| │ │ └── colab_notebook.ipynb # Jupyter notebook for Colab |
| │ │ |
| │ └── shared/ # 🟣 Cross-cutting Concerns |
| │ ├── __init__.py # |
| │ ├── config.py # Settings from .env (Pydantic) |
| │ ├── logger.py # Logging factory |
| │ └── constants.py # Named constants (no magic numbers) |
| │ |
| ├── tests/ # Test suite |
| │ ├── __init__.py # |
| │ ├── unit/ # Unit tests (no external deps) |
| │ │ ├── test_entities.py # Domain entity tests |
| │ │ ├── test_use_cases.py # Use case tests (mocked deps) |
| │ │ └── test_signal_processor.py # Signal processing tests |
| │ └── integration/ # Integration tests |
| │ ├── test_api.py # API endpoint tests |
| │ └── test_rabbitmq.py # Message broker tests |
| │ |
| ├── migrations/ # Alembic database migrations |
| │ ├── env.py # |
| │ └── versions/ # |
| │ |
| └── scripts/ # Utility scripts |
| ├── mock_iot_sender.py # Simulate IoT data |
| └── seed_database.py # Seed initial data |
| ``` |
|
|
| --- |
|
|
| ## 8. Deployment Architecture |
|
|
| ### 8.1. Deployment Topology |
|
|
| ``` |
| ┌──────────────────────────────────┐ |
| │ GitHub Repository │ |
| │ bp-monitoring-pipeline │ |
| └───────────┬──────────┬───────────┘ |
| │ │ |
| git push │ │ git clone |
| (subset) │ │ (full repo) |
| │ │ |
| ┌──────────────────▼──┐ ┌────▼──────────────────┐ |
| │ 🤗 Hugging Face │ │ 🧪 Kaggle │ |
| │ Spaces │ │ (GPU: T4) │ |
| │ │ │ │ |
| │ ┌────────────────┐ │ │ ┌──────────────────┐ │ |
| │ │ Docker │ │ │ │ run_consumer.py │ │ |
| │ │ ┌──────────┐ │ │ │ │ │ │ |
| │ │ │ FastAPI │ │ │ │ │ ETL #2 │ │ |
| │ │ │ ETL #1 │ │ │ │ │ + GAN │ │ |
| │ │ │ port 7860│ │ │ │ │ + VGTL-Net │ │ |
| │ │ └──────────┘ │ │ │ └──────────────────┘ │ |
| │ └────────────────┘ │ └────────────────────────┘ |
| └───────┬────┬────────┘ │ │ |
| │ │ │ │ |
| ┌──────────▼┐ ┌▼──────────┐ ┌─────▼──┐ ┌──▼───────┐ |
| │ ☁️Supabase │ │🐰CloudAMQP│ │☁️Supa- │ │🐰Cloud- │ |
| │ PostgreSQL│ │ RabbitMQ │ │ base │ │ AMQP │ |
| │ (Store) │ │ (Publish) │ │ (R/W) │ │(Consume)│ |
| └───────────┘ └────────────┘ └────────┘ └─────────┘ |
| ``` |
|
|
| ### 8.2. Technology Stack |
|
|
| | Layer | Teknologi | Versi | Justifikasi | |
| |-------|-----------|-------|-------------| |
| | **Runtime** | Python | 3.11+ | Async support, type hints, ecosystem ML matang | |
| | **Web Framework** | FastAPI | ≥ 0.110 | Async-native, auto-docs (Swagger), Pydantic integration | |
| | **ORM** | SQLAlchemy | ≥ 2.0 | Async support, mature, database-agnostic | |
| | **Database Driver** | asyncpg | ≥ 0.29 | Fastest Python PostgreSQL async driver | |
| | **Database** | PostgreSQL (Supabase) | 15 | JSONB support, reliability, free tier 500MB | |
| | **Message Broker** | RabbitMQ (CloudAMQP) | 3.x | AMQP protocol, persistent messages, free tier 1M msg/month | |
| | **Message Client** | aio-pika | ≥ 9.4 | Async RabbitMQ client, SSL support | |
| | **Signal Processing** | SciPy | ≥ 1.12 | Industry-standard scientific computing | |
| | **Deep Learning** | PyTorch | ≥ 2.0 | GPU inference, GAN + VGTL-Net compatibility | |
| | **Validation** | Pydantic | ≥ 2.0 | DTO validation, settings management | |
| | **Containerization** | Docker | — | Reproducible deployments | |
| | **Hosting (API)** | Hugging Face Spaces | Free | Docker SDK, 2 vCPU, 16GB RAM | |
| | **Hosting (GPU)** | Kaggle | Free | T4 GPU, 15GB VRAM | |
|
|
| ### 8.3. Entry Points |
|
|
| | # | Entry Point | Platform | Command | Fungsi | |
| |---|-------------|----------|---------|--------| |
| | 1 | FastAPI Server | HF Spaces | `uvicorn src.interface.api.app:create_app` | Terima PPG dari HP, store, publish | |
| | 2 | MQ Consumer | Kaggle | `python -m src.interface.consumer.run_consumer` | Consume, preprocess, predict, store | |
|
|
| > **📌 Kedua entry point berbagi ~85% codebase yang sama.** Yang berbeda hanyalah bootstrapping dan dependency wiring. |
|
|
| --- |
|
|
| ## 9. Configuration Management |
|
|
| Seluruh konfigurasi dikelola melalui **environment variables** menggunakan Pydantic Settings. Tidak ada credentials yang di-hardcode. |
|
|
| ### Environment Variables |
|
|
| | Variable | Deskripsi | Contoh | Wajib | |
| |----------|-----------|--------|:-----:| |
| | `DATABASE_URL` | PostgreSQL connection string (async) | `postgresql+asyncpg://user:pass@host:6543/db` | ✅ | |
| | `RABBITMQ_URL` | RabbitMQ/CloudAMQP connection string | `amqps://user:pass@host/vhost` | ✅ | |
| | `APP_PORT` | Port untuk FastAPI (default: 7860 untuk HF) | `7860` | ❌ | |
| | `APP_HOST` | Host binding (default: 0.0.0.0) | `0.0.0.0` | ❌ | |
| | `DEBUG` | Enable debug mode | `false` | ❌ | |
| | `LOG_LEVEL` | Logging level | `INFO` | ❌ | |
|
|
| ### Secrets Management per Platform |
|
|
| | Platform | Metode | Lokasi | |
| |----------|--------|--------| |
| | HF Spaces | Space Secrets | Settings → Variables and Secrets | |
| | Kaggle | Colab Secrets | 🔑 Sidebar → Secrets | |
| | Local Development | `.env` file | Root project directory | |
|
|
| --- |
|
|
| ## 10. Testing Strategy |
|
|
| ### Testing Pyramid |
|
|
| ``` |
| ╱╲ |
| ╱ ╲ E2E Tests |
| ╱ ╲ (curl → HF Spaces → Supabase → Colab) |
| ╱──────╲ |
| ╱ ╲ Integration Tests |
| ╱ ╲ (API + real DB, Consumer + real Queue) |
| ╱────────────╲ |
| ╱ ╲ Unit Tests |
| ╱ ╲ (Entities, Use Cases, Processors — mocked deps) |
| ╱══════════════════╲ |
| ``` |
|
|
| ### Test Categories |
|
|
| | Kategori | Scope | Dependencies | Tools | |
| |----------|-------|-------------|-------| |
| | **Unit** | Domain entities, value objects | Tidak ada | `pytest` | |
| | **Unit** | Use cases | Mocked repositories & services | `pytest`, `unittest.mock` | |
| | **Unit** | Signal processor | Tidak ada (pure NumPy/SciPy) | `pytest`, `numpy.testing` | |
| | **Integration** | API endpoints | Test database (SQLite/PostgreSQL) | `pytest`, `httpx.AsyncClient` | |
| | **Integration** | Message broker | Test RabbitMQ (Docker) | `pytest`, `aio-pika` | |
| | **E2E** | Full pipeline | All services running | `curl`, `pytest` | |
|
|
| ### Testability by Layer |
|
|
| | Layer | Testable Without External Deps? | Bagaimana | |
| |-------|:-------------------------------:|-----------| |
| | Domain | ✅ Ya | Pure Python — tidak ada dependency | |
| | Application | ✅ Ya | Inject mock repositories & services | |
| | Infrastructure | ⚠️ Perlu test DB/Queue | Gunakan testcontainers atau in-memory substitutes | |
| | Interface | ⚠️ Perlu running app | Gunakan `TestClient` (FastAPI) | |
|
|
| --- |
|
|
| ## 11. Glossary |
|
|
| | Istilah | Definisi | |
| |---------|----------| |
| | **PPG** | Photoplethysmography — teknik optik non-invasif untuk mengukur perubahan volume darah dalam jaringan mikrovaskular | |
| | **ABP** | Arterial Blood Pressure — tekanan darah arteri yang diprediksi oleh model | |
| | **SBP** | Systolic Blood Pressure — tekanan darah saat jantung berkontraksi (angka atas) | |
| | **DBP** | Diastolic Blood Pressure — tekanan darah saat jantung berelaksasi (angka bawah) | |
| | **GAN** | Generative Adversarial Network — arsitektur deep learning untuk mentranslasikan sinyal PPG ke ECG | |
| | **VGTL-Net** | Model deep learning yang menerima PPG + ECG untuk memprediksi ABP | |
| | **ETL** | Extract-Transform-Load — pola pipeline data | |
| | **Entity** | Objek domain dengan identitas unik dan lifecycle | |
| | **Value Object** | Objek domain immutable yang didefinisikan oleh atributnya | |
| | **DTO** | Data Transfer Object — objek untuk transfer data antar layer boundary | |
| | **Repository** | Abstraksi akses data yang menyembunyikan detail persistence | |
| | **Use Case** | Orkestrasi satu unit business logic yang spesifik | |
| | **Message Broker** | Middleware yang mengelola antrian pesan antara producer dan consumer | |
| | **Dependency Injection** | Teknik menyediakan dependency dari luar, bukan membuat di dalam class | |
|
|
| --- |
|
|
| ## 12. References |
|
|
| | # | Referensi | Penulis | |
| |---|-----------|---------| |
| | 1 | *Clean Architecture: A Craftsman's Guide to Software Structure and Design* | Robert C. Martin (2017) | |
| | 2 | *Design Patterns: Elements of Reusable Object-Oriented Software* | Gamma, Helm, Johnson, Vlissides (1994) | |
| | 3 | *Patterns of Enterprise Application Architecture* | Martin Fowler (2002) | |
| | 4 | *Domain-Driven Design: Tackling Complexity in the Heart of Software* | Eric Evans (2003) | |
| | 5 | FastAPI Documentation | https://fastapi.tiangolo.com | |
| | 6 | SQLAlchemy 2.0 Documentation | https://docs.sqlalchemy.org | |
| | 7 | RabbitMQ Tutorials | https://www.rabbitmq.com/tutorials | |
| | 8 | Hugging Face Spaces Docker SDK | https://huggingface.co/docs/hub/spaces-sdks-docker | |
|
|
| --- |
|
|
| <div align="center"> |
|
|
| --- |
|
|
| *Dokumen ini di-maintain oleh Data Engineering Team.* |
| *Untuk pertanyaan, silakan buat issue di repository GitHub.* |
|
|
| **© 2026 BP Monitoring Pipeline — All rights reserved.** |
|
|
| </div> |
|
|
