bank-statement-api / docs /ARCHITECTURE.md
3v324v23's picture
Feat: Added New column in xlsx
fbb917f
|
Raw History Blame Contribute Delete
10 kB
# BeFree Architecture Guide
This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor.
---
## Monorepo Layout
BeFree is structured as a monorepo containing a Python backend (orchestrated by `api_server.py`) and a TypeScript/Next.js frontend.
The backend architecture consists of a central **FastAPI entry point** (`api_server.py`), a router layer (`routers/`), a core logic layer (`core/`), a shared models layer (`models/`), two local library modules (`auth/` and `overwatch/`), and four internal Python packages installed as editable installs (`-e`).
```
new/
β”œβ”€β”€ api_server.py # Central FastAPI entry point & app configuration
β”œβ”€β”€ auth/ # Local module: authentication & rules database tables
β”œβ”€β”€ overwatch/ # Local module: structured event logging (observability)
β”œβ”€β”€ models/ # Shared Pydantic data models for request/response serialization
β”œβ”€β”€ core/ # Core business logic, pipeline initialization, statement IO, and middlewares
β”œβ”€β”€ routers/ # Modular FastAPI APIRouters split by feature domain
β”œβ”€β”€ shared/ # Editable Package: shared Pydantic data models & constants
β”œβ”€β”€ bank-extractor/ # Editable Package: PDF structure & layout coordinate extractor (sibling)
β”œβ”€β”€ bank-classifier/ # Editable Package: multi-engine classification pipeline
β”œβ”€β”€ bank-hitl/ # Editable Package: Human-in-the-loop web reviews (sibling)
└── befree-frontend/ # Standalone Next.js 15 SPA Frontend
```
---
## Component Breakdown
### 1. `api_server.py`
- **Responsibility**: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from `core.middleware`), and loading/mounting sub-routers (from `routers/`).
- **Size/Complexity**: A very thin and clean entry point orchestrating other backend components.
### 2. `models/` (Data Model Layer)
- **Responsibility**: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift.
- **Key Modules**:
- `models/statements.py` β€” Statement payloads
- `models/transactions.py` β€” Transaction row formats
- `models/layout.py` β€” Bounding box coordinate schemas
- `models/classification.py` β€” ML/LLM classification models
- `models/export_models.py` β€” Excel/CSV export/download metadata schemas
- `models/admin_models.py` β€” Categories and rules CRUD payloads
### 3. `core/` (Core Logic & Infrastructure)
- **Responsibility**: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares.
- **Key Modules**:
- `core/state.py` β€” In-memory statement database, directory resolution
- `core/pipeline.py` β€” Thread-safe classification pipeline instance management
- `core/helpers.py` β€” Non-IO utilities (sorting, cleaning audit flags)
- `core/statement_io.py` β€” Excel output generation and balance swapping persistence
- `core/extraction.py` β€” PDF text-coordinate extraction background task wrappers
- `core/middleware.py` β€” Global exception, audit logs, and request ID middleware handlers
### 4. `routers/` (Modular API Router Layer)
- **Responsibility**: Serves HTTP endpoints grouped logically by business domain.
- **Routers**:
- `routers/auth.py` β€” Login and logout authentication endpoints
- `routers/settings_router.py` β€” Application configuration view/modification endpoints
- `routers/pdf.py` β€” Dynamic PDF serving and verification endpoints
- `routers/history.py` β€” Statement archival, reopening, deletion, and duplication endpoints
- `routers/flagged.py` β€” Suspect transaction listing endpoints
- `routers/dashboard.py` β€” Recent statements, activity logs, and metric stats endpoints
- `routers/upload.py` β€” Direct statement file uploads
- `routers/layout.py` β€” Coordinate extraction, grid column assignment, and user verify overrides
- `routers/processing.py` β€” In-progress status updates, cancellation, and progress saves
- `routers/classification.py` β€” Batch classifying, single classification overrides, and excel reference updates
- `routers/review.py` β€” Interactive verification sessions and review status updates
- `routers/transactions.py` β€” Transactions pagination, CRUD, and debit-credit swapping
- `routers/export.py` β€” Excel summaries, CSV/XLSX downloads, and audit reports
- `routers/admin.py` β€” Overwatch monitoring insights and categories CRUD
- `routers/admin_rules.py` β€” Classification rules configuration and suggestions promotions
### 5. `auth/` (Local Module)
- **Responsibility**: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding.
- **Key Files**:
- `auth/models.py` β€” SQLAlchemy ORM models (`User`, `UserSession`, `AuditLog`)
- `auth/rule_models.py` β€” ORM models for DB-backed rules (`ClassificationRule`, `ClassificationCategory`, `ClassificationGroup`, `BpayBiller`, `LearnedRuleLog`)
- `auth/service.py` β€” Password hashing, JWT validation, and user role permission decorators
- **Database**: Relational database (PostgreSQL in production/NeonDB in dev).
### 6. `shared/` (Editable Package)
- **Responsibility**: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift.
- **Key Files**:
- `shared/models.py` β€” Data models (`RawTransaction`, `NormTransaction`, `StatementAudit`, `PipelineResult`, `HITLEdit`)
- `shared/constants.py` β€” Enums (`TxType` for DEBIT/CREDIT/UNKNOWN, `AuditStatus` for CLEAN/SUSPECT/CRITICAL)
- **Interface**: Exported Python module `shared`.
### 7. `bank-extractor/` (Editable Sibling Package)
- **Responsibility**: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells.
- **Interface**: Installed via editable pip dependency. Exposes coordinate extraction modules.
### 8. `bank-classifier/` (Editable Package)
- **Responsibility**: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers.
- **Key Files**:
- `bank_classifier/pipeline.py` β€” Main entry point orchestrating rule, ML, and LLM classifiers
- `bank_classifier/engines/rule_engine.py` β€” Matches transaction descriptions against DB rules
- `bank_classifier/engines/llm_engine.py` β€” Google Gemini API connector with batched API calls
- `bank_classifier/rule_learner.py` β€” Analyzes LLM classifications and generates candidate rules
- **Interface**: Exposed as `ClassificationPipeline.classify_batch(transactions)` returning classified `NormTransaction` objects.
### 9. `bank-hitl/` (Editable Sibling Package)
- **Responsibility**: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications.
- **Interface**: Installed as editable pip dependency `bank-hitl[web]`.
### 10. `overwatch/` (Local Module)
- **Responsibility**: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events.
- **Key Files**:
- `overwatch/store.py` β€” Thread-safe `LogStore` layer
- `overwatch/models.py` β€” Pydantic `OverwatchEvent` and SQLAlchemy `OverwatchEventORM`
- `overwatch/event_log.py` β€” Non-blocking context manager `EventLog`
- **Database**: SQLite database running in WAL (Write-Ahead Logging) mode.
### 11. `befree-frontend/` (Next.js Application)
- **Responsibility**: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs.
- **Structure**: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside `befree-frontend/features/`.
---
## Key Design Decisions
### 1. Internal Editable Packages (`-e`)
To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (`bank-extractor`, `bank-classifier`, `bank-hitl`, and `shared`). Using `-e` (editable) installs allows these modules to be developed in place, versioned separately, and isolated.
### 2. Dual Database Strategy: PostgreSQL + SQLite WAL
- **PostgreSQL**: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well.
- **SQLite (Overwatch)**: Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance.
### 3. Classification Pipeline Hierarchy
The classification pipeline executes three tiers in sequence, stopping at the first successful match:
1. **Rule Engine**: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions.
2. **ML Engine**: Sklearn classifier (currently disabled by default).
3. **LLM Engine (Gemini)**: Serves as a semantic quality fallback. Used only when deterministic rules fail.
4. **RuleLearner**: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine.
### 4. Router-Based Backend Layout
By refactoring the monolithic `api_server.py` into individual, self-contained `APIRouter` files within `routers/`, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.