Spaces:
Sleeping
Sleeping
|
Download docs/ARCHITECTURE.md from Coeinterns/bank-statement-api: direct link, hf CLI and curl.
- Browser
- Download file 10 kB
-
https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/ARCHITECTURE.md
- Command line
-
hf download hf://spaces/Coeinterns/bank-statement-api/docs/ARCHITECTURE.md
-
curl -L -o ARCHITECTURE.md https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/ARCHITECTURE.md
10 kB
| # BeFree Architecture Guide | |
| This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor. | |
| --- | |
| ## Monorepo Layout | |
| BeFree is structured as a monorepo containing a Python backend (orchestrated by `api_server.py`) and a TypeScript/Next.js frontend. | |
| The backend architecture consists of a central **FastAPI entry point** (`api_server.py`), a router layer (`routers/`), a core logic layer (`core/`), a shared models layer (`models/`), two local library modules (`auth/` and `overwatch/`), and four internal Python packages installed as editable installs (`-e`). | |
| ``` | |
| new/ | |
| βββ api_server.py # Central FastAPI entry point & app configuration | |
| βββ auth/ # Local module: authentication & rules database tables | |
| βββ overwatch/ # Local module: structured event logging (observability) | |
| βββ models/ # Shared Pydantic data models for request/response serialization | |
| βββ core/ # Core business logic, pipeline initialization, statement IO, and middlewares | |
| βββ routers/ # Modular FastAPI APIRouters split by feature domain | |
| βββ shared/ # Editable Package: shared Pydantic data models & constants | |
| βββ bank-extractor/ # Editable Package: PDF structure & layout coordinate extractor (sibling) | |
| βββ bank-classifier/ # Editable Package: multi-engine classification pipeline | |
| βββ bank-hitl/ # Editable Package: Human-in-the-loop web reviews (sibling) | |
| βββ befree-frontend/ # Standalone Next.js 15 SPA Frontend | |
| ``` | |
| --- | |
| ## Component Breakdown | |
| ### 1. `api_server.py` | |
| - **Responsibility**: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from `core.middleware`), and loading/mounting sub-routers (from `routers/`). | |
| - **Size/Complexity**: A very thin and clean entry point orchestrating other backend components. | |
| ### 2. `models/` (Data Model Layer) | |
| - **Responsibility**: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift. | |
| - **Key Modules**: | |
| - `models/statements.py` β Statement payloads | |
| - `models/transactions.py` β Transaction row formats | |
| - `models/layout.py` β Bounding box coordinate schemas | |
| - `models/classification.py` β ML/LLM classification models | |
| - `models/export_models.py` β Excel/CSV export/download metadata schemas | |
| - `models/admin_models.py` β Categories and rules CRUD payloads | |
| ### 3. `core/` (Core Logic & Infrastructure) | |
| - **Responsibility**: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares. | |
| - **Key Modules**: | |
| - `core/state.py` β In-memory statement database, directory resolution | |
| - `core/pipeline.py` β Thread-safe classification pipeline instance management | |
| - `core/helpers.py` β Non-IO utilities (sorting, cleaning audit flags) | |
| - `core/statement_io.py` β Excel output generation and balance swapping persistence | |
| - `core/extraction.py` β PDF text-coordinate extraction background task wrappers | |
| - `core/middleware.py` β Global exception, audit logs, and request ID middleware handlers | |
| ### 4. `routers/` (Modular API Router Layer) | |
| - **Responsibility**: Serves HTTP endpoints grouped logically by business domain. | |
| - **Routers**: | |
| - `routers/auth.py` β Login and logout authentication endpoints | |
| - `routers/settings_router.py` β Application configuration view/modification endpoints | |
| - `routers/pdf.py` β Dynamic PDF serving and verification endpoints | |
| - `routers/history.py` β Statement archival, reopening, deletion, and duplication endpoints | |
| - `routers/flagged.py` β Suspect transaction listing endpoints | |
| - `routers/dashboard.py` β Recent statements, activity logs, and metric stats endpoints | |
| - `routers/upload.py` β Direct statement file uploads | |
| - `routers/layout.py` β Coordinate extraction, grid column assignment, and user verify overrides | |
| - `routers/processing.py` β In-progress status updates, cancellation, and progress saves | |
| - `routers/classification.py` β Batch classifying, single classification overrides, and excel reference updates | |
| - `routers/review.py` β Interactive verification sessions and review status updates | |
| - `routers/transactions.py` β Transactions pagination, CRUD, and debit-credit swapping | |
| - `routers/export.py` β Excel summaries, CSV/XLSX downloads, and audit reports | |
| - `routers/admin.py` β Overwatch monitoring insights and categories CRUD | |
| - `routers/admin_rules.py` β Classification rules configuration and suggestions promotions | |
| ### 5. `auth/` (Local Module) | |
| - **Responsibility**: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding. | |
| - **Key Files**: | |
| - `auth/models.py` β SQLAlchemy ORM models (`User`, `UserSession`, `AuditLog`) | |
| - `auth/rule_models.py` β ORM models for DB-backed rules (`ClassificationRule`, `ClassificationCategory`, `ClassificationGroup`, `BpayBiller`, `LearnedRuleLog`) | |
| - `auth/service.py` β Password hashing, JWT validation, and user role permission decorators | |
| - **Database**: Relational database (PostgreSQL in production/NeonDB in dev). | |
| ### 6. `shared/` (Editable Package) | |
| - **Responsibility**: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift. | |
| - **Key Files**: | |
| - `shared/models.py` β Data models (`RawTransaction`, `NormTransaction`, `StatementAudit`, `PipelineResult`, `HITLEdit`) | |
| - `shared/constants.py` β Enums (`TxType` for DEBIT/CREDIT/UNKNOWN, `AuditStatus` for CLEAN/SUSPECT/CRITICAL) | |
| - **Interface**: Exported Python module `shared`. | |
| ### 7. `bank-extractor/` (Editable Sibling Package) | |
| - **Responsibility**: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells. | |
| - **Interface**: Installed via editable pip dependency. Exposes coordinate extraction modules. | |
| ### 8. `bank-classifier/` (Editable Package) | |
| - **Responsibility**: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers. | |
| - **Key Files**: | |
| - `bank_classifier/pipeline.py` β Main entry point orchestrating rule, ML, and LLM classifiers | |
| - `bank_classifier/engines/rule_engine.py` β Matches transaction descriptions against DB rules | |
| - `bank_classifier/engines/llm_engine.py` β Google Gemini API connector with batched API calls | |
| - `bank_classifier/rule_learner.py` β Analyzes LLM classifications and generates candidate rules | |
| - **Interface**: Exposed as `ClassificationPipeline.classify_batch(transactions)` returning classified `NormTransaction` objects. | |
| ### 9. `bank-hitl/` (Editable Sibling Package) | |
| - **Responsibility**: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications. | |
| - **Interface**: Installed as editable pip dependency `bank-hitl[web]`. | |
| ### 10. `overwatch/` (Local Module) | |
| - **Responsibility**: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events. | |
| - **Key Files**: | |
| - `overwatch/store.py` β Thread-safe `LogStore` layer | |
| - `overwatch/models.py` β Pydantic `OverwatchEvent` and SQLAlchemy `OverwatchEventORM` | |
| - `overwatch/event_log.py` β Non-blocking context manager `EventLog` | |
| - **Database**: SQLite database running in WAL (Write-Ahead Logging) mode. | |
| ### 11. `befree-frontend/` (Next.js Application) | |
| - **Responsibility**: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs. | |
| - **Structure**: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside `befree-frontend/features/`. | |
| --- | |
| ## Key Design Decisions | |
| ### 1. Internal Editable Packages (`-e`) | |
| To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (`bank-extractor`, `bank-classifier`, `bank-hitl`, and `shared`). Using `-e` (editable) installs allows these modules to be developed in place, versioned separately, and isolated. | |
| ### 2. Dual Database Strategy: PostgreSQL + SQLite WAL | |
| - **PostgreSQL**: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well. | |
| - **SQLite (Overwatch)**: Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance. | |
| ### 3. Classification Pipeline Hierarchy | |
| The classification pipeline executes three tiers in sequence, stopping at the first successful match: | |
| 1. **Rule Engine**: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions. | |
| 2. **ML Engine**: Sklearn classifier (currently disabled by default). | |
| 3. **LLM Engine (Gemini)**: Serves as a semantic quality fallback. Used only when deterministic rules fail. | |
| 4. **RuleLearner**: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine. | |
| ### 4. Router-Based Backend Layout | |
| By refactoring the monolithic `api_server.py` into individual, self-contained `APIRouter` files within `routers/`, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling. | |