# BeFree Architecture Guide This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor. --- ## Monorepo Layout BeFree is structured as a monorepo containing a Python backend (orchestrated by `api_server.py`) and a TypeScript/Next.js frontend. The backend architecture consists of a central **FastAPI entry point** (`api_server.py`), a router layer (`routers/`), a core logic layer (`core/`), a shared models layer (`models/`), two local library modules (`auth/` and `overwatch/`), and four internal Python packages installed as editable installs (`-e`). ``` new/ ├── api_server.py # Central FastAPI entry point & app configuration ├── auth/ # Local module: authentication & rules database tables ├── overwatch/ # Local module: structured event logging (observability) ├── models/ # Shared Pydantic data models for request/response serialization ├── core/ # Core business logic, pipeline initialization, statement IO, and middlewares ├── routers/ # Modular FastAPI APIRouters split by feature domain ├── shared/ # Editable Package: shared Pydantic data models & constants ├── bank-extractor/ # Editable Package: PDF structure & layout coordinate extractor (sibling) ├── bank-classifier/ # Editable Package: multi-engine classification pipeline ├── bank-hitl/ # Editable Package: Human-in-the-loop web reviews (sibling) └── befree-frontend/ # Standalone Next.js 15 SPA Frontend ``` --- ## Component Breakdown ### 1. `api_server.py` - **Responsibility**: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from `core.middleware`), and loading/mounting sub-routers (from `routers/`). - **Size/Complexity**: A very thin and clean entry point orchestrating other backend components. ### 2. `models/` (Data Model Layer) - **Responsibility**: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift. - **Key Modules**: - `models/statements.py` — Statement payloads - `models/transactions.py` — Transaction row formats - `models/layout.py` — Bounding box coordinate schemas - `models/classification.py` — ML/LLM classification models - `models/export_models.py` — Excel/CSV export/download metadata schemas - `models/admin_models.py` — Categories and rules CRUD payloads ### 3. `core/` (Core Logic & Infrastructure) - **Responsibility**: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares. - **Key Modules**: - `core/state.py` — In-memory statement database, directory resolution - `core/pipeline.py` — Thread-safe classification pipeline instance management - `core/helpers.py` — Non-IO utilities (sorting, cleaning audit flags) - `core/statement_io.py` — Excel output generation and balance swapping persistence - `core/extraction.py` — PDF text-coordinate extraction background task wrappers - `core/middleware.py` — Global exception, audit logs, and request ID middleware handlers ### 4. `routers/` (Modular API Router Layer) - **Responsibility**: Serves HTTP endpoints grouped logically by business domain. - **Routers**: - `routers/auth.py` — Login and logout authentication endpoints - `routers/settings_router.py` — Application configuration view/modification endpoints - `routers/pdf.py` — Dynamic PDF serving and verification endpoints - `routers/history.py` — Statement archival, reopening, deletion, and duplication endpoints - `routers/flagged.py` — Suspect transaction listing endpoints - `routers/dashboard.py` — Recent statements, activity logs, and metric stats endpoints - `routers/upload.py` — Direct statement file uploads - `routers/layout.py` — Coordinate extraction, grid column assignment, and user verify overrides - `routers/processing.py` — In-progress status updates, cancellation, and progress saves - `routers/classification.py` — Batch classifying, single classification overrides, and excel reference updates - `routers/review.py` — Interactive verification sessions and review status updates - `routers/transactions.py` — Transactions pagination, CRUD, and debit-credit swapping - `routers/export.py` — Excel summaries, CSV/XLSX downloads, and audit reports - `routers/admin.py` — Overwatch monitoring insights and categories CRUD - `routers/admin_rules.py` — Classification rules configuration and suggestions promotions ### 5. `auth/` (Local Module) - **Responsibility**: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding. - **Key Files**: - `auth/models.py` — SQLAlchemy ORM models (`User`, `UserSession`, `AuditLog`) - `auth/rule_models.py` — ORM models for DB-backed rules (`ClassificationRule`, `ClassificationCategory`, `ClassificationGroup`, `BpayBiller`, `LearnedRuleLog`) - `auth/service.py` — Password hashing, JWT validation, and user role permission decorators - **Database**: Relational database (PostgreSQL in production/NeonDB in dev). ### 6. `shared/` (Editable Package) - **Responsibility**: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift. - **Key Files**: - `shared/models.py` — Data models (`RawTransaction`, `NormTransaction`, `StatementAudit`, `PipelineResult`, `HITLEdit`) - `shared/constants.py` — Enums (`TxType` for DEBIT/CREDIT/UNKNOWN, `AuditStatus` for CLEAN/SUSPECT/CRITICAL) - **Interface**: Exported Python module `shared`. ### 7. `bank-extractor/` (Editable Sibling Package) - **Responsibility**: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells. - **Interface**: Installed via editable pip dependency. Exposes coordinate extraction modules. ### 8. `bank-classifier/` (Editable Package) - **Responsibility**: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers. - **Key Files**: - `bank_classifier/pipeline.py` — Main entry point orchestrating rule, ML, and LLM classifiers - `bank_classifier/engines/rule_engine.py` — Matches transaction descriptions against DB rules - `bank_classifier/engines/llm_engine.py` — Google Gemini API connector with batched API calls - `bank_classifier/rule_learner.py` — Analyzes LLM classifications and generates candidate rules - **Interface**: Exposed as `ClassificationPipeline.classify_batch(transactions)` returning classified `NormTransaction` objects. ### 9. `bank-hitl/` (Editable Sibling Package) - **Responsibility**: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications. - **Interface**: Installed as editable pip dependency `bank-hitl[web]`. ### 10. `overwatch/` (Local Module) - **Responsibility**: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events. - **Key Files**: - `overwatch/store.py` — Thread-safe `LogStore` layer - `overwatch/models.py` — Pydantic `OverwatchEvent` and SQLAlchemy `OverwatchEventORM` - `overwatch/event_log.py` — Non-blocking context manager `EventLog` - **Database**: SQLite database running in WAL (Write-Ahead Logging) mode. ### 11. `befree-frontend/` (Next.js Application) - **Responsibility**: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs. - **Structure**: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside `befree-frontend/features/`. --- ## Key Design Decisions ### 1. Internal Editable Packages (`-e`) To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (`bank-extractor`, `bank-classifier`, `bank-hitl`, and `shared`). Using `-e` (editable) installs allows these modules to be developed in place, versioned separately, and isolated. ### 2. Dual Database Strategy: PostgreSQL + SQLite WAL - **PostgreSQL**: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well. - **SQLite (Overwatch)**: Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance. ### 3. Classification Pipeline Hierarchy The classification pipeline executes three tiers in sequence, stopping at the first successful match: 1. **Rule Engine**: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions. 2. **ML Engine**: Sklearn classifier (currently disabled by default). 3. **LLM Engine (Gemini)**: Serves as a semantic quality fallback. Used only when deterministic rules fail. 4. **RuleLearner**: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine. ### 4. Router-Based Backend Layout By refactoring the monolithic `api_server.py` into individual, self-contained `APIRouter` files within `routers/`, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.