Spaces:
Sleeping
Sleeping
File size: 10,011 Bytes
fbb917f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 | # BeFree Architecture Guide
This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor.
---
## Monorepo Layout
BeFree is structured as a monorepo containing a Python backend (orchestrated by `api_server.py`) and a TypeScript/Next.js frontend.
The backend architecture consists of a central **FastAPI entry point** (`api_server.py`), a router layer (`routers/`), a core logic layer (`core/`), a shared models layer (`models/`), two local library modules (`auth/` and `overwatch/`), and four internal Python packages installed as editable installs (`-e`).
```
new/
βββ api_server.py # Central FastAPI entry point & app configuration
βββ auth/ # Local module: authentication & rules database tables
βββ overwatch/ # Local module: structured event logging (observability)
βββ models/ # Shared Pydantic data models for request/response serialization
βββ core/ # Core business logic, pipeline initialization, statement IO, and middlewares
βββ routers/ # Modular FastAPI APIRouters split by feature domain
βββ shared/ # Editable Package: shared Pydantic data models & constants
βββ bank-extractor/ # Editable Package: PDF structure & layout coordinate extractor (sibling)
βββ bank-classifier/ # Editable Package: multi-engine classification pipeline
βββ bank-hitl/ # Editable Package: Human-in-the-loop web reviews (sibling)
βββ befree-frontend/ # Standalone Next.js 15 SPA Frontend
```
---
## Component Breakdown
### 1. `api_server.py`
- **Responsibility**: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from `core.middleware`), and loading/mounting sub-routers (from `routers/`).
- **Size/Complexity**: A very thin and clean entry point orchestrating other backend components.
### 2. `models/` (Data Model Layer)
- **Responsibility**: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift.
- **Key Modules**:
- `models/statements.py` β Statement payloads
- `models/transactions.py` β Transaction row formats
- `models/layout.py` β Bounding box coordinate schemas
- `models/classification.py` β ML/LLM classification models
- `models/export_models.py` β Excel/CSV export/download metadata schemas
- `models/admin_models.py` β Categories and rules CRUD payloads
### 3. `core/` (Core Logic & Infrastructure)
- **Responsibility**: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares.
- **Key Modules**:
- `core/state.py` β In-memory statement database, directory resolution
- `core/pipeline.py` β Thread-safe classification pipeline instance management
- `core/helpers.py` β Non-IO utilities (sorting, cleaning audit flags)
- `core/statement_io.py` β Excel output generation and balance swapping persistence
- `core/extraction.py` β PDF text-coordinate extraction background task wrappers
- `core/middleware.py` β Global exception, audit logs, and request ID middleware handlers
### 4. `routers/` (Modular API Router Layer)
- **Responsibility**: Serves HTTP endpoints grouped logically by business domain.
- **Routers**:
- `routers/auth.py` β Login and logout authentication endpoints
- `routers/settings_router.py` β Application configuration view/modification endpoints
- `routers/pdf.py` β Dynamic PDF serving and verification endpoints
- `routers/history.py` β Statement archival, reopening, deletion, and duplication endpoints
- `routers/flagged.py` β Suspect transaction listing endpoints
- `routers/dashboard.py` β Recent statements, activity logs, and metric stats endpoints
- `routers/upload.py` β Direct statement file uploads
- `routers/layout.py` β Coordinate extraction, grid column assignment, and user verify overrides
- `routers/processing.py` β In-progress status updates, cancellation, and progress saves
- `routers/classification.py` β Batch classifying, single classification overrides, and excel reference updates
- `routers/review.py` β Interactive verification sessions and review status updates
- `routers/transactions.py` β Transactions pagination, CRUD, and debit-credit swapping
- `routers/export.py` β Excel summaries, CSV/XLSX downloads, and audit reports
- `routers/admin.py` β Overwatch monitoring insights and categories CRUD
- `routers/admin_rules.py` β Classification rules configuration and suggestions promotions
### 5. `auth/` (Local Module)
- **Responsibility**: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding.
- **Key Files**:
- `auth/models.py` β SQLAlchemy ORM models (`User`, `UserSession`, `AuditLog`)
- `auth/rule_models.py` β ORM models for DB-backed rules (`ClassificationRule`, `ClassificationCategory`, `ClassificationGroup`, `BpayBiller`, `LearnedRuleLog`)
- `auth/service.py` β Password hashing, JWT validation, and user role permission decorators
- **Database**: Relational database (PostgreSQL in production/NeonDB in dev).
### 6. `shared/` (Editable Package)
- **Responsibility**: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift.
- **Key Files**:
- `shared/models.py` β Data models (`RawTransaction`, `NormTransaction`, `StatementAudit`, `PipelineResult`, `HITLEdit`)
- `shared/constants.py` β Enums (`TxType` for DEBIT/CREDIT/UNKNOWN, `AuditStatus` for CLEAN/SUSPECT/CRITICAL)
- **Interface**: Exported Python module `shared`.
### 7. `bank-extractor/` (Editable Sibling Package)
- **Responsibility**: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells.
- **Interface**: Installed via editable pip dependency. Exposes coordinate extraction modules.
### 8. `bank-classifier/` (Editable Package)
- **Responsibility**: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers.
- **Key Files**:
- `bank_classifier/pipeline.py` β Main entry point orchestrating rule, ML, and LLM classifiers
- `bank_classifier/engines/rule_engine.py` β Matches transaction descriptions against DB rules
- `bank_classifier/engines/llm_engine.py` β Google Gemini API connector with batched API calls
- `bank_classifier/rule_learner.py` β Analyzes LLM classifications and generates candidate rules
- **Interface**: Exposed as `ClassificationPipeline.classify_batch(transactions)` returning classified `NormTransaction` objects.
### 9. `bank-hitl/` (Editable Sibling Package)
- **Responsibility**: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications.
- **Interface**: Installed as editable pip dependency `bank-hitl[web]`.
### 10. `overwatch/` (Local Module)
- **Responsibility**: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events.
- **Key Files**:
- `overwatch/store.py` β Thread-safe `LogStore` layer
- `overwatch/models.py` β Pydantic `OverwatchEvent` and SQLAlchemy `OverwatchEventORM`
- `overwatch/event_log.py` β Non-blocking context manager `EventLog`
- **Database**: SQLite database running in WAL (Write-Ahead Logging) mode.
### 11. `befree-frontend/` (Next.js Application)
- **Responsibility**: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs.
- **Structure**: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside `befree-frontend/features/`.
---
## Key Design Decisions
### 1. Internal Editable Packages (`-e`)
To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (`bank-extractor`, `bank-classifier`, `bank-hitl`, and `shared`). Using `-e` (editable) installs allows these modules to be developed in place, versioned separately, and isolated.
### 2. Dual Database Strategy: PostgreSQL + SQLite WAL
- **PostgreSQL**: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well.
- **SQLite (Overwatch)**: Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance.
### 3. Classification Pipeline Hierarchy
The classification pipeline executes three tiers in sequence, stopping at the first successful match:
1. **Rule Engine**: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions.
2. **ML Engine**: Sklearn classifier (currently disabled by default).
3. **LLM Engine (Gemini)**: Serves as a semantic quality fallback. Used only when deterministic rules fail.
4. **RuleLearner**: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine.
### 4. Router-Based Backend Layout
By refactoring the monolithic `api_server.py` into individual, self-contained `APIRouter` files within `routers/`, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.
|