File size: 10,011 Bytes
fbb917f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
# BeFree Architecture Guide

This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor.

---

## Monorepo Layout

BeFree is structured as a monorepo containing a Python backend (orchestrated by `api_server.py`) and a TypeScript/Next.js frontend. 

The backend architecture consists of a central **FastAPI entry point** (`api_server.py`), a router layer (`routers/`), a core logic layer (`core/`), a shared models layer (`models/`), two local library modules (`auth/` and `overwatch/`), and four internal Python packages installed as editable installs (`-e`).

```
new/
β”œβ”€β”€ api_server.py              # Central FastAPI entry point & app configuration
β”œβ”€β”€ auth/                      # Local module: authentication & rules database tables
β”œβ”€β”€ overwatch/                 # Local module: structured event logging (observability)
β”œβ”€β”€ models/                    # Shared Pydantic data models for request/response serialization
β”œβ”€β”€ core/                      # Core business logic, pipeline initialization, statement IO, and middlewares
β”œβ”€β”€ routers/                   # Modular FastAPI APIRouters split by feature domain
β”œβ”€β”€ shared/                    # Editable Package: shared Pydantic data models & constants
β”œβ”€β”€ bank-extractor/            # Editable Package: PDF structure & layout coordinate extractor (sibling)
β”œβ”€β”€ bank-classifier/           # Editable Package: multi-engine classification pipeline
β”œβ”€β”€ bank-hitl/                 # Editable Package: Human-in-the-loop web reviews (sibling)
└── befree-frontend/           # Standalone Next.js 15 SPA Frontend
```

---

## Component Breakdown

### 1. `api_server.py`
- **Responsibility**: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from `core.middleware`), and loading/mounting sub-routers (from `routers/`).
- **Size/Complexity**: A very thin and clean entry point orchestrating other backend components.

### 2. `models/` (Data Model Layer)
- **Responsibility**: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift.
- **Key Modules**:
  - `models/statements.py` β€” Statement payloads
  - `models/transactions.py` β€” Transaction row formats
  - `models/layout.py` β€” Bounding box coordinate schemas
  - `models/classification.py` β€” ML/LLM classification models
  - `models/export_models.py` β€” Excel/CSV export/download metadata schemas
  - `models/admin_models.py` β€” Categories and rules CRUD payloads

### 3. `core/` (Core Logic & Infrastructure)
- **Responsibility**: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares.
- **Key Modules**:
  - `core/state.py` β€” In-memory statement database, directory resolution
  - `core/pipeline.py` β€” Thread-safe classification pipeline instance management
  - `core/helpers.py` β€” Non-IO utilities (sorting, cleaning audit flags)
  - `core/statement_io.py` β€” Excel output generation and balance swapping persistence
  - `core/extraction.py` β€” PDF text-coordinate extraction background task wrappers
  - `core/middleware.py` β€” Global exception, audit logs, and request ID middleware handlers

### 4. `routers/` (Modular API Router Layer)
- **Responsibility**: Serves HTTP endpoints grouped logically by business domain.
- **Routers**:
  - `routers/auth.py` β€” Login and logout authentication endpoints
  - `routers/settings_router.py` β€” Application configuration view/modification endpoints
  - `routers/pdf.py` β€” Dynamic PDF serving and verification endpoints
  - `routers/history.py` β€” Statement archival, reopening, deletion, and duplication endpoints
  - `routers/flagged.py` β€” Suspect transaction listing endpoints
  - `routers/dashboard.py` β€” Recent statements, activity logs, and metric stats endpoints
  - `routers/upload.py` β€” Direct statement file uploads
  - `routers/layout.py` β€” Coordinate extraction, grid column assignment, and user verify overrides
  - `routers/processing.py` β€” In-progress status updates, cancellation, and progress saves
  - `routers/classification.py` β€” Batch classifying, single classification overrides, and excel reference updates
  - `routers/review.py` β€” Interactive verification sessions and review status updates
  - `routers/transactions.py` β€” Transactions pagination, CRUD, and debit-credit swapping
  - `routers/export.py` β€” Excel summaries, CSV/XLSX downloads, and audit reports
  - `routers/admin.py` β€” Overwatch monitoring insights and categories CRUD
  - `routers/admin_rules.py` β€” Classification rules configuration and suggestions promotions

### 5. `auth/` (Local Module)
- **Responsibility**: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding.
- **Key Files**:
  - `auth/models.py` β€” SQLAlchemy ORM models (`User`, `UserSession`, `AuditLog`)
  - `auth/rule_models.py` β€” ORM models for DB-backed rules (`ClassificationRule`, `ClassificationCategory`, `ClassificationGroup`, `BpayBiller`, `LearnedRuleLog`)
  - `auth/service.py` β€” Password hashing, JWT validation, and user role permission decorators
- **Database**: Relational database (PostgreSQL in production/NeonDB in dev).

### 6. `shared/` (Editable Package)
- **Responsibility**: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift.
- **Key Files**:
  - `shared/models.py` β€” Data models (`RawTransaction`, `NormTransaction`, `StatementAudit`, `PipelineResult`, `HITLEdit`)
  - `shared/constants.py` β€” Enums (`TxType` for DEBIT/CREDIT/UNKNOWN, `AuditStatus` for CLEAN/SUSPECT/CRITICAL)
- **Interface**: Exported Python module `shared`.

### 7. `bank-extractor/` (Editable Sibling Package)
- **Responsibility**: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells.
- **Interface**: Installed via editable pip dependency. Exposes coordinate extraction modules.

### 8. `bank-classifier/` (Editable Package)
- **Responsibility**: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers.
- **Key Files**:
  - `bank_classifier/pipeline.py` β€” Main entry point orchestrating rule, ML, and LLM classifiers
  - `bank_classifier/engines/rule_engine.py` β€” Matches transaction descriptions against DB rules
  - `bank_classifier/engines/llm_engine.py` β€” Google Gemini API connector with batched API calls
  - `bank_classifier/rule_learner.py` β€” Analyzes LLM classifications and generates candidate rules
- **Interface**: Exposed as `ClassificationPipeline.classify_batch(transactions)` returning classified `NormTransaction` objects.

### 9. `bank-hitl/` (Editable Sibling Package)
- **Responsibility**: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications.
- **Interface**: Installed as editable pip dependency `bank-hitl[web]`.

### 10. `overwatch/` (Local Module)
- **Responsibility**: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events.
- **Key Files**:
  - `overwatch/store.py` β€” Thread-safe `LogStore` layer
  - `overwatch/models.py` β€” Pydantic `OverwatchEvent` and SQLAlchemy `OverwatchEventORM`
  - `overwatch/event_log.py` β€” Non-blocking context manager `EventLog`
- **Database**: SQLite database running in WAL (Write-Ahead Logging) mode.

### 11. `befree-frontend/` (Next.js Application)
- **Responsibility**: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs.
- **Structure**: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside `befree-frontend/features/`.

---

## Key Design Decisions

### 1. Internal Editable Packages (`-e`)
To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (`bank-extractor`, `bank-classifier`, `bank-hitl`, and `shared`). Using `-e` (editable) installs allows these modules to be developed in place, versioned separately, and isolated. 

### 2. Dual Database Strategy: PostgreSQL + SQLite WAL
- **PostgreSQL**: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well.
- **SQLite (Overwatch)**: Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance.

### 3. Classification Pipeline Hierarchy
The classification pipeline executes three tiers in sequence, stopping at the first successful match:
1. **Rule Engine**: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions.
2. **ML Engine**: Sklearn classifier (currently disabled by default).
3. **LLM Engine (Gemini)**: Serves as a semantic quality fallback. Used only when deterministic rules fail.
4. **RuleLearner**: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine.

### 4. Router-Based Backend Layout
By refactoring the monolithic `api_server.py` into individual, self-contained `APIRouter` files within `routers/`, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.