Spaces:
Sleeping
Download docs/ARCHITECTURE.md from Coeinterns/bank-statement-api: direct link, hf CLI and curl.
- Browser
- Download file 10 kB
-
https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/ARCHITECTURE.md
- Command line
-
hf download hf://spaces/Coeinterns/bank-statement-api/docs/ARCHITECTURE.md
-
curl -L -o ARCHITECTURE.md https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/ARCHITECTURE.md
BeFree Architecture Guide
This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor.
Monorepo Layout
BeFree is structured as a monorepo containing a Python backend (orchestrated by api_server.py) and a TypeScript/Next.js frontend.
The backend architecture consists of a central FastAPI entry point (api_server.py), a router layer (routers/), a core logic layer (core/), a shared models layer (models/), two local library modules (auth/ and overwatch/), and four internal Python packages installed as editable installs (-e).
new/
βββ api_server.py # Central FastAPI entry point & app configuration
βββ auth/ # Local module: authentication & rules database tables
βββ overwatch/ # Local module: structured event logging (observability)
βββ models/ # Shared Pydantic data models for request/response serialization
βββ core/ # Core business logic, pipeline initialization, statement IO, and middlewares
βββ routers/ # Modular FastAPI APIRouters split by feature domain
βββ shared/ # Editable Package: shared Pydantic data models & constants
βββ bank-extractor/ # Editable Package: PDF structure & layout coordinate extractor (sibling)
βββ bank-classifier/ # Editable Package: multi-engine classification pipeline
βββ bank-hitl/ # Editable Package: Human-in-the-loop web reviews (sibling)
βββ befree-frontend/ # Standalone Next.js 15 SPA Frontend
Component Breakdown
1. api_server.py
- Responsibility: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from
core.middleware), and loading/mounting sub-routers (fromrouters/). - Size/Complexity: A very thin and clean entry point orchestrating other backend components.
2. models/ (Data Model Layer)
- Responsibility: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift.
- Key Modules:
models/statements.pyβ Statement payloadsmodels/transactions.pyβ Transaction row formatsmodels/layout.pyβ Bounding box coordinate schemasmodels/classification.pyβ ML/LLM classification modelsmodels/export_models.pyβ Excel/CSV export/download metadata schemasmodels/admin_models.pyβ Categories and rules CRUD payloads
3. core/ (Core Logic & Infrastructure)
- Responsibility: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares.
- Key Modules:
core/state.pyβ In-memory statement database, directory resolutioncore/pipeline.pyβ Thread-safe classification pipeline instance managementcore/helpers.pyβ Non-IO utilities (sorting, cleaning audit flags)core/statement_io.pyβ Excel output generation and balance swapping persistencecore/extraction.pyβ PDF text-coordinate extraction background task wrapperscore/middleware.pyβ Global exception, audit logs, and request ID middleware handlers
4. routers/ (Modular API Router Layer)
- Responsibility: Serves HTTP endpoints grouped logically by business domain.
- Routers:
routers/auth.pyβ Login and logout authentication endpointsrouters/settings_router.pyβ Application configuration view/modification endpointsrouters/pdf.pyβ Dynamic PDF serving and verification endpointsrouters/history.pyβ Statement archival, reopening, deletion, and duplication endpointsrouters/flagged.pyβ Suspect transaction listing endpointsrouters/dashboard.pyβ Recent statements, activity logs, and metric stats endpointsrouters/upload.pyβ Direct statement file uploadsrouters/layout.pyβ Coordinate extraction, grid column assignment, and user verify overridesrouters/processing.pyβ In-progress status updates, cancellation, and progress savesrouters/classification.pyβ Batch classifying, single classification overrides, and excel reference updatesrouters/review.pyβ Interactive verification sessions and review status updatesrouters/transactions.pyβ Transactions pagination, CRUD, and debit-credit swappingrouters/export.pyβ Excel summaries, CSV/XLSX downloads, and audit reportsrouters/admin.pyβ Overwatch monitoring insights and categories CRUDrouters/admin_rules.pyβ Classification rules configuration and suggestions promotions
5. auth/ (Local Module)
- Responsibility: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding.
- Key Files:
auth/models.pyβ SQLAlchemy ORM models (User,UserSession,AuditLog)auth/rule_models.pyβ ORM models for DB-backed rules (ClassificationRule,ClassificationCategory,ClassificationGroup,BpayBiller,LearnedRuleLog)auth/service.pyβ Password hashing, JWT validation, and user role permission decorators
- Database: Relational database (PostgreSQL in production/NeonDB in dev).
6. shared/ (Editable Package)
- Responsibility: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift.
- Key Files:
shared/models.pyβ Data models (RawTransaction,NormTransaction,StatementAudit,PipelineResult,HITLEdit)shared/constants.pyβ Enums (TxTypefor DEBIT/CREDIT/UNKNOWN,AuditStatusfor CLEAN/SUSPECT/CRITICAL)
- Interface: Exported Python module
shared.
7. bank-extractor/ (Editable Sibling Package)
- Responsibility: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells.
- Interface: Installed via editable pip dependency. Exposes coordinate extraction modules.
8. bank-classifier/ (Editable Package)
- Responsibility: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers.
- Key Files:
bank_classifier/pipeline.pyβ Main entry point orchestrating rule, ML, and LLM classifiersbank_classifier/engines/rule_engine.pyβ Matches transaction descriptions against DB rulesbank_classifier/engines/llm_engine.pyβ Google Gemini API connector with batched API callsbank_classifier/rule_learner.pyβ Analyzes LLM classifications and generates candidate rules
- Interface: Exposed as
ClassificationPipeline.classify_batch(transactions)returning classifiedNormTransactionobjects.
9. bank-hitl/ (Editable Sibling Package)
- Responsibility: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications.
- Interface: Installed as editable pip dependency
bank-hitl[web].
10. overwatch/ (Local Module)
- Responsibility: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events.
- Key Files:
overwatch/store.pyβ Thread-safeLogStorelayeroverwatch/models.pyβ PydanticOverwatchEventand SQLAlchemyOverwatchEventORMoverwatch/event_log.pyβ Non-blocking context managerEventLog
- Database: SQLite database running in WAL (Write-Ahead Logging) mode.
11. befree-frontend/ (Next.js Application)
- Responsibility: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs.
- Structure: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside
befree-frontend/features/.
Key Design Decisions
1. Internal Editable Packages (-e)
To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (bank-extractor, bank-classifier, bank-hitl, and shared). Using -e (editable) installs allows these modules to be developed in place, versioned separately, and isolated.
2. Dual Database Strategy: PostgreSQL + SQLite WAL
- PostgreSQL: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well.
- SQLite (Overwatch): Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance.
3. Classification Pipeline Hierarchy
The classification pipeline executes three tiers in sequence, stopping at the first successful match:
- Rule Engine: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions.
- ML Engine: Sklearn classifier (currently disabled by default).
- LLM Engine (Gemini): Serves as a semantic quality fallback. Used only when deterministic rules fail.
- RuleLearner: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine.
4. Router-Based Backend Layout
By refactoring the monolithic api_server.py into individual, self-contained APIRouter files within routers/, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.