bank-statement-api / docs /ARCHITECTURE.md
3v324v23's picture
Feat: Added New column in xlsx
fbb917f
|
Raw History Blame Contribute Delete
10 kB

BeFree Architecture Guide

This document describes the high-level architecture, module boundaries, and design decisions of the BeFree Bank Statement Processor.


Monorepo Layout

BeFree is structured as a monorepo containing a Python backend (orchestrated by api_server.py) and a TypeScript/Next.js frontend.

The backend architecture consists of a central FastAPI entry point (api_server.py), a router layer (routers/), a core logic layer (core/), a shared models layer (models/), two local library modules (auth/ and overwatch/), and four internal Python packages installed as editable installs (-e).

new/
β”œβ”€β”€ api_server.py              # Central FastAPI entry point & app configuration
β”œβ”€β”€ auth/                      # Local module: authentication & rules database tables
β”œβ”€β”€ overwatch/                 # Local module: structured event logging (observability)
β”œβ”€β”€ models/                    # Shared Pydantic data models for request/response serialization
β”œβ”€β”€ core/                      # Core business logic, pipeline initialization, statement IO, and middlewares
β”œβ”€β”€ routers/                   # Modular FastAPI APIRouters split by feature domain
β”œβ”€β”€ shared/                    # Editable Package: shared Pydantic data models & constants
β”œβ”€β”€ bank-extractor/            # Editable Package: PDF structure & layout coordinate extractor (sibling)
β”œβ”€β”€ bank-classifier/           # Editable Package: multi-engine classification pipeline
β”œβ”€β”€ bank-hitl/                 # Editable Package: Human-in-the-loop web reviews (sibling)
└── befree-frontend/           # Standalone Next.js 15 SPA Frontend

Component Breakdown

1. api_server.py

  • Responsibility: App configuration, CORS setup, FastAPI app instantiation, mounting HTTP middlewares (from core.middleware), and loading/mounting sub-routers (from routers/).
  • Size/Complexity: A very thin and clean entry point orchestrating other backend components.

2. models/ (Data Model Layer)

  • Responsibility: Defines pure Pydantic request/response validation schemas used across HTTP routers to prevent drift.
  • Key Modules:
    • models/statements.py β€” Statement payloads
    • models/transactions.py β€” Transaction row formats
    • models/layout.py β€” Bounding box coordinate schemas
    • models/classification.py β€” ML/LLM classification models
    • models/export_models.py β€” Excel/CSV export/download metadata schemas
    • models/admin_models.py β€” Categories and rules CRUD payloads

3. core/ (Core Logic & Infrastructure)

  • Responsibility: Coordinates in-memory state management, pipeline initialization, spreadsheet serialization, and FastAPI middlewares.
  • Key Modules:
    • core/state.py β€” In-memory statement database, directory resolution
    • core/pipeline.py β€” Thread-safe classification pipeline instance management
    • core/helpers.py β€” Non-IO utilities (sorting, cleaning audit flags)
    • core/statement_io.py β€” Excel output generation and balance swapping persistence
    • core/extraction.py β€” PDF text-coordinate extraction background task wrappers
    • core/middleware.py β€” Global exception, audit logs, and request ID middleware handlers

4. routers/ (Modular API Router Layer)

  • Responsibility: Serves HTTP endpoints grouped logically by business domain.
  • Routers:
    • routers/auth.py β€” Login and logout authentication endpoints
    • routers/settings_router.py β€” Application configuration view/modification endpoints
    • routers/pdf.py β€” Dynamic PDF serving and verification endpoints
    • routers/history.py β€” Statement archival, reopening, deletion, and duplication endpoints
    • routers/flagged.py β€” Suspect transaction listing endpoints
    • routers/dashboard.py β€” Recent statements, activity logs, and metric stats endpoints
    • routers/upload.py β€” Direct statement file uploads
    • routers/layout.py β€” Coordinate extraction, grid column assignment, and user verify overrides
    • routers/processing.py β€” In-progress status updates, cancellation, and progress saves
    • routers/classification.py β€” Batch classifying, single classification overrides, and excel reference updates
    • routers/review.py β€” Interactive verification sessions and review status updates
    • routers/transactions.py β€” Transactions pagination, CRUD, and debit-credit swapping
    • routers/export.py β€” Excel summaries, CSV/XLSX downloads, and audit reports
    • routers/admin.py β€” Overwatch monitoring insights and categories CRUD
    • routers/admin_rules.py β€” Classification rules configuration and suggestions promotions

5. auth/ (Local Module)

  • Responsibility: Manages security, JWT issuance/revocation, user models, DB rules storage, and seeding.
  • Key Files:
    • auth/models.py β€” SQLAlchemy ORM models (User, UserSession, AuditLog)
    • auth/rule_models.py β€” ORM models for DB-backed rules (ClassificationRule, ClassificationCategory, ClassificationGroup, BpayBiller, LearnedRuleLog)
    • auth/service.py β€” Password hashing, JWT validation, and user role permission decorators
  • Database: Relational database (PostgreSQL in production/NeonDB in dev).

6. shared/ (Editable Package)

  • Responsibility: Defines the common Pydantic v2 schemas and enums used across all backend modules and packages to prevent model definition drift.
  • Key Files:
    • shared/models.py β€” Data models (RawTransaction, NormTransaction, StatementAudit, PipelineResult, HITLEdit)
    • shared/constants.py β€” Enums (TxType for DEBIT/CREDIT/UNKNOWN, AuditStatus for CLEAN/SUSPECT/CRITICAL)
  • Interface: Exported Python module shared.

7. bank-extractor/ (Editable Sibling Package)

  • Responsibility: Parses PDFs, analyzes textual structures and bounding boxes, detects headers, maps column layouts, and extracts raw text cells.
  • Interface: Installed via editable pip dependency. Exposes coordinate extraction modules.

8. bank-classifier/ (Editable Package)

  • Responsibility: Classifies normalized transactions into categorizable expense buckets using three distinct fallback tiers.
  • Key Files:
    • bank_classifier/pipeline.py β€” Main entry point orchestrating rule, ML, and LLM classifiers
    • bank_classifier/engines/rule_engine.py β€” Matches transaction descriptions against DB rules
    • bank_classifier/engines/llm_engine.py β€” Google Gemini API connector with batched API calls
    • bank_classifier/rule_learner.py β€” Analyzes LLM classifications and generates candidate rules
  • Interface: Exposed as ClassificationPipeline.classify_batch(transactions) returning classified NormTransaction objects.

9. bank-hitl/ (Editable Sibling Package)

  • Responsibility: Provides Human-In-The-Loop review logic, tracking manual corrections, and recording audit trails when users override automated classifications.
  • Interface: Installed as editable pip dependency bank-hitl[web].

10. overwatch/ (Local Module)

  • Responsibility: Structured, non-blocking observability logging. It records timestamps, log levels, messages, and arbitrary metadata dicts for all extraction and classification events.
  • Key Files:
    • overwatch/store.py β€” Thread-safe LogStore layer
    • overwatch/models.py β€” Pydantic OverwatchEvent and SQLAlchemy OverwatchEventORM
    • overwatch/event_log.py β€” Non-blocking context manager EventLog
  • Database: SQLite database running in WAL (Write-Ahead Logging) mode.

11. befree-frontend/ (Next.js Application)

  • Responsibility: User interface. Displays dashboard metrics, upload forms, coordinate layout inspectors, transaction tables, and audit logs.
  • Structure: Next.js 15 TypeScript project using page routes, with individual subfeatures modularized inside befree-frontend/features/.

Key Design Decisions

1. Internal Editable Packages (-e)

To prevent the codebase from growing into a single tangled monolith, the core logic was split into independent modules (bank-extractor, bank-classifier, bank-hitl, and shared). Using -e (editable) installs allows these modules to be developed in place, versioned separately, and isolated.

2. Dual Database Strategy: PostgreSQL + SQLite WAL

  • PostgreSQL: Used for Auth sessions, Audit logging, and Rule/Category storage. PostgreSQL offers high relational integrity, robust ACID guarantees, and handles concurrent client writes well.
  • SQLite (Overwatch): Used for the structured observability log store. Overwatch is write-heavy and append-only. Writing hundreds of events per upload to an external PostgreSQL DB introduces network latency and locks database pools. Writing to a local SQLite database in WAL (Write-Ahead Logging) mode allows fast, thread-safe, non-blocking disk writes without impacting main API performance.

3. Classification Pipeline Hierarchy

The classification pipeline executes three tiers in sequence, stopping at the first successful match:

  1. Rule Engine: Fast, deterministic, keyword-based database lookup. Zero network latency and zero cost. Matches the vast majority of recurring transactions.
  2. ML Engine: Sklearn classifier (currently disabled by default).
  3. LLM Engine (Gemini): Serves as a semantic quality fallback. Used only when deterministic rules fail.
  4. RuleLearner: Promotes LLM classifications back to permanent keyword rules in the database, closing the loop to ensure future matching is handled by the high-speed Rule Engine.

4. Router-Based Backend Layout

By refactoring the monolithic api_server.py into individual, self-contained APIRouter files within routers/, we have established clean, modular, and maintainable boundaries for backend functionality. This makes the code easier to navigate for developers, cleaner to test, and ready for future scaling.