Spaces:
Sleeping
Sleeping
|
Download docs/DATA_FLOW.md from Coeinterns/bank-statement-api: direct link, hf CLI and curl.
- Browser
- Download file 7.11 kB
-
https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/DATA_FLOW.md
- Command line
-
hf download hf://spaces/Coeinterns/bank-statement-api/docs/DATA_FLOW.md
-
curl -L -o DATA_FLOW.md https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/DATA_FLOW.md
7.11 kB
BeFree Data Flow Lifecycle
This document describes the complete lifecycle of a bank statement as it passes through the BeFree Bank Statement Processor, from initial PDF upload to finalized Excel export.
Data Lifecycle Diagram
[ Upload PDF ]
β
βΌ (POST /api/v1/statements/upload)
[ Save PDF to Disk ] βββΊ Overwatch Event (stage="upload")
β
βΌ (bank-extractor)
[ Text & Coordinates Extraction ] βββΊ Overwatch Events (stage="layout_detection", "coordinate_extraction")
β
βΌ (core.helpers / models.transactions)
[ Normalization ] βββΊ RawTransaction to NormTransaction βββΊ Overwatch Event (stage="normalisation")
β
βΌ (core.helpers / core.statement_io)
[ Math Audit ] βββΊ StatementAudit calculated βββΊ Overwatch Event (stage="audit")
β
βΌ (Cache JSON)
[ Cache NormTransaction List ] βββΊ Stored as data.json
β
βΌ (bank-classifier / ClassificationPipeline)
[ Three-Tier Classification ] βββΊ RuleEngine βββΊ MLEngine (Disabled) βββΊ LLMEngine (Gemini)
β β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βΌ
[ Rule Promotion ] βββΊ RuleLearner logs suggestions βββΊ Overwatch Event (stage="rule_learned")
β
βΌ (Logging)
[ Write classification_latest.jsonl ]
β
βΌ (POST /api/v1/statements/.../transactions)
[ Human-in-the-Loop Review ] βββΊ User corrections saved to data/corrections/
β
βΌ (GET /api/v1/statements/.../download)
[ Excel / CSV Export ] βββΊ Compiled via openpyxl βββΊ Saved to data/output/{id}.xlsx
Detailed Data Lifecycle Stages
1. Ingestion (Upload)
- Endpoint:
POST /api/v1/statements/upload(handled byrouters/upload.py) - Actions:
- The client uploads one or more PDF files.
- A unique
statement_id(UUIDv4) is generated. - The PDF file is written to the filesystem at
data/uploads/{statement_id}/statement.pdf. - Overwatch Event: Emits log with
stage="upload".
2. Layout Detection & Text Extraction
- Module:
bank-extractor(background tasks managed incore/extraction.py) - Actions:
- The extractor parses the PDF document page-by-page.
- Detects headers and column boundaries (bounding box coordinates
x_min,x_max,y). - Outputs a list of
RawTransactionmodels containing string representation of rows. - Data Model:
shared.models.RawTransaction - Overwatch Events: Emits logs with
stage="layout_detection"andstage="coordinate_extraction".
3. Normalization
- Module:
core/helpers.py(specifically_to_transaction_row_global/_sort_dicts_chronologically) - Actions:
- The string values of
RawTransactionare cleaned. - Debit/Credit columns are coerced into
Decimalstructures. - Date strings (e.g.
24 Nov,12/05/2026) are parsed into standarddatetime.dateobjects. - A unique
tx_id(e.g.,tx_page_index_hash) is generated for each row. - Data Model:
models.transactions.TransactionRowModel - Overwatch Event: Emits log with
stage="normalisation".
- The string values of
4. Statement Auditing
- Module:
core/helpers.py/core/statement_io.py - Actions:
- Performs mathematical verification of statement pages: $$\text{Computed Closing Balance} = \text{Opening Balance} + \sum \text{Credits} - \sum \text{Debits}$$
- Flags balance mismatches or sequence gaps between pages.
- Data Model:
shared.models.StatementAuditandshared.models.PageAudit - Cache: The verified
NormTransactionarray and theStatementAuditresult are saved as a JSON cache file atdata/uploads/{statement_id}/data.json. - Overwatch Event: Emits log with
stage="audit".
5. Classification
- Module:
bank-classifier(viaClassificationPipeline.classify_batch()) - Actions:
- Executes a three-stage classification process:
- RuleEngine: Scans the transaction description against keywords loaded from the database table
classification_rules. If a keyword matches, it assigns the category. - MLEngine: An offline Machine Learning classifier (disabled by default,
use_ml=False). - LLMEngine: Passes unclassified transactions in batches to the Google Gemini API with a system prompt detailing available categories.
- RuleEngine: Scans the transaction description against keywords loaded from the database table
- RuleLearner: Promotes frequently matched LLM categories to candidate database rules.
- Logging: The classification results are logged to
data/logs/classification_latest.jsonl. - Overwatch Events: Emits
stage="classification_started",stage="transaction_classified",stage="classification_completed", andstage="rule_learned".
- Executes a three-stage classification process:
6. Human-in-the-Loop (HITL) Review
- Module:
bank-hitl/routers/review.py/routers/transactions.py - Actions:
- Transactions marked as
flagged(due to low extraction confidence or balance issues) are displayed in the review interface. - Users can correct dates, swap debit/credit columns, modify descriptions, or assign manual category overrides.
- Manual edits are written to
data/corrections/{statement_id}_corrections.jsonand applied to the active transaction log cache.
- Transactions marked as
7. Export
- Endpoint:
GET /api/v1/statements/{id}/download?format=xlsx - Actions:
- Loads the corrected
NormTransactionlog from cache. - Uses
openpyxlto build a styled spreadsheet. - Generates a summary dashboard tab containing monthly breakdowns and expense category ratios.
- Writes the final workbook to
data/output/{statement_id}.xlsxand streams it to the user.
- Loads the corrected
The App Settings Configuration File
The system behavior is controlled by data/settings.json. It is loaded at server startup and can be modified via PATCH /api/v1/settings.
{
"ocrEnabled": true,
"confidenceThreshold": 0.8,
"autoExportEnabled": false,
"auditRetentionDays": 90,
"defaultExportFormat": "xlsx",
"extractLast3Months": true
}
Settings Impact on Data Flow:
ocrEnabled: If true, triggers optical character recognition fallback on scanned PDFs insidebank-extractor.confidenceThreshold: Transactions extracted with a confidence score below this threshold are marked asflagged(status set torevieworflagged), forcing them into the Human-in-the-loop review queue.auditRetentionDays: Governs the database event prune window insideoverwatch/store.py(deletes events older than the specified retention period).extractLast3Months: If set totrue, theGET /api/v1/statements/{id}/transactionsendpoint filters and flags transactions falling outside the latest 3 calendar months of the statement's date range. Only transactions within the latest 3-month window are selected for automatic classification, optimizing Gemini API usage costs.