bank-statement-api / docs /CLASSIFICATION_PIPELINE.md
3v324v23's picture
Merge Conflict solved | Build error solved
205f25f
|
Raw History Blame Contribute Delete
6.32 kB
# BeFree Classification Pipeline
This document describes the three-tier classification architecture, keyword matching criteria, auto-learning mechanics, and audit log structures of the BeFree classification pipeline.
---
## Pipeline Overview
The classification pipeline matches transaction logs into standardized expense categories. It uses a **sequential fallback model** consisting of three engines. The pipeline evaluates the engines in order and terminates at the first engine that yields a non-None classification result:
$$\text{Transaction} \longrightarrow [\text{Rule Engine}] \overset{\text{None}}{\longrightarrow} [\text{ML Engine}] \overset{\text{None}}{\longrightarrow} [\text{LLM Engine (Gemini)}] \longrightarrow \text{Category}$$
---
## 1. Engine Tiers
### 1.1 The Rule Engine
The Rule Engine acts as the first, fast, deterministic tier. It retrieves active categorization rules from PostgreSQL (`classification_rules` and `classification_categories` tables).
#### Match Modes:
- **Substring Match**: Used for seeded rules (priority $\ge 10$) and manual rules. Matches keywords anywhere inside the description string (e.g. `"woolworths"` matches `"WOOLWORTHS METRO SYDNEY"`).
- **Word-Boundary Match**: Automatically enabled for auto-learned rules (priority $< 10$) or short keywords ($\le 4$ characters) to prevent false matches (e.g., keyword `"bar"` matches `"BAR DE GUSTATION"` but does not match `"SUBURBAN"`).
- Word boundary patterns are compiled as: `r'(?:^|\b|(?<=\s))' + re.escape(keyword) + r'(?:\b|(?=\s)|$)'`
- Enabled automatically by `_needs_word_boundary(keyword)` if the keyword is $\le 4$ characters, is in a predefined generic boundary keyword list (`"bar"`, `"pay"`, `"shop"`, `"online"`, `"visa"`, `"atm"`, `"cc"`, `"net"`), or contains non-alphanumeric characters.
#### Payment Rail Features:
- **OSKO / NPP Transfers**: Extracts payer names or intent purposes before cleaning description noise.
- If description indicates OSKO/NPP, it extracts the reference payload using regex patterns.
- Matches the extracted name against `osko_reference` target rules.
- If no rule matches, unclassified OSKO debit transfers default to `"Transfers & Internal Movements"` or `"External Transfer"`.
- **BPAY Lookup**: Matches numeric biller codes against the `bpay_billers` table for instant categorization (e.g. biller code `7000` maps deterministically to `"Water Bill & Utilities"`).
---
### 1.2 The ML Engine
The Machine Learning Engine is an offline classifier utilizing TF-IDF and Logistic Regression.
- **Status**: Disabled by default in settings (`use_ml=False`).
- **Feature Extraction**: Character n-grams (range: 2 to 4 characters) to handle typos, merchant name variations, and locations.
- **Model Storage**: Loaded dynamically from pickle file `data/models/category_classifier.pkl`.
- **Threshold Gates**:
- `MIN_CONFIDENCE = 0.15`: Predictions below 15% fall through to the LLM.
- `MIN_MARGIN = 0.05`: Rejects predictions where the probability difference between the top-1 and top-2 predicted categories is less than 5% (ambiguous predictions).
---
### 1.3 The LLM Engine (Google Gemini)
Serves as the high-accuracy semantic fallback for transactions that did not match rules or ML models.
- **Model**: `gemini-2.5-flash`
- **Execution Flow**:
- **Batching**: Transactions are grouped into batches of up to 25 (`_BATCH_SIZE = 25`).
- **Concurrency**: Batches are processed concurrently in a `ThreadPoolExecutor` with up to 8 threads (`_MAX_WORKERS = 8`).
- **Retries**: Incorporates exponential back-off retries to handle rate limits (`HTTP 429` quota limits) or transient connection timeouts.
- **Prompt Structure**: Details standard categories, provides transaction examples, describes extraction formats, and demands a structured JSON array return detailing `tx_id`, `category`, `third_party_name`, `confidence`, and `reasoning`.
---
## 2. Rule Learner & Promotion
The **RuleLearner** automatically converts frequent LLM classifications into high-speed database rules to minimize future API costs.
### 2.1 Suggestion Log Criteria
After LLM classification, candidates are written to `learned_rule_logs` in a `pending` state if they meet the following:
- Classification Source must be `llm`.
- LLM confidence score must be $\ge 0.85$ (`MIN_CONFIDENCE_TO_ACTIVATE`).
- The keyword must not be in the `GENERIC_STOP_WORDS` list (like `"payment"`, `"transfer"`, `"ref"`, suburb names, etc.) or duplicate active rules.
### 2.2 Promotion Thresholds
The `scripts/clean_and_promote_rules.py` script runs periodically (via cron or admin console triggers) to evaluate pending suggestions and promote them to active rules if they meet either threshold:
$$\text{Condition 1}: \text{Hit Count} \ge 2 \quad \text{AND} \quad \text{Confidence} \ge 0.90$$
$$\text{Condition 2}: \text{Hit Count} \ge 1 \quad \text{AND} \quad \text{Confidence} \ge 0.95$$
Once promoted, a rule record is created in `classification_rules` and the log status changes to `approved`. Rejected logs are marked `rejected`.
---
## 3. Classification Logging Schema
Every classification decision is recorded in the audit log file `data/logs/classification_latest.jsonl`.
### JSONL Schema Fields:
```json
{
"tx_id": "tx_1_8f93e2",
"logged_at": "2026-06-28T21:42:03.123456Z",
"date": "2026-05-24",
"description": "WOOLWORTHS SUPERMARKETS SYDNEY",
"amount": -74.32,
"tx_type": "debit",
"category": "Food and Groceries",
"source": "rule",
"confidence": 1.0,
"matched_keyword": "woolworths",
"matched_rule_name": "rule_food_groceries",
"ml_top_scores": null,
"llm_prompt_tokens": null,
"llm_response_raw": null,
"rule_learned": false,
"learned_keyword": null,
"engines_tried": ["rule"],
"engines_failed": []
}
```
---
## 4. `ClassificationSource` Enum
The `ClassificationSource` enum (declared in `bank_classifier/models.py`) documents how the category was determined:
- `"rule"`: Determined by a matching rule in the database.
- `"ml"`: Determined by the Logistic Regression classifier.
- `"llm"`: Determined by the Google Gemini API.
- `"manual"`: Assigned via a user manual override in the review dashboard.
- `"fallback"`: Set by system fallback rules (e.g. unclassified OSKO transfers).
- `"unknown"`: Default state before execution.