bank-statement-api / docs /CLASSIFICATION_PIPELINE.md
3v324v23's picture
Merge Conflict solved | Build error solved
205f25f
|
Raw History Blame Contribute Delete
6.32 kB

BeFree Classification Pipeline

This document describes the three-tier classification architecture, keyword matching criteria, auto-learning mechanics, and audit log structures of the BeFree classification pipeline.


Pipeline Overview

The classification pipeline matches transaction logs into standardized expense categories. It uses a sequential fallback model consisting of three engines. The pipeline evaluates the engines in order and terminates at the first engine that yields a non-None classification result:

Transaction⟶[Rule Engine][ML Engine][LLM Engine (Gemini)]⟶Category\text{Transaction} \longrightarrow [\text{Rule Engine}] \overset{\text{None}}{\longrightarrow} [\text{ML Engine}] \overset{\text{None}}{\longrightarrow} [\text{LLM Engine (Gemini)}] \longrightarrow \text{Category}


1. Engine Tiers

1.1 The Rule Engine

The Rule Engine acts as the first, fast, deterministic tier. It retrieves active categorization rules from PostgreSQL (classification_rules and classification_categories tables).

Match Modes:

  • Substring Match: Used for seeded rules (priority $\ge 10$) and manual rules. Matches keywords anywhere inside the description string (e.g. "woolworths" matches "WOOLWORTHS METRO SYDNEY").
  • Word-Boundary Match: Automatically enabled for auto-learned rules (priority $< 10$) or short keywords ($\le 4$ characters) to prevent false matches (e.g., keyword "bar" matches "BAR DE GUSTATION" but does not match "SUBURBAN").
    • Word boundary patterns are compiled as: r'(?:^|\b|(?<=\s))' + re.escape(keyword) + r'(?:\b|(?=\s)|$)'
    • Enabled automatically by _needs_word_boundary(keyword) if the keyword is $\le 4$ characters, is in a predefined generic boundary keyword list ("bar", "pay", "shop", "online", "visa", "atm", "cc", "net"), or contains non-alphanumeric characters.

Payment Rail Features:

  • OSKO / NPP Transfers: Extracts payer names or intent purposes before cleaning description noise.
    • If description indicates OSKO/NPP, it extracts the reference payload using regex patterns.
    • Matches the extracted name against osko_reference target rules.
    • If no rule matches, unclassified OSKO debit transfers default to "Transfers & Internal Movements" or "External Transfer".
  • BPAY Lookup: Matches numeric biller codes against the bpay_billers table for instant categorization (e.g. biller code 7000 maps deterministically to "Water Bill & Utilities").

1.2 The ML Engine

The Machine Learning Engine is an offline classifier utilizing TF-IDF and Logistic Regression.

  • Status: Disabled by default in settings (use_ml=False).
  • Feature Extraction: Character n-grams (range: 2 to 4 characters) to handle typos, merchant name variations, and locations.
  • Model Storage: Loaded dynamically from pickle file data/models/category_classifier.pkl.
  • Threshold Gates:
    • MIN_CONFIDENCE = 0.15: Predictions below 15% fall through to the LLM.
    • MIN_MARGIN = 0.05: Rejects predictions where the probability difference between the top-1 and top-2 predicted categories is less than 5% (ambiguous predictions).

1.3 The LLM Engine (Google Gemini)

Serves as the high-accuracy semantic fallback for transactions that did not match rules or ML models.

  • Model: gemini-2.5-flash
  • Execution Flow:
    • Batching: Transactions are grouped into batches of up to 25 (_BATCH_SIZE = 25).
    • Concurrency: Batches are processed concurrently in a ThreadPoolExecutor with up to 8 threads (_MAX_WORKERS = 8).
    • Retries: Incorporates exponential back-off retries to handle rate limits (HTTP 429 quota limits) or transient connection timeouts.
    • Prompt Structure: Details standard categories, provides transaction examples, describes extraction formats, and demands a structured JSON array return detailing tx_id, category, third_party_name, confidence, and reasoning.

2. Rule Learner & Promotion

The RuleLearner automatically converts frequent LLM classifications into high-speed database rules to minimize future API costs.

2.1 Suggestion Log Criteria

After LLM classification, candidates are written to learned_rule_logs in a pending state if they meet the following:

  • Classification Source must be llm.
  • LLM confidence score must be $\ge 0.85$ (MIN_CONFIDENCE_TO_ACTIVATE).
  • The keyword must not be in the GENERIC_STOP_WORDS list (like "payment", "transfer", "ref", suburb names, etc.) or duplicate active rules.

2.2 Promotion Thresholds

The scripts/clean_and_promote_rules.py script runs periodically (via cron or admin console triggers) to evaluate pending suggestions and promote them to active rules if they meet either threshold:

Condition 1:Hit Count≥2ANDConfidence≥0.90\text{Condition 1}: \text{Hit Count} \ge 2 \quad \text{AND} \quad \text{Confidence} \ge 0.90

Condition 2:Hit Count≥1ANDConfidence≥0.95\text{Condition 2}: \text{Hit Count} \ge 1 \quad \text{AND} \quad \text{Confidence} \ge 0.95

Once promoted, a rule record is created in classification_rules and the log status changes to approved. Rejected logs are marked rejected.


3. Classification Logging Schema

Every classification decision is recorded in the audit log file data/logs/classification_latest.jsonl.

JSONL Schema Fields:

{
  "tx_id": "tx_1_8f93e2",
  "logged_at": "2026-06-28T21:42:03.123456Z",
  "date": "2026-05-24",
  "description": "WOOLWORTHS SUPERMARKETS SYDNEY",
  "amount": -74.32,
  "tx_type": "debit",
  "category": "Food and Groceries",
  "source": "rule",
  "confidence": 1.0,
  "matched_keyword": "woolworths",
  "matched_rule_name": "rule_food_groceries",
  "ml_top_scores": null,
  "llm_prompt_tokens": null,
  "llm_response_raw": null,
  "rule_learned": false,
  "learned_keyword": null,
  "engines_tried": ["rule"],
  "engines_failed": []
}

4. ClassificationSource Enum

The ClassificationSource enum (declared in bank_classifier/models.py) documents how the category was determined:

  • "rule": Determined by a matching rule in the database.
  • "ml": Determined by the Logistic Regression classifier.
  • "llm": Determined by the Google Gemini API.
  • "manual": Assigned via a user manual override in the review dashboard.
  • "fallback": Set by system fallback rules (e.g. unclassified OSKO transfers).
  • "unknown": Default state before execution.