# BeFree Classification Pipeline This document describes the three-tier classification architecture, keyword matching criteria, auto-learning mechanics, and audit log structures of the BeFree classification pipeline. --- ## Pipeline Overview The classification pipeline matches transaction logs into standardized expense categories. It uses a **sequential fallback model** consisting of three engines. The pipeline evaluates the engines in order and terminates at the first engine that yields a non-None classification result: $$\text{Transaction} \longrightarrow [\text{Rule Engine}] \overset{\text{None}}{\longrightarrow} [\text{ML Engine}] \overset{\text{None}}{\longrightarrow} [\text{LLM Engine (Gemini)}] \longrightarrow \text{Category}$$ --- ## 1. Engine Tiers ### 1.1 The Rule Engine The Rule Engine acts as the first, fast, deterministic tier. It retrieves active categorization rules from PostgreSQL (`classification_rules` and `classification_categories` tables). #### Match Modes: - **Substring Match**: Used for seeded rules (priority $\ge 10$) and manual rules. Matches keywords anywhere inside the description string (e.g. `"woolworths"` matches `"WOOLWORTHS METRO SYDNEY"`). - **Word-Boundary Match**: Automatically enabled for auto-learned rules (priority $< 10$) or short keywords ($\le 4$ characters) to prevent false matches (e.g., keyword `"bar"` matches `"BAR DE GUSTATION"` but does not match `"SUBURBAN"`). - Word boundary patterns are compiled as: `r'(?:^|\b|(?<=\s))' + re.escape(keyword) + r'(?:\b|(?=\s)|$)'` - Enabled automatically by `_needs_word_boundary(keyword)` if the keyword is $\le 4$ characters, is in a predefined generic boundary keyword list (`"bar"`, `"pay"`, `"shop"`, `"online"`, `"visa"`, `"atm"`, `"cc"`, `"net"`), or contains non-alphanumeric characters. #### Payment Rail Features: - **OSKO / NPP Transfers**: Extracts payer names or intent purposes before cleaning description noise. - If description indicates OSKO/NPP, it extracts the reference payload using regex patterns. - Matches the extracted name against `osko_reference` target rules. - If no rule matches, unclassified OSKO debit transfers default to `"Transfers & Internal Movements"` or `"External Transfer"`. - **BPAY Lookup**: Matches numeric biller codes against the `bpay_billers` table for instant categorization (e.g. biller code `7000` maps deterministically to `"Water Bill & Utilities"`). --- ### 1.2 The ML Engine The Machine Learning Engine is an offline classifier utilizing TF-IDF and Logistic Regression. - **Status**: Disabled by default in settings (`use_ml=False`). - **Feature Extraction**: Character n-grams (range: 2 to 4 characters) to handle typos, merchant name variations, and locations. - **Model Storage**: Loaded dynamically from pickle file `data/models/category_classifier.pkl`. - **Threshold Gates**: - `MIN_CONFIDENCE = 0.15`: Predictions below 15% fall through to the LLM. - `MIN_MARGIN = 0.05`: Rejects predictions where the probability difference between the top-1 and top-2 predicted categories is less than 5% (ambiguous predictions). --- ### 1.3 The LLM Engine (Google Gemini) Serves as the high-accuracy semantic fallback for transactions that did not match rules or ML models. - **Model**: `gemini-2.5-flash` - **Execution Flow**: - **Batching**: Transactions are grouped into batches of up to 25 (`_BATCH_SIZE = 25`). - **Concurrency**: Batches are processed concurrently in a `ThreadPoolExecutor` with up to 8 threads (`_MAX_WORKERS = 8`). - **Retries**: Incorporates exponential back-off retries to handle rate limits (`HTTP 429` quota limits) or transient connection timeouts. - **Prompt Structure**: Details standard categories, provides transaction examples, describes extraction formats, and demands a structured JSON array return detailing `tx_id`, `category`, `third_party_name`, `confidence`, and `reasoning`. --- ## 2. Rule Learner & Promotion The **RuleLearner** automatically converts frequent LLM classifications into high-speed database rules to minimize future API costs. ### 2.1 Suggestion Log Criteria After LLM classification, candidates are written to `learned_rule_logs` in a `pending` state if they meet the following: - Classification Source must be `llm`. - LLM confidence score must be $\ge 0.85$ (`MIN_CONFIDENCE_TO_ACTIVATE`). - The keyword must not be in the `GENERIC_STOP_WORDS` list (like `"payment"`, `"transfer"`, `"ref"`, suburb names, etc.) or duplicate active rules. ### 2.2 Promotion Thresholds The `scripts/clean_and_promote_rules.py` script runs periodically (via cron or admin console triggers) to evaluate pending suggestions and promote them to active rules if they meet either threshold: $$\text{Condition 1}: \text{Hit Count} \ge 2 \quad \text{AND} \quad \text{Confidence} \ge 0.90$$ $$\text{Condition 2}: \text{Hit Count} \ge 1 \quad \text{AND} \quad \text{Confidence} \ge 0.95$$ Once promoted, a rule record is created in `classification_rules` and the log status changes to `approved`. Rejected logs are marked `rejected`. --- ## 3. Classification Logging Schema Every classification decision is recorded in the audit log file `data/logs/classification_latest.jsonl`. ### JSONL Schema Fields: ```json { "tx_id": "tx_1_8f93e2", "logged_at": "2026-06-28T21:42:03.123456Z", "date": "2026-05-24", "description": "WOOLWORTHS SUPERMARKETS SYDNEY", "amount": -74.32, "tx_type": "debit", "category": "Food and Groceries", "source": "rule", "confidence": 1.0, "matched_keyword": "woolworths", "matched_rule_name": "rule_food_groceries", "ml_top_scores": null, "llm_prompt_tokens": null, "llm_response_raw": null, "rule_learned": false, "learned_keyword": null, "engines_tried": ["rule"], "engines_failed": [] } ``` --- ## 4. `ClassificationSource` Enum The `ClassificationSource` enum (declared in `bank_classifier/models.py`) documents how the category was determined: - `"rule"`: Determined by a matching rule in the database. - `"ml"`: Determined by the Logistic Regression classifier. - `"llm"`: Determined by the Google Gemini API. - `"manual"`: Assigned via a user manual override in the review dashboard. - `"fallback"`: Set by system fallback rules (e.g. unclassified OSKO transfers). - `"unknown"`: Default state before execution.