Spaces:
Sleeping
Sleeping
|
Download docs/CLASSIFICATION_PIPELINE.md from Coeinterns/bank-statement-api: direct link, hf CLI and curl.
- Browser
- Download file 6.32 kB
-
https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/CLASSIFICATION_PIPELINE.md
- Command line
-
hf download hf://spaces/Coeinterns/bank-statement-api/docs/CLASSIFICATION_PIPELINE.md
-
curl -L -o CLASSIFICATION_PIPELINE.md https://huggingface.co/spaces/Coeinterns/bank-statement-api/resolve/main/docs/CLASSIFICATION_PIPELINE.md
6.32 kB
| # BeFree Classification Pipeline | |
| This document describes the three-tier classification architecture, keyword matching criteria, auto-learning mechanics, and audit log structures of the BeFree classification pipeline. | |
| --- | |
| ## Pipeline Overview | |
| The classification pipeline matches transaction logs into standardized expense categories. It uses a **sequential fallback model** consisting of three engines. The pipeline evaluates the engines in order and terminates at the first engine that yields a non-None classification result: | |
| $$\text{Transaction} \longrightarrow [\text{Rule Engine}] \overset{\text{None}}{\longrightarrow} [\text{ML Engine}] \overset{\text{None}}{\longrightarrow} [\text{LLM Engine (Gemini)}] \longrightarrow \text{Category}$$ | |
| --- | |
| ## 1. Engine Tiers | |
| ### 1.1 The Rule Engine | |
| The Rule Engine acts as the first, fast, deterministic tier. It retrieves active categorization rules from PostgreSQL (`classification_rules` and `classification_categories` tables). | |
| #### Match Modes: | |
| - **Substring Match**: Used for seeded rules (priority $\ge 10$) and manual rules. Matches keywords anywhere inside the description string (e.g. `"woolworths"` matches `"WOOLWORTHS METRO SYDNEY"`). | |
| - **Word-Boundary Match**: Automatically enabled for auto-learned rules (priority $< 10$) or short keywords ($\le 4$ characters) to prevent false matches (e.g., keyword `"bar"` matches `"BAR DE GUSTATION"` but does not match `"SUBURBAN"`). | |
| - Word boundary patterns are compiled as: `r'(?:^|\b|(?<=\s))' + re.escape(keyword) + r'(?:\b|(?=\s)|$)'` | |
| - Enabled automatically by `_needs_word_boundary(keyword)` if the keyword is $\le 4$ characters, is in a predefined generic boundary keyword list (`"bar"`, `"pay"`, `"shop"`, `"online"`, `"visa"`, `"atm"`, `"cc"`, `"net"`), or contains non-alphanumeric characters. | |
| #### Payment Rail Features: | |
| - **OSKO / NPP Transfers**: Extracts payer names or intent purposes before cleaning description noise. | |
| - If description indicates OSKO/NPP, it extracts the reference payload using regex patterns. | |
| - Matches the extracted name against `osko_reference` target rules. | |
| - If no rule matches, unclassified OSKO debit transfers default to `"Transfers & Internal Movements"` or `"External Transfer"`. | |
| - **BPAY Lookup**: Matches numeric biller codes against the `bpay_billers` table for instant categorization (e.g. biller code `7000` maps deterministically to `"Water Bill & Utilities"`). | |
| --- | |
| ### 1.2 The ML Engine | |
| The Machine Learning Engine is an offline classifier utilizing TF-IDF and Logistic Regression. | |
| - **Status**: Disabled by default in settings (`use_ml=False`). | |
| - **Feature Extraction**: Character n-grams (range: 2 to 4 characters) to handle typos, merchant name variations, and locations. | |
| - **Model Storage**: Loaded dynamically from pickle file `data/models/category_classifier.pkl`. | |
| - **Threshold Gates**: | |
| - `MIN_CONFIDENCE = 0.15`: Predictions below 15% fall through to the LLM. | |
| - `MIN_MARGIN = 0.05`: Rejects predictions where the probability difference between the top-1 and top-2 predicted categories is less than 5% (ambiguous predictions). | |
| --- | |
| ### 1.3 The LLM Engine (Google Gemini) | |
| Serves as the high-accuracy semantic fallback for transactions that did not match rules or ML models. | |
| - **Model**: `gemini-2.5-flash` | |
| - **Execution Flow**: | |
| - **Batching**: Transactions are grouped into batches of up to 25 (`_BATCH_SIZE = 25`). | |
| - **Concurrency**: Batches are processed concurrently in a `ThreadPoolExecutor` with up to 8 threads (`_MAX_WORKERS = 8`). | |
| - **Retries**: Incorporates exponential back-off retries to handle rate limits (`HTTP 429` quota limits) or transient connection timeouts. | |
| - **Prompt Structure**: Details standard categories, provides transaction examples, describes extraction formats, and demands a structured JSON array return detailing `tx_id`, `category`, `third_party_name`, `confidence`, and `reasoning`. | |
| --- | |
| ## 2. Rule Learner & Promotion | |
| The **RuleLearner** automatically converts frequent LLM classifications into high-speed database rules to minimize future API costs. | |
| ### 2.1 Suggestion Log Criteria | |
| After LLM classification, candidates are written to `learned_rule_logs` in a `pending` state if they meet the following: | |
| - Classification Source must be `llm`. | |
| - LLM confidence score must be $\ge 0.85$ (`MIN_CONFIDENCE_TO_ACTIVATE`). | |
| - The keyword must not be in the `GENERIC_STOP_WORDS` list (like `"payment"`, `"transfer"`, `"ref"`, suburb names, etc.) or duplicate active rules. | |
| ### 2.2 Promotion Thresholds | |
| The `scripts/clean_and_promote_rules.py` script runs periodically (via cron or admin console triggers) to evaluate pending suggestions and promote them to active rules if they meet either threshold: | |
| $$\text{Condition 1}: \text{Hit Count} \ge 2 \quad \text{AND} \quad \text{Confidence} \ge 0.90$$ | |
| $$\text{Condition 2}: \text{Hit Count} \ge 1 \quad \text{AND} \quad \text{Confidence} \ge 0.95$$ | |
| Once promoted, a rule record is created in `classification_rules` and the log status changes to `approved`. Rejected logs are marked `rejected`. | |
| --- | |
| ## 3. Classification Logging Schema | |
| Every classification decision is recorded in the audit log file `data/logs/classification_latest.jsonl`. | |
| ### JSONL Schema Fields: | |
| ```json | |
| { | |
| "tx_id": "tx_1_8f93e2", | |
| "logged_at": "2026-06-28T21:42:03.123456Z", | |
| "date": "2026-05-24", | |
| "description": "WOOLWORTHS SUPERMARKETS SYDNEY", | |
| "amount": -74.32, | |
| "tx_type": "debit", | |
| "category": "Food and Groceries", | |
| "source": "rule", | |
| "confidence": 1.0, | |
| "matched_keyword": "woolworths", | |
| "matched_rule_name": "rule_food_groceries", | |
| "ml_top_scores": null, | |
| "llm_prompt_tokens": null, | |
| "llm_response_raw": null, | |
| "rule_learned": false, | |
| "learned_keyword": null, | |
| "engines_tried": ["rule"], | |
| "engines_failed": [] | |
| } | |
| ``` | |
| --- | |
| ## 4. `ClassificationSource` Enum | |
| The `ClassificationSource` enum (declared in `bank_classifier/models.py`) documents how the category was determined: | |
| - `"rule"`: Determined by a matching rule in the database. | |
| - `"ml"`: Determined by the Logistic Regression classifier. | |
| - `"llm"`: Determined by the Google Gemini API. | |
| - `"manual"`: Assigned via a user manual override in the review dashboard. | |
| - `"fallback"`: Set by system fallback rules (e.g. unclassified OSKO transfers). | |
| - `"unknown"`: Default state before execution. | |