File size: 6,316 Bytes
205f25f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
# BeFree Classification Pipeline

This document describes the three-tier classification architecture, keyword matching criteria, auto-learning mechanics, and audit log structures of the BeFree classification pipeline.

---

## Pipeline Overview

The classification pipeline matches transaction logs into standardized expense categories. It uses a **sequential fallback model** consisting of three engines. The pipeline evaluates the engines in order and terminates at the first engine that yields a non-None classification result:

$$\text{Transaction} \longrightarrow [\text{Rule Engine}] \overset{\text{None}}{\longrightarrow} [\text{ML Engine}] \overset{\text{None}}{\longrightarrow} [\text{LLM Engine (Gemini)}] \longrightarrow \text{Category}$$

---

## 1. Engine Tiers

### 1.1 The Rule Engine
The Rule Engine acts as the first, fast, deterministic tier. It retrieves active categorization rules from PostgreSQL (`classification_rules` and `classification_categories` tables).

#### Match Modes:
- **Substring Match**: Used for seeded rules (priority $\ge 10$) and manual rules. Matches keywords anywhere inside the description string (e.g. `"woolworths"` matches `"WOOLWORTHS METRO SYDNEY"`).
- **Word-Boundary Match**: Automatically enabled for auto-learned rules (priority $< 10$) or short keywords ($\le 4$ characters) to prevent false matches (e.g., keyword `"bar"` matches `"BAR DE GUSTATION"` but does not match `"SUBURBAN"`).
  - Word boundary patterns are compiled as: `r'(?:^|\b|(?<=\s))' + re.escape(keyword) + r'(?:\b|(?=\s)|$)'`
  - Enabled automatically by `_needs_word_boundary(keyword)` if the keyword is $\le 4$ characters, is in a predefined generic boundary keyword list (`"bar"`, `"pay"`, `"shop"`, `"online"`, `"visa"`, `"atm"`, `"cc"`, `"net"`), or contains non-alphanumeric characters.

#### Payment Rail Features:
- **OSKO / NPP Transfers**: Extracts payer names or intent purposes before cleaning description noise. 
  - If description indicates OSKO/NPP, it extracts the reference payload using regex patterns.
  - Matches the extracted name against `osko_reference` target rules.
  - If no rule matches, unclassified OSKO debit transfers default to `"Transfers & Internal Movements"` or `"External Transfer"`.
- **BPAY Lookup**: Matches numeric biller codes against the `bpay_billers` table for instant categorization (e.g. biller code `7000` maps deterministically to `"Water Bill & Utilities"`).

---

### 1.2 The ML Engine
The Machine Learning Engine is an offline classifier utilizing TF-IDF and Logistic Regression.
- **Status**: Disabled by default in settings (`use_ml=False`).
- **Feature Extraction**: Character n-grams (range: 2 to 4 characters) to handle typos, merchant name variations, and locations.
- **Model Storage**: Loaded dynamically from pickle file `data/models/category_classifier.pkl`.
- **Threshold Gates**:
  - `MIN_CONFIDENCE = 0.15`: Predictions below 15% fall through to the LLM.
  - `MIN_MARGIN = 0.05`: Rejects predictions where the probability difference between the top-1 and top-2 predicted categories is less than 5% (ambiguous predictions).

---

### 1.3 The LLM Engine (Google Gemini)
Serves as the high-accuracy semantic fallback for transactions that did not match rules or ML models.
- **Model**: `gemini-2.5-flash`
- **Execution Flow**:
  - **Batching**: Transactions are grouped into batches of up to 25 (`_BATCH_SIZE = 25`).
  - **Concurrency**: Batches are processed concurrently in a `ThreadPoolExecutor` with up to 8 threads (`_MAX_WORKERS = 8`).
  - **Retries**: Incorporates exponential back-off retries to handle rate limits (`HTTP 429` quota limits) or transient connection timeouts.
  - **Prompt Structure**: Details standard categories, provides transaction examples, describes extraction formats, and demands a structured JSON array return detailing `tx_id`, `category`, `third_party_name`, `confidence`, and `reasoning`.

---

## 2. Rule Learner & Promotion

The **RuleLearner** automatically converts frequent LLM classifications into high-speed database rules to minimize future API costs.

### 2.1 Suggestion Log Criteria
After LLM classification, candidates are written to `learned_rule_logs` in a `pending` state if they meet the following:
- Classification Source must be `llm`.
- LLM confidence score must be $\ge 0.85$ (`MIN_CONFIDENCE_TO_ACTIVATE`).
- The keyword must not be in the `GENERIC_STOP_WORDS` list (like `"payment"`, `"transfer"`, `"ref"`, suburb names, etc.) or duplicate active rules.

### 2.2 Promotion Thresholds
The `scripts/clean_and_promote_rules.py` script runs periodically (via cron or admin console triggers) to evaluate pending suggestions and promote them to active rules if they meet either threshold:

$$\text{Condition 1}: \text{Hit Count} \ge 2 \quad \text{AND} \quad \text{Confidence} \ge 0.90$$

$$\text{Condition 2}: \text{Hit Count} \ge 1 \quad \text{AND} \quad \text{Confidence} \ge 0.95$$

Once promoted, a rule record is created in `classification_rules` and the log status changes to `approved`. Rejected logs are marked `rejected`.

---

## 3. Classification Logging Schema

Every classification decision is recorded in the audit log file `data/logs/classification_latest.jsonl`.

### JSONL Schema Fields:
```json
{
  "tx_id": "tx_1_8f93e2",
  "logged_at": "2026-06-28T21:42:03.123456Z",
  "date": "2026-05-24",
  "description": "WOOLWORTHS SUPERMARKETS SYDNEY",
  "amount": -74.32,
  "tx_type": "debit",
  "category": "Food and Groceries",
  "source": "rule",
  "confidence": 1.0,
  "matched_keyword": "woolworths",
  "matched_rule_name": "rule_food_groceries",
  "ml_top_scores": null,
  "llm_prompt_tokens": null,
  "llm_response_raw": null,
  "rule_learned": false,
  "learned_keyword": null,
  "engines_tried": ["rule"],
  "engines_failed": []
}
```

---

## 4. `ClassificationSource` Enum

The `ClassificationSource` enum (declared in `bank_classifier/models.py`) documents how the category was determined:
- `"rule"`: Determined by a matching rule in the database.
- `"ml"`: Determined by the Logistic Regression classifier.
- `"llm"`: Determined by the Google Gemini API.
- `"manual"`: Assigned via a user manual override in the review dashboard.
- `"fallback"`: Set by system fallback rules (e.g. unclassified OSKO transfers).
- `"unknown"`: Default state before execution.