Upload README.md with huggingface_hub

README.md

@@ -12,35 +12,57 @@ tags:
 datasets:
   - DoDataThings/us-bank-transaction-categories
 pipeline_tag: text-classification
-model-index:
-  - name: distilbert-us-transaction-classifier
-    results:
-      - task:
-          type: text-classification
-          name: Transaction Classification
-        metrics:
-          - type: accuracy
-            value: 0.9975
-            name: Validation Accuracy
 ---
 
 # DistilBERT US Bank Transaction Classifier
 
-Fine-tuned DistilBERT model that classifies US bank transaction descriptions into 16 spending categories. Built for real bank statement formats — the messy, abbreviated, ALL-CAPS descriptions you actually see on Chase, Apple Card, PayPal, and Capital One statements.
+A fine-tuned DistilBERT model for classifying US bank transaction descriptions into 16 spending categories. Designed as a **fallback layer** in a multi-tier classification pipeline — not a standalone classifier.
 
-## Why This Exists
+## What This Is (and Isn't)
 
-Off-the-shelf transaction classifiers are trained on clean data like `"Starbucks coffee"`. Real bank statements look like this:
+This model was built to handle the long tail of transaction descriptions that merchant rules and bank-provided categories don't cover. It works best as one component in a system:
 
+1. **Merchant rules** — pattern matching catches known merchants (highest accuracy)
+2. **Bank-provided categories** — map the bank's own classifications (when available)
+3. **This model** — classifies everything else (the fallback)
+4. **User overrides** — manual corrections for edge cases
+
+On its own, this model will make mistakes. Bank transaction descriptions are messy, abbreviated, and sometimes genuinely ambiguous — the same store can be "Groceries" or "Shopping" depending on what was purchased, and no classifier can know that from the description alone.
+
+## Training
+
+Fine-tuned on 16,000 synthetic transaction descriptions generated to match real US bank statement formats. The synthetic data covers six format templates (Chase ACH, Apple Card addresses, PayPal prefixes, etc.) with randomized merchants, store numbers, and addresses.
+
+```
+Model:       DistilBERT-base-uncased + LoRA (r=32, alpha=64)
+Dataset:     16,000 synthetic samples, 1,000 per category
+Trainable:   1.8M / 68.7M parameters (2.6%)
+Training:    20 epochs, ~5 minutes on consumer GPU
 ```
-PAYPAL           INST XFER  GOOGLE YOUTUBE  WEB ID: PAYPALSI77
-AMAZON MKTPL*RJ7GA07V1
-TST*TAISHOKEN RAMEN - MI
-WELLS FARGO IFI  DDA TO DDA FP0WP73DKR      WEB ID: INTFITRVOS
-AUTOMATIC PAYMENT - THANK
+
+### Loss Curve
+
+```
+Epoch  Train Loss   Val Loss  Train Acc   Val Acc
+─────────────────────────────────────────────────
+    1      2.670      2.246     17.6%     47.2%
+    2      1.685      1.066     60.0%     74.6%
+    5      0.355      0.250     90.6%     93.8%
+   10      0.086      0.062     97.7%     98.3%
+   15      0.036      0.038     99.0%     99.0%
+   20      0.028      0.033     99.2%     99.3%
 ```
 
-We tested two popular HuggingFace transaction classifiers on real US bank descriptions. They scored **4/25** and **9/25**. This model scores **36/40**.
+### Honest Assessment
+
+The validation accuracy (99.3%) is on synthetic data — the same distribution the model was trained on. Real-world performance is lower because:
+
+- **Bank descriptions are noisier** than synthetic data. Abbreviations, custom POS prefixes, and institution-specific formats the generator didn't cover.
+- **Some categories are genuinely ambiguous.** WALGREENS is both a pharmacy (Healthcare) and a retail store (Shopping). ABC Stores in Hawaii sells groceries and souvenirs. The "correct" answer depends on what was purchased.
+- **The model doesn't know every merchant.** It learned patterns and common names, but niche local businesses or new companies won't match training data.
+- **ACH-format transactions are hard.** `INSTITUTION PURPOSE PPD ID:` looks the same whether it's insurance, utilities, or a subscription. The institution name is the only signal, and the model's vocabulary here is limited.
+
+In our testing against ~1,800 real transactions with bank/rule-derived labels, the model's raw accuracy was ~85%. However, about 6% of the "ground truth" labels were actually wrong (bank miscategorizations), and the model correctly overrode them — adjusted accuracy is ~92%.
 
 ## Categories (16)
 
@@ -49,104 +71,36 @@ We tested two popular HuggingFace transaction classifiers on real US bank descri
 | Restaurants | Fast food, sit-down, coffee shops, food delivery |
 | Groceries | Supermarkets, warehouse clubs, farmers markets |
 | Shopping | Retail, online purchases, department stores |
-| Transportation | Gas, rideshare, auto maintenance, parking, transit |
+| Transportation | Gas, rideshare, auto maintenance, parking |
 | Entertainment | Movies, events, gaming |
 | Utilities | Electric, internet, phone, water |
-| Subscription | Streaming, SaaS, news, software subscriptions |
+| Subscription | Streaming, SaaS, news, software |
 | Healthcare | Pharmacy, doctor, dentist, therapy |
 | Insurance | Auto, home, health, life insurance |
 | Housing | Rent, mortgage, home maintenance |
-| Travel | Hotels, airlines, car rental, booking sites |
+| Travel | Hotels, airlines, car rental |
 | Education | Online courses, books, tuition |
 | Personal Care | Salon, gym, beauty, spa |
-| Transfer | CC autopay, Zelle/Venmo sends, bank transfers, loan payments |
+| Transfer | CC autopay, Zelle/Venmo sends, bank transfers |
 | Income | Payroll, direct deposit, interest, refunds |
 | Fees | Bank fees, late fees, service charges |
 
-**Note:** "Business" is intentionally not a category. Whether a transaction is a business expense depends on which *account* it's charged to, not the merchant. An Anthropic subscription on a business account is a business expense; on a personal card it's a personal subscription. Both are classified as "Subscription" — the account context is a separate layer.
-
-## Performance
-
-```
-Validation Accuracy:  99.75%  (2,394/2,400)
-Real-World Accuracy:  90.0%   (36/40 on unseen bank descriptions)
-
-Per-Category Validation Accuracy:
-  Education           100.0%
-  Entertainment       100.0%
-  Fees                100.0%
-  Groceries           100.0%
-  Healthcare          100.0%
-  Housing             100.0%
-  Income              100.0%
-  Insurance           100.0%
-  Personal Care       100.0%
-  Restaurants         100.0%
-  Subscription        100.0%
-  Transfer            100.0%
-  Transportation      100.0%
-  Travel              100.0%
-  Utilities           100.0%
-  Shopping             96.1%
-```
-
-### Loss Curve
-
-```
-Epoch  Train Loss   Val Loss  Train Acc   Val Acc
-─────────────────────────────────────────────────
-    1      2.6755     2.2898     16.5%     41.5%
-    2      1.6954     1.0686     59.1%     74.5%
-    5      0.3614     0.2245     90.5%     94.4%
-   10      0.0708     0.0468     98.2%     98.5%
-   15      0.0320     0.0160     99.1%     99.6%
-   20      0.0212     0.0144     99.4%     99.8%
-```
-
-### Real-World Test Results
-
-Tested on actual transaction descriptions from US bank statements (not seen during training):
-
-```
-✓ Zelle payment to JOHN SMITH, CITY, CA    → Transfer        100%
-✓ AUTOMATIC PAYMENT - THANK                → Transfer        100%
-✓ STARBUCKS #12345                         → Restaurants      93%
-✓ CHEVRON 0203721                          → Transportation  100%
-✓ Netflix                                  → Subscription    100%
-✓ TARGET 00014720                          → Shopping        100%
-✓ FARMERS INS BILLING                      → Insurance       100%
-✓ UBER EATS                               → Restaurants     100%
-✓ WHOLE FOODS                              → Groceries       100%
-✓ AMAZON MKTPL*RJ7GA07V1                  → Shopping        100%
-✓ AMAZON WEB SERVICES                      → Subscription     95%
-✓ Mortgage payment                         → Housing         100%
-✓ WELLS FARGO IFI DDA TO DDA ...          → Transfer        100%
-✓ Patelco CU PAYROLL PPD ID: ...          → Income           99%
-```
+"Business" is intentionally not a category. Whether a transaction is a business expense depends on which account it's charged to, not the description. That's an account-level annotation, not a classification task.
 
 ## Usage
 
-### Python (Transformers)
+### Python
 
 ```python
 from transformers import pipeline
 
 classifier = pipeline("text-classification", model="DoDataThings/distilbert-us-transaction-classifier")
 
-transactions = [
-    "STARBUCKS #1234",
-    "AMAZON MKTPL*AB1CD2EF3",
-    "Zelle payment to JANE DOE, SEATTLE, WA 12345678901",
-    "AUTOMATIC PAYMENT - THANK",
-    "FARMERS INS BILLING",
-]
-
-for text in transactions:
-    result = classifier(text)[0]
-    print(f"{text:50s} → {result['label']:20s} {result['score']:.0%}")
+result = classifier("STARBUCKS #1234")
+print(result)  # [{'label': 'Restaurants', 'score': 0.93}]
 ```
 
-### JavaScript (Transformers.js / ONNX)
+### JavaScript (Transformers.js)
 
 ```javascript
 const { pipeline } = require('@xenova/transformers');
@@ -157,59 +111,21 @@ const classifier = await pipeline(
 );
 
 const result = await classifier('STARBUCKS #1234');
-console.log(result); // [{ label: 'Restaurants', score: 0.93 }]
+// [{ label: 'Restaurants', score: 0.93 }]
 ```
 
-### ONNX Runtime (direct)
-
-The model includes an ONNX export in the `onnx/` subdirectory for use with ONNX Runtime, Transformers.js, or any ONNX-compatible runtime.
-
-## Training Details
-
-| Parameter | Value |
-|-----------|-------|
-| Base model | `distilbert-base-uncased` |
-| Method | LoRA (r=32, alpha=64, dropout=0.1) |
-| Target modules | q_lin, k_lin, v_lin, out_lin + classifier head |
-| Trainable params | 1,782,544 / 68,748,320 (2.6%) |
-| Dataset | 16,000 synthetic transactions (1,000 per category) |
-| Epochs | 20 |
-| Batch size | 32 |
-| Learning rate | 3e-5 (linear warmup 10%) |
-| Training time | ~5 minutes on NVIDIA RTX GPU |
-
-### Training Data
-
-The model was trained on synthetic transaction descriptions generated to match real US bank statement formats. Six distinct format templates cover the major US banks:
-
-1. **ACH format** — fixed-width columns with `WEB ID:` or `PPD ID:` suffixes
-2. **Merchant + store number** — `MERCHANT #1234` or `MERCHANT*ORDERID`
-3. **Full address** — `MERCHANT ADDRESS CITY ZIP STATE COUNTRY`
-4. **PayPal prefix** — `PreApproved Payment Bill User Payment: MERCHANT`
-5. **Action prefix** — `Withdrawal from DESCRIPTION` / `Deposit from DESCRIPTION`
-6. **Simple** — `MERCHANT` or `MERCHANT.COM`
-
-Variations include randomized capitalization, spacing, store numbers, order IDs, city/state, and POS prefixes (`SQ *`, `TST*`).
-
-The synthetic dataset is published separately at [DoDataThings/us-bank-transaction-categories](https://huggingface.co/datasets/DoDataThings/us-bank-transaction-categories).
-
-## Recommended Use
-
-This model works best as **one layer in a classification pipeline**:
+An ONNX export is included in the `onnx/` subdirectory.
 
-1. **Merchant rules** (pattern matching) — catches known merchants and structural patterns
-2. **Bank-provided categories** — map bank's own classifications to your categories
-3. **This model** — classifies everything else
-4. **User overrides** — permanent manual corrections
+## Training Data
 
-The model handles the long tail that rules and bank categories miss. For the highest accuracy, combine all four layers.
+The synthetic dataset is published at [DoDataThings/us-bank-transaction-categories](https://huggingface.co/datasets/DoDataThings/us-bank-transaction-categories). The generator script is open source — you can extend the merchant pools, add format templates, or increase sample counts to improve accuracy for your use case.
 
 ## Limitations
 
-- Trained on US bank statement formats only — may not work well with international bank descriptions
-- Shopping is the weakest category (96.1%) due to overlap with Groceries and Subscription
-- Single-word descriptions like "Payment" are ambiguous — low confidence, should be handled by rules
-- The model classifies by transaction description only — it cannot determine account-level context (personal vs business)
+- **US bank formats only.** Trained on Chase, Apple Card, PayPal, and Capital One statement patterns. International bank descriptions will not classify well.
+- **Synthetic training data.** The model learned from generated descriptions, not real transaction data. It may miss patterns that only appear in real bank exports.
+- **Shopping is the weakest category** (~92% on synthetic) due to overlap with Subscription (Amazon sells both products and subscriptions) and Groceries (warehouse clubs).
+- **Not a standalone solution.** This model is a fallback layer. For production use, pair it with merchant rules and bank category mappings for best results.
 
 ## License