ITheEqualizer/banking77-intent-error-predictor

Banking77 Intent Error Predictor

Maintainer: Ali Zakaee (ITheEqualizer)

This is a compact advisory complement for the included 77-way banking-support intent router. It estimates when that primary router is likely to be wrong so a fixed review budget can be spent on the riskiest requests. It does not approve transactions, make financial decisions, or replace access control or human review.

Quickstart

python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python -m banking_intent_error_predictor.reference_consumer "I was charged twice for one transfer"

The consumer returns either enqueue_human_review or an advisory route_to_<intent>_handler action and never executes that action.

Positive review example: a short or ambiguous request whose score geometry resembles a known primary-router error. Negative review example: a clear request such as “How do I activate my new card?” when the primary prediction is confident and the learned error risk stays below the reviewed threshold.

Task and data

The input is one non-empty English query of at most 512 characters. The bundled primary model produces probabilities in the exact 77-label BANKING77 order. The complement transforms those probabilities into class scores, predicted-intent one-hot features, top-score margin, normalized entropy, length, word count, digit count, question-mark presence, and a small negation indicator. It returns an error probability and recommends review at 0.27443790545050933 or above.

BANKING77 contains 13,083 author-released online-banking queries under CC BY 4.0. Data is pinned to source commit 57ec275d8078af65b7731c2a98be812d844a6d6b; exact file hashes are in dataset_manifest.json. Normalized-text hashes create 60/20/20 primary-train, complement-train, and validation partitions. Seven official-test rows overlapping development text are removed; the other 3,073 official-test rows form the untouched lockbox. Duplicate normalized text cannot cross development partitions.

Architecture and measured results

The primary baseline is word/character TF-IDF plus multinomial logistic regression. The error predictor is histogram gradient boosting with 140 iterations, 15 leaves, learning rate 0.06, minimum leaf size 25, balanced binary loss, and L2 regularization 1.0. This was chosen over publishing another saturated primary intent classifier because the measured downstream decision is which requests deserve review.

At about 20% review on validation, the learned model caught 168 of 210 primary errors (80.0%) versus 158 of 210 (75.2%) for the top-two-margin rule. Routed accuracy was 97.38% at 79.97% coverage versus 96.75% for margin at the same coverage. On the untouched lockbox, it caught 263 of 335 errors (78.51%); 72 errors were missed and 326 correct predictions were unnecessarily reviewed. Routed accuracy was 97.10% at 80.83% coverage.

The contradictory result matters: ranking PR-AUC was 0.485 on validation and 0.520 on lockbox, below margin-only PR-AUC of 0.516 and 0.586. The release claim is therefore limited to the predeclared fixed review-budget operating point, not better global ranking or universal calibration. The lockbox error-recall Wilson 95% interval is approximately 73.8%–82.6%.

Prediction trace

For “I was charged twice for one transfer,” text is normalized with NFKC, case folding, and whitespace collapse. The primary TF-IDF model produces 77 probabilities and an intent. Those probabilities plus bounded shape features enter the gradient-boosted error model. Its score is compared with the fixed threshold; above threshold the reference consumer requests review, otherwise it exposes the primary intent route.

Model and threshold selection used validation only. The official-test lockbox was evaluated once for the selected specification. A second clean fit produced exactly identical validation and lockbox scores and decisions; serialization round-trip was exact.

Limitations and integration

• English only; label coverage reflects BANKING77 and is not a bank’s live taxonomy.
• The complement is specific to the bundled primary model, label order, preprocessing, and versioned checksums.
• BANKING77 is small and older than current products, policies, fraud patterns, and user language.
• Error-ranking PR-AUC is worse than the margin baseline even though the fixed-budget decision is better.
• Review recommendation is advisory. Never use it to approve, deny, block, refund, authenticate, or move money.
• Monitor review rate, per-intent misses, drift, and calibration on local labeled traffic. Roll back by disabling the complement and using the included margin-only baseline or sending all requests to the existing safe path.

Reproduction

python -m pip install -r requirements-train.txt
python -m banking_intent_error_predictor.release_train --cache-dir cache --output-dir reproduced --reference reproduction_reference.npz

The command downloads only immutable, checksummed public source files, fits locally, permits at most eight numeric threads, and requires exact decision-score reproduction.

Licensing

Code and newly fitted model weights are MIT licensed. BANKING77 is CC BY 4.0; attribution and source hashes are in THIRD_PARTY_NOTICES.md and dataset_manifest.json.