README.md

metadata

title: RiscAutious
emoji: 🏦
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: false
license: mit

RiscAutious

LoRA vs. full fine-tuning — how much accuracy do you keep for 0.3% of the trainable parameters?

DistilBERT is fine-tuned to route customer banking queries to one of 77 support intents (the public banking77 dataset). The same model is trained two ways — full fine-tuning and LoRA — and compared on accuracy, trainable parameter count, training time, and inference latency.

Result: 92.2% test accuracy with LoRA vs 91.8% with full fine-tuning, training 321× fewer parameters 2.4× faster.

The repo also contains a documented negative result: the original task — predicting loan risk grade from borrower text — was proven unlearnable, with the evidence kept.

LoRA is implemented from scratch as a plain nn.Module. No peft. The point of this project is showing the low-rank mechanism explicitly, not showing that a library call works.

---

Status

Complete and verified end to end: data prepared, both modes trained, comparison generated, demo serving locally. The data layer was built and verified before any model code existed — bugs in tokenization or splitting are nearly invisible once a training loop runs on top of them, and you end up blaming the model.

Every step, decision, and bug is logged in ailog.md.

---

The LoRA mechanism in one paragraph

Fine-tuning normally updates a pretrained weight matrix W directly. One DistilBERT attention projection is 768×768 = 589,824 parameters, and there are four of them per layer across six layers. LoRA freezes W and learns a low-rank correction instead:

W_effective = W + (alpha / r) · B·A        A: (r, 768)   B: (768, r)

With r = 8 that is 8·768 + 768·8 = 12,288 trainable parameters per adapted projection — about 2% of the original. The forward pass never builds W + BA; it runs the two paths separately and adds them, keeping the memory saving:

y = xWᵀ + b  +  (alpha/r)·((xAᵀ)Bᵀ)

B is initialized to zeros, so at step 0 the adapted model is numerically identical to the pretrained one and training departs smoothly from it rather than from a random perturbation.

Adapters go on the query and value projections only (q_lin, v_lin in DistilBERT) — the original paper's finding is that this gives the best accuracy per parameter.

---

Quickstart

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 1. Prepare data (banking77, ~13k rows, downloads from the HF Hub)
python data/download.py

# 2. Verify tokenization, splits, and batch shapes
python -m data.dataset --inspect

# 3. See the parameter counts before training anything
python -m models.classifier

# 4. Train both modes (~4 min and ~9 min on Apple Silicon; faster on a T4)
python train.py --mode lora --epochs 6
python train.py --mode full --epochs 6

# 5. Build the comparison table -> results/comparison.{md,csv}
python evaluate.py --per-class

# 6. Serve the demo at http://127.0.0.1:7860
python app.py

On Google Colab (free GPU)

Runtime → Change runtime type → T4 GPU, then one cell:

!git clone https://github.com/<you>/RiscAutious.git
%cd RiscAutious
!pip install -q gradio scikit-learn      # torch + transformers are preinstalled

!python data/download.py
!python train.py --mode lora --epochs 6
!python train.py --mode full --epochs 6
!python evaluate.py --per-class

Device selection is automatic (CUDA → MPS → CPU); nothing to configure. If you hit CUDA OOM, add --batch-size 16.

Reproducing the negative result

The LendingClub experiment is still wired up. It needs the 1.7 GB raw dump, pulled automatically from a Hub mirror (no Kaggle account required):

python data/download.py --source hf --text-column desc --rows 8000
python train.py --mode lora --epochs 3

--text-column desc is required. Left to auto-resolve it picks desc anyway, but be explicit — title and purpose look like text and are actually 14-value dropdowns.

---

Methodology

Task. 77-way single-label classification. Input is one short customer banking query; output is a support intent such as card_arrival or declined_transfer.

Data. banking77 — 12,989 real queries after deduplication, stratified 70/15/15 into train/val/test (9,093 / 1,948 / 1,948). Classes are near-uniform (~170 rows each, the largest is 1.7% of the data), so the majority baseline is 1.7% and accuracy is a meaningful headline number here in a way it was not for the imbalanced loan task.

Why this task and not the original one. Loan grade is computed by LendingClub from credit-bureau data, not from what the borrower wrote — three separate models confirmed it is unlearnable from text (see the negative result below). banking77's label is determined by the text: someone writing "my card hasn't arrived" is the card_arrival intent. Swapping the dataset changed only data/download.py; every line of the LoRA implementation, training loop, and evaluator was untouched.

Sequence length. max_length=64, which truncates 0.3% of examples (p99 is 42 words). Attention is quadratic in sequence length, so this is not a free parameter — download.py prints the length distribution to set it from.

Comparison fairness. Both modes use the same seed, the same split, the same tokenizer, and the same epoch count. Learning rates deliberately differ (LoRA ~1e-3, full ~2e-5) — LoRA adapters start at zero and must travel far, while full fine-tuning at 1e-3 would wash out the pretrained weights. Using one LR for both would make the comparison look decisive for the wrong reason.

Metrics. Accuracy and macro F1 on the held-out test split, never validation (validation picks the best epoch, so reporting on it is optimistic). Macro F1 is reported alongside accuracy because it weights all 77 classes equally regardless of size; here the two agree closely (0.922 vs 92.2%), which itself says the model is not winning by neglecting rare classes.

---

Results

banking77 intent classification, 12,989 rows, 77 classes, 6 epochs, batch 32, Apple Silicon MPS. Generated by evaluate.py into results/comparison.md.

Model	Test accuracy	Macro F1	Trainable params	% of total	Train time	Latency / example	Checkpoint
Majority baseline	1.7%	0.000	0	0%	—	—	—
Full fine-tuning	91.8%	0.918	66,422,093	100.000%	521.2s	11.9 ms	253 MB
LoRA (r=8)	92.2%	0.922	206,669	0.310%	220.3s	8.9 ms	817 KB

LoRA matched and slightly beat full fine-tuning while training 321× fewer parameters, 2.4× faster, into a checkpoint 318× smaller.

The +0.4 point edge is within run-to-run noise — the honest claim is "no measurable accuracy cost", not "LoRA is better". A plausible reason it does not lose: with only 9,093 training examples, updating all 66M parameters invites overfitting, and the low-rank constraint acts as a regularizer. Full fine-tuning's train accuracy reached 96.3% against 90.8% validation; LoRA's gap was similar but from a lower-capacity start.

Latency is identical by construction (8.9 vs 11.9 ms — the difference is measurement noise, not architecture). Both run the same 66M-parameter forward pass. LoRA saves training cost and storage, not inference time. Anyone claiming otherwise has misunderstood the method.

---

The negative result: loan risk grade is not predictable from text

This project originally aimed to predict LendingClub loan risk grade (A–G) from the borrower's written loan description. That task turned out to be impossible, and proving it rigorously is part of the work. Evidence is preserved in results/grade_null_result/.

Measured on 12,000 real borrower descriptions from the LendingClub dump:

Model	Test accuracy	Majority baseline	Lift
TF-IDF + logistic regression	32.9%	32.8%	+0.1%
DistilBERT, full fine-tuning	31.6%	31.5%	+0.1%
DistilBERT, LoRA r=8	32.2%	31.5%	+0.7%

Three independent models, none beating always-guess-the-most-common-grade.

Why. LendingClub computes the grade from FICO score, debt-to-income ratio, and credit history. The borrower's free text is not an input to that function. The missing ingredient is information, not examples — more rows would only measure zero more precisely.

How to tell this apart from underfitting. The signature is train ≈ val ≈ baseline. Full fine-tuning with 66M trainable parameters on 5,600 examples reached only 35.2% train accuracy while validation fell to 29.0%: it began memorising noise rather than finding structure. A capacity or data-volume problem looks the opposite — train accuracy climbs far above validation.

A methodological note worth stealing. The TF-IDF control took seconds to run and answered the question that two 10-minute GPU-free training runs could not: is there any signal here at all? Run the cheap linear baseline first. If bag-of-words finds nothing, a transformer will not rescue you.

Also measured along the way, on the same text: desc → loan purpose reaches 67-72% (baseline 59%), so the text is perfectly learnable — just not for grade. And LendingClub's title column, which looks like free text, has only 14 distinct values; it is a dropdown. Details in ailog.md.

---

Project layout

data/
  download.py      fetch/clean data -> text + label CSV + labels.json
  dataset.py       tokenization, stratified splits, DataLoaders
models/
  lora.py          LoRALinear from scratch; injection + freezing helpers
  classifier.py    DistilBERT + linear head, lora/full modes
train.py           training loop, --mode lora|full
evaluate.py        comparison table -> results/ (markdown + csv)
app.py             Gradio demo
results/
  comparison.md    the headline table
  grade_null_result/   preserved evidence for the unlearnable loan task
DEPLOY.md          pushing the demo to a free Hugging Face Space
ailog.md           log of every prompt, change, bug, and outcome

---

Disclaimer

A demonstration of parameter-efficient fine-tuning, trained on the public banking77 research dataset. It is not connected to any bank, cannot see or act on any account, and its predictions should not be used to route real customer requests without human review.