README.md

---
title: RiscAutious
emoji: 🏦
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.20.0
app_file: app.py
pinned: false
license: mit
---

# RiscAutious

**LoRA vs. full fine-tuning — how much accuracy do you keep for 0.3% of the trainable parameters?**

DistilBERT is fine-tuned to route customer banking queries to one of 77 support intents
(the public [banking77](https://huggingface.co/datasets/mteb/banking77) dataset). The
same model is trained two ways — full fine-tuning and LoRA — and compared on accuracy,
trainable parameter count, training time, and inference latency.

**Result: 92.2% test accuracy with LoRA vs 91.8% with full fine-tuning, training 321×
fewer parameters 2.4× faster.**

The repo also contains a documented **negative result**: the original task — predicting
loan risk grade from borrower text — was proven unlearnable, with the evidence kept.

LoRA is implemented from scratch as a plain `nn.Module`. No `peft`. The point of this
project is showing the low-rank mechanism explicitly, not showing that a library call
works.

---

## Status

Complete and verified end to end: data prepared, both modes trained, comparison
generated, demo serving locally. The data layer was built and verified before any model
code existed — bugs in tokenization or splitting are nearly invisible once a training
loop runs on top of them, and you end up blaming the model.

Every step, decision, and bug is logged in [ailog.md](ailog.md).

---

## The LoRA mechanism in one paragraph

Fine-tuning normally updates a pretrained weight matrix `W` directly. One DistilBERT
attention projection is 768×768 = 589,824 parameters, and there are four of them per
layer across six layers. LoRA freezes `W` and learns a low-rank correction instead:

```
W_effective = W + (alpha / r) · B·A        A: (r, 768)   B: (768, r)
```

With `r = 8` that is `8·768 + 768·8 = 12,288` trainable parameters per adapted
projection — about 2% of the original. The forward pass never builds `W + BA`; it runs
the two paths separately and adds them, keeping the memory saving:

```
y = xWᵀ + b  +  (alpha/r)·((xAᵀ)Bᵀ)
```

`B` is initialized to zeros, so at step 0 the adapted model is numerically identical to
the pretrained one and training departs smoothly from it rather than from a random
perturbation.

Adapters go on the **query** and **value** projections only (`q_lin`, `v_lin` in
DistilBERT) — the original paper's finding is that this gives the best accuracy per
parameter.

---

## Quickstart

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 1. Prepare data (banking77, ~13k rows, downloads from the HF Hub)
python data/download.py

# 2. Verify tokenization, splits, and batch shapes
python -m data.dataset --inspect

# 3. See the parameter counts before training anything
python -m models.classifier

# 4. Train both modes (~4 min and ~9 min on Apple Silicon; faster on a T4)
python train.py --mode lora --epochs 6
python train.py --mode full --epochs 6

# 5. Build the comparison table -> results/comparison.{md,csv}
python evaluate.py --per-class

# 6. Serve the demo at http://127.0.0.1:7860
python app.py
```

### On Google Colab (free GPU)

Runtime → Change runtime type → **T4 GPU**, then one cell:

```python
!git clone https://github.com/<you>/RiscAutious.git
%cd RiscAutious
!pip install -q gradio scikit-learn      # torch + transformers are preinstalled

!python data/download.py
!python train.py --mode lora --epochs 6
!python train.py --mode full --epochs 6
!python evaluate.py --per-class
```

Device selection is automatic (CUDA → MPS → CPU); nothing to configure. If you hit
CUDA OOM, add `--batch-size 16`.

### Reproducing the negative result

The LendingClub experiment is still wired up. It needs the 1.7 GB raw dump, pulled
automatically from a Hub mirror (no Kaggle account required):

```bash
python data/download.py --source hf --text-column desc --rows 8000
python train.py --mode lora --epochs 3
```

`--text-column desc` is required. Left to auto-resolve it picks `desc` anyway, but be
explicit — `title` and `purpose` look like text and are actually 14-value dropdowns.

---

## Methodology

**Task.** 77-way single-label classification. Input is one short customer banking
query; output is a support intent such as `card_arrival` or `declined_transfer`.

**Data.** banking77 — 12,989 real queries after deduplication, stratified 70/15/15 into
train/val/test (9,093 / 1,948 / 1,948). Classes are near-uniform (~170 rows each, the
largest is 1.7% of the data), so the majority baseline is 1.7% and accuracy is a
meaningful headline number here in a way it was not for the imbalanced loan task.

**Why this task and not the original one.** Loan grade is computed by LendingClub from
credit-bureau data, not from what the borrower wrote — three separate models confirmed
it is unlearnable from text (see the negative result below). banking77's label *is*
determined by the text: someone writing "my card hasn't arrived" **is** the
`card_arrival` intent. Swapping the dataset changed only `data/download.py`; every line
of the LoRA implementation, training loop, and evaluator was untouched.

**Sequence length.** `max_length=64`, which truncates 0.3% of examples (p99 is 42
words). Attention is quadratic in sequence length, so this is not a free parameter —
`download.py` prints the length distribution to set it from.

**Comparison fairness.** Both modes use the same seed, the same split, the same
tokenizer, and the same epoch count. Learning rates deliberately differ (LoRA ~1e-3,
full ~2e-5) — LoRA adapters start at zero and must travel far, while full fine-tuning
at 1e-3 would wash out the pretrained weights. Using one LR for both would make the
comparison look decisive for the wrong reason.

**Metrics.** Accuracy and macro F1 on the held-out test split, never validation
(validation picks the best epoch, so reporting on it is optimistic). Macro F1 is
reported alongside accuracy because it weights all 77 classes equally regardless of
size; here the two agree closely (0.922 vs 92.2%), which itself says the model is not
winning by neglecting rare classes.

---

## Results

banking77 intent classification, 12,989 rows, 77 classes, 6 epochs, batch 32, Apple
Silicon MPS. Generated by `evaluate.py` into [results/comparison.md](results/comparison.md).

| Model | Test accuracy | Macro F1 | Trainable params | % of total | Train time | Latency / example | Checkpoint |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Majority baseline | 1.7% | 0.000 | 0 | 0% | — | — | — |
| Full fine-tuning | 91.8% | 0.918 | 66,422,093 | 100.000% | 521.2s | 11.9 ms | 253 MB |
| **LoRA (r=8)** | **92.2%** | **0.922** | **206,669** | **0.310%** | **220.3s** | 8.9 ms | **817 KB** |

**LoRA matched and slightly beat full fine-tuning while training 321× fewer
parameters, 2.4× faster, into a checkpoint 318× smaller.**

The +0.4 point edge is within run-to-run noise — the honest claim is "no measurable
accuracy cost", not "LoRA is better". A plausible reason it does not lose: with only
9,093 training examples, updating all 66M parameters invites overfitting, and the
low-rank constraint acts as a regularizer. Full fine-tuning's train accuracy reached
96.3% against 90.8% validation; LoRA's gap was similar but from a lower-capacity start.

Latency is identical by construction (8.9 vs 11.9 ms — the difference is measurement
noise, not architecture). Both run the same 66M-parameter forward pass. **LoRA saves
training cost and storage, not inference time.** Anyone claiming otherwise has
misunderstood the method.

---

## The negative result: loan risk grade is not predictable from text

This project originally aimed to predict LendingClub loan risk grade (A–G) from the
borrower's written loan description. **That task turned out to be impossible**, and
proving it rigorously is part of the work. Evidence is preserved in
[results/grade_null_result/](results/grade_null_result/).

Measured on 12,000 real borrower descriptions from the LendingClub dump:

| Model | Test accuracy | Majority baseline | Lift |
| --- | --- | --- | --- |
| TF-IDF + logistic regression | 32.9% | 32.8% | **+0.1%** |
| DistilBERT, full fine-tuning | 31.6% | 31.5% | +0.1% |
| DistilBERT, LoRA r=8 | 32.2% | 31.5% | +0.7% |

Three independent models, none beating always-guess-the-most-common-grade.

**Why.** LendingClub *computes* the grade from FICO score, debt-to-income ratio, and
credit history. The borrower's free text is not an input to that function. The missing
ingredient is information, not examples — more rows would only measure zero more
precisely.

**How to tell this apart from underfitting.** The signature is `train ≈ val ≈ baseline`.
Full fine-tuning with 66M trainable parameters on 5,600 examples reached only 35.2%
train accuracy while validation *fell* to 29.0%: it began memorising noise rather than
finding structure. A capacity or data-volume problem looks the opposite — train
accuracy climbs far above validation.

**A methodological note worth stealing.** The TF-IDF control took seconds to run and
answered the question that two 10-minute GPU-free training runs could not: is there any
signal here at all? Run the cheap linear baseline first. If bag-of-words finds nothing,
a transformer will not rescue you.

Also measured along the way, on the same text: `desc` → loan purpose reaches 67-72%
(baseline 59%), so the text is perfectly learnable — just not for *grade*. And
LendingClub's `title` column, which looks like free text, has only **14 distinct
values**; it is a dropdown. Details in [ailog.md](ailog.md).

---

## Project layout

```
data/
  download.py      fetch/clean data -> text + label CSV + labels.json
  dataset.py       tokenization, stratified splits, DataLoaders
models/
  lora.py          LoRALinear from scratch; injection + freezing helpers
  classifier.py    DistilBERT + linear head, lora/full modes
train.py           training loop, --mode lora|full
evaluate.py        comparison table -> results/ (markdown + csv)
app.py             Gradio demo
results/
  comparison.md    the headline table
  grade_null_result/   preserved evidence for the unlearnable loan task
DEPLOY.md          pushing the demo to a free Hugging Face Space
ailog.md           log of every prompt, change, bug, and outcome
```

---

## Disclaimer

A demonstration of parameter-efficient fine-tuning, trained on the public banking77
research dataset. It is not connected to any bank, cannot see or act on any account,
and its predictions should not be used to route real customer requests without
human review.