--- title: RiscAutious emoji: ๐Ÿฆ colorFrom: blue colorTo: indigo sdk: gradio sdk_version: 6.20.0 app_file: app.py pinned: false license: mit --- # RiscAutious **LoRA vs. full fine-tuning โ€” how much accuracy do you keep for 0.3% of the trainable parameters?** DistilBERT is fine-tuned to route customer banking queries to one of 77 support intents (the public [banking77](https://huggingface.co/datasets/mteb/banking77) dataset). The same model is trained two ways โ€” full fine-tuning and LoRA โ€” and compared on accuracy, trainable parameter count, training time, and inference latency. **Result: 92.2% test accuracy with LoRA vs 91.8% with full fine-tuning, training 321ร— fewer parameters 2.4ร— faster.** The repo also contains a documented **negative result**: the original task โ€” predicting loan risk grade from borrower text โ€” was proven unlearnable, with the evidence kept. LoRA is implemented from scratch as a plain `nn.Module`. No `peft`. The point of this project is showing the low-rank mechanism explicitly, not showing that a library call works. --- ## Status Complete and verified end to end: data prepared, both modes trained, comparison generated, demo serving locally. The data layer was built and verified before any model code existed โ€” bugs in tokenization or splitting are nearly invisible once a training loop runs on top of them, and you end up blaming the model. Every step, decision, and bug is logged in [ailog.md](ailog.md). --- ## The LoRA mechanism in one paragraph Fine-tuning normally updates a pretrained weight matrix `W` directly. One DistilBERT attention projection is 768ร—768 = 589,824 parameters, and there are four of them per layer across six layers. LoRA freezes `W` and learns a low-rank correction instead: ``` W_effective = W + (alpha / r) ยท BยทA A: (r, 768) B: (768, r) ``` With `r = 8` that is `8ยท768 + 768ยท8 = 12,288` trainable parameters per adapted projection โ€” about 2% of the original. The forward pass never builds `W + BA`; it runs the two paths separately and adds them, keeping the memory saving: ``` y = xWแต€ + b + (alpha/r)ยท((xAแต€)Bแต€) ``` `B` is initialized to zeros, so at step 0 the adapted model is numerically identical to the pretrained one and training departs smoothly from it rather than from a random perturbation. Adapters go on the **query** and **value** projections only (`q_lin`, `v_lin` in DistilBERT) โ€” the original paper's finding is that this gives the best accuracy per parameter. --- ## Quickstart ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt # 1. Prepare data (banking77, ~13k rows, downloads from the HF Hub) python data/download.py # 2. Verify tokenization, splits, and batch shapes python -m data.dataset --inspect # 3. See the parameter counts before training anything python -m models.classifier # 4. Train both modes (~4 min and ~9 min on Apple Silicon; faster on a T4) python train.py --mode lora --epochs 6 python train.py --mode full --epochs 6 # 5. Build the comparison table -> results/comparison.{md,csv} python evaluate.py --per-class # 6. Serve the demo at http://127.0.0.1:7860 python app.py ``` ### On Google Colab (free GPU) Runtime โ†’ Change runtime type โ†’ **T4 GPU**, then one cell: ```python !git clone https://github.com//RiscAutious.git %cd RiscAutious !pip install -q gradio scikit-learn # torch + transformers are preinstalled !python data/download.py !python train.py --mode lora --epochs 6 !python train.py --mode full --epochs 6 !python evaluate.py --per-class ``` Device selection is automatic (CUDA โ†’ MPS โ†’ CPU); nothing to configure. If you hit CUDA OOM, add `--batch-size 16`. ### Reproducing the negative result The LendingClub experiment is still wired up. It needs the 1.7 GB raw dump, pulled automatically from a Hub mirror (no Kaggle account required): ```bash python data/download.py --source hf --text-column desc --rows 8000 python train.py --mode lora --epochs 3 ``` `--text-column desc` is required. Left to auto-resolve it picks `desc` anyway, but be explicit โ€” `title` and `purpose` look like text and are actually 14-value dropdowns. --- ## Methodology **Task.** 77-way single-label classification. Input is one short customer banking query; output is a support intent such as `card_arrival` or `declined_transfer`. **Data.** banking77 โ€” 12,989 real queries after deduplication, stratified 70/15/15 into train/val/test (9,093 / 1,948 / 1,948). Classes are near-uniform (~170 rows each, the largest is 1.7% of the data), so the majority baseline is 1.7% and accuracy is a meaningful headline number here in a way it was not for the imbalanced loan task. **Why this task and not the original one.** Loan grade is computed by LendingClub from credit-bureau data, not from what the borrower wrote โ€” three separate models confirmed it is unlearnable from text (see the negative result below). banking77's label *is* determined by the text: someone writing "my card hasn't arrived" **is** the `card_arrival` intent. Swapping the dataset changed only `data/download.py`; every line of the LoRA implementation, training loop, and evaluator was untouched. **Sequence length.** `max_length=64`, which truncates 0.3% of examples (p99 is 42 words). Attention is quadratic in sequence length, so this is not a free parameter โ€” `download.py` prints the length distribution to set it from. **Comparison fairness.** Both modes use the same seed, the same split, the same tokenizer, and the same epoch count. Learning rates deliberately differ (LoRA ~1e-3, full ~2e-5) โ€” LoRA adapters start at zero and must travel far, while full fine-tuning at 1e-3 would wash out the pretrained weights. Using one LR for both would make the comparison look decisive for the wrong reason. **Metrics.** Accuracy and macro F1 on the held-out test split, never validation (validation picks the best epoch, so reporting on it is optimistic). Macro F1 is reported alongside accuracy because it weights all 77 classes equally regardless of size; here the two agree closely (0.922 vs 92.2%), which itself says the model is not winning by neglecting rare classes. --- ## Results banking77 intent classification, 12,989 rows, 77 classes, 6 epochs, batch 32, Apple Silicon MPS. Generated by `evaluate.py` into [results/comparison.md](results/comparison.md). | Model | Test accuracy | Macro F1 | Trainable params | % of total | Train time | Latency / example | Checkpoint | | --- | --- | --- | --- | --- | --- | --- | --- | | Majority baseline | 1.7% | 0.000 | 0 | 0% | โ€” | โ€” | โ€” | | Full fine-tuning | 91.8% | 0.918 | 66,422,093 | 100.000% | 521.2s | 11.9 ms | 253 MB | | **LoRA (r=8)** | **92.2%** | **0.922** | **206,669** | **0.310%** | **220.3s** | 8.9 ms | **817 KB** | **LoRA matched and slightly beat full fine-tuning while training 321ร— fewer parameters, 2.4ร— faster, into a checkpoint 318ร— smaller.** The +0.4 point edge is within run-to-run noise โ€” the honest claim is "no measurable accuracy cost", not "LoRA is better". A plausible reason it does not lose: with only 9,093 training examples, updating all 66M parameters invites overfitting, and the low-rank constraint acts as a regularizer. Full fine-tuning's train accuracy reached 96.3% against 90.8% validation; LoRA's gap was similar but from a lower-capacity start. Latency is identical by construction (8.9 vs 11.9 ms โ€” the difference is measurement noise, not architecture). Both run the same 66M-parameter forward pass. **LoRA saves training cost and storage, not inference time.** Anyone claiming otherwise has misunderstood the method. --- ## The negative result: loan risk grade is not predictable from text This project originally aimed to predict LendingClub loan risk grade (Aโ€“G) from the borrower's written loan description. **That task turned out to be impossible**, and proving it rigorously is part of the work. Evidence is preserved in [results/grade_null_result/](results/grade_null_result/). Measured on 12,000 real borrower descriptions from the LendingClub dump: | Model | Test accuracy | Majority baseline | Lift | | --- | --- | --- | --- | | TF-IDF + logistic regression | 32.9% | 32.8% | **+0.1%** | | DistilBERT, full fine-tuning | 31.6% | 31.5% | +0.1% | | DistilBERT, LoRA r=8 | 32.2% | 31.5% | +0.7% | Three independent models, none beating always-guess-the-most-common-grade. **Why.** LendingClub *computes* the grade from FICO score, debt-to-income ratio, and credit history. The borrower's free text is not an input to that function. The missing ingredient is information, not examples โ€” more rows would only measure zero more precisely. **How to tell this apart from underfitting.** The signature is `train โ‰ˆ val โ‰ˆ baseline`. Full fine-tuning with 66M trainable parameters on 5,600 examples reached only 35.2% train accuracy while validation *fell* to 29.0%: it began memorising noise rather than finding structure. A capacity or data-volume problem looks the opposite โ€” train accuracy climbs far above validation. **A methodological note worth stealing.** The TF-IDF control took seconds to run and answered the question that two 10-minute GPU-free training runs could not: is there any signal here at all? Run the cheap linear baseline first. If bag-of-words finds nothing, a transformer will not rescue you. Also measured along the way, on the same text: `desc` โ†’ loan purpose reaches 67-72% (baseline 59%), so the text is perfectly learnable โ€” just not for *grade*. And LendingClub's `title` column, which looks like free text, has only **14 distinct values**; it is a dropdown. Details in [ailog.md](ailog.md). --- ## Project layout ``` data/ download.py fetch/clean data -> text + label CSV + labels.json dataset.py tokenization, stratified splits, DataLoaders models/ lora.py LoRALinear from scratch; injection + freezing helpers classifier.py DistilBERT + linear head, lora/full modes train.py training loop, --mode lora|full evaluate.py comparison table -> results/ (markdown + csv) app.py Gradio demo results/ comparison.md the headline table grade_null_result/ preserved evidence for the unlearnable loan task DEPLOY.md pushing the demo to a free Hugging Face Space ailog.md log of every prompt, change, bug, and outcome ``` --- ## Disclaimer A demonstration of parameter-efficient fine-tuning, trained on the public banking77 research dataset. It is not connected to any bank, cannot see or act on any account, and its predictions should not be used to route real customer requests without human review.