lora-finetune-walkthrough

A from-scratch LoRA adapter for sentiment classification on GLUE SST-2, built on top of distilbert-base-uncased. LoRA factors are injected into the attention query/value projections; the frozen backbone is not stored here, only the trainable LoRA and classifier-head weights (lora_adapter.pt).

• Task: text classification (SST-2 sentiment, binary)
• Base model: distilbert-base-uncased
• Adapter: low-rank update on attention q/v projections

Results

Measured on an RTX 5090 with a small-scale slice (2000 train / 500 val) for one epoch, rank 8:

• Baseline (frozen base, random classifier head): 0.4000 validation accuracy
• LoRA-tuned: 0.8440 validation accuracy (an accuracy lift of +0.4440)
• Trainable parameters: 739,586, which is 1.10% of the 67.1M total
• Peak GPU memory: 1200.2 MB

LoRA lifts validation accuracy from 0.40 to about 0.84 while training only about 1% of the parameters. Read this as a small-scale RTX 5090 benchmark rather than a converged result.

Files

• lora_adapter.pt - state dict of the trainable weights (LoRA factors plus classifier head). Load it into a DistilBERT model wired with the same LoRA configuration (rank 8 on q_lin/v_lin).

Reproduce

Code and full walkthrough: https://github.com/narinzar/lora-finetune-walkthrough

License

MIT