Text Classification
Transformers
lora
fine-tuning
adaptive
research
nested-lora
synaptic-plasticity
rank-adaptation
Instructions to use Simo76/Unified-LoRA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Simo76/Unified-LoRA with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Simo76/Unified-LoRA")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Simo76/Unified-LoRA", dtype="auto") - Notebooks
- Google Colab
- Kaggle
Revise README for improved clarity and structure
Browse filesUpdated README to enhance clarity and structure, including detailed descriptions of operational modes, validation results, quick start guide, technical details, installation instructions, citation, and contact information.
README.md
CHANGED
|
@@ -1,2 +1,136 @@
|
|
| 1 |
-
# Unified
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Unified LoRA
|
| 2 |
+
|
| 3 |
+
**Adaptive Single/Multi/Mirror LoRA framework with automatic mode switching driven by synaptic stress signals.**
|
| 4 |
+
|
| 5 |
+
[](https://opensource.org/licenses/Apache-2.0)
|
| 6 |
+
|
| 7 |
+
## Overview
|
| 8 |
+
|
| 9 |
+
Unified LoRA is a dynamic parameter-efficient fine-tuning system that automatically switches between three operational modes based on training stress:
|
| 10 |
+
|
| 11 |
+
- **Mode 0 (Single)**: Shared adapter for low-conflict scenarios
|
| 12 |
+
- **Mode 1 (Multi)**: Task-specific LoRA adapters for moderate stress
|
| 13 |
+
- **Mode 2 (Mirror)**: Stability snapshots for catastrophic forgetting prevention
|
| 14 |
+
|
| 15 |
+
The system uses a synaptic control parameter **φ(t)** derived from:
|
| 16 |
+
- **C**: Task conflict (weight space variance)
|
| 17 |
+
- **E**: Multi-task error
|
| 18 |
+
- **S**: Memory stability
|
| 19 |
+
|
| 20 |
+
```
|
| 21 |
+
φ(t) = f(C, E, S, ΔC, ΔE, ΔS)
|
| 22 |
+
```
|
| 23 |
+
|
| 24 |
+
## Validation Results
|
| 25 |
+
|
| 26 |
+
### 🧪 Production LLM (Tinker + Llama-3.2-1B)
|
| 27 |
+
|
| 28 |
+
Full stress→recovery cycle demonstrated on cloud GPU:
|
| 29 |
+
|
| 30 |
+
```
|
| 31 |
+
[250] Mode=1 φ=0.333 E_s=0.500 lr=5e-4 (Multi stable)
|
| 32 |
+
>>> SHOCK @ step 300
|
| 33 |
+
[350] Mode=2 φ=0.827 E_s=4.979 lr=1e-4 (Mirror activated)
|
| 34 |
+
>>> RECOVERY @ step 500
|
| 35 |
+
[550] Mode=1 φ=0.371 E_s=0.521 lr=5e-4 (Multi return)
|
| 36 |
+
[700] Mode=1 φ=0.333 E_s=0.500 lr=5e-4 (baseline restored)
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
**Key finding**: Complete reversibility (φ: 0.33 → 0.83 → 0.33)
|
| 40 |
+
|
| 41 |
+
### 📊 Standard Benchmark (GLUE MRPC + DistilBERT)
|
| 42 |
+
|
| 43 |
+
Performance parity with baseline LoRA:
|
| 44 |
+
|
| 45 |
+
| Method | F1 | Accuracy | φ final | Mode |
|
| 46 |
+
|--------|-----|----------|---------|------|
|
| 47 |
+
| Baseline LoRA | 0.785 | 0.646 | - | - |
|
| 48 |
+
| Unified LoRA | 0.785 | 0.646 | 0.367 | 1 |
|
| 49 |
+
|
| 50 |
+
**Key finding**: Zero degradation with adaptive control active
|
| 51 |
+
|
| 52 |
+
## Quick Start
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
from unified_lora import UnifiedController
|
| 56 |
+
|
| 57 |
+
# Initialize controller
|
| 58 |
+
controller = UnifiedController(
|
| 59 |
+
alpha=0.1, # φ(t) learning rate
|
| 60 |
+
beta=0.9, # EMA smoothing
|
| 61 |
+
theta0=0.3, # Single/Multi threshold
|
| 62 |
+
theta1=0.7 # Multi/Mirror threshold
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
# Training loop
|
| 66 |
+
for step, batch in enumerate(train_loader):
|
| 67 |
+
outputs = model(**batch)
|
| 68 |
+
loss = outputs.loss
|
| 69 |
+
|
| 70 |
+
# Update controller and get adaptive LR
|
| 71 |
+
new_lr = controller.update(loss.item())
|
| 72 |
+
|
| 73 |
+
# Apply new learning rate
|
| 74 |
+
for g in optimizer.param_groups:
|
| 75 |
+
g['lr'] = new_lr
|
| 76 |
+
|
| 77 |
+
# Standard backprop
|
| 78 |
+
loss.backward()
|
| 79 |
+
optimizer.step()
|
| 80 |
+
optimizer.zero_grad()
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
## Technical Details
|
| 84 |
+
|
| 85 |
+
### FSM State Transitions
|
| 86 |
+
|
| 87 |
+
```
|
| 88 |
+
φ < 0.3 → Mode 0 (Single) LR = 5e-5
|
| 89 |
+
φ < 0.7 → Mode 1 (Multi) LR = 3e-5
|
| 90 |
+
φ ≥ 0.7 → Mode 2 (Mirror) LR = 1e-5
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
### Stress Signal Computation
|
| 94 |
+
|
| 95 |
+
The metrics **C** (conflict), **E** (error), and **S** (stability) are normalized to [0,1] to maintain φ(t) well-conditioned:
|
| 96 |
+
|
| 97 |
+
```python
|
| 98 |
+
E_smooth = β * E_smooth + (1 - β) * loss
|
| 99 |
+
D = E_smooth / (1 + E_smooth) # Normalize to [0,1]
|
| 100 |
+
φ = (1 - α) * φ + α * D # EMA update
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
This normalization ensures stable FSM transitions and prevents numerical instabilities during training.
|
| 104 |
+
|
| 105 |
+
## Installation
|
| 106 |
+
|
| 107 |
+
```bash
|
| 108 |
+
pip install transformers peft torch
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
## Citation
|
| 112 |
+
|
| 113 |
+
If you use Unified LoRA in your research, please cite:
|
| 114 |
+
|
| 115 |
+
```bibtex
|
| 116 |
+
@software{unified_lora_2025,
|
| 117 |
+
author = {Simona Vargiu},
|
| 118 |
+
title = {Unified LoRA: Adaptive Parameter-Efficient Fine-Tuning},
|
| 119 |
+
year = {2025},
|
| 120 |
+
url = {https://github.com/Sva76/Unified-LoRA}
|
| 121 |
+
}
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
## License
|
| 125 |
+
|
| 126 |
+
Apache License 2.0 - see [LICENSE](LICENSE) for details.
|
| 127 |
+
|
| 128 |
+
## Contact
|
| 129 |
+
|
| 130 |
+
**Simona Vargiu** (Independent Researcher)
|
| 131 |
+
|
| 132 |
+
For collaboration inquiries: [simona.vargiu.malta@gmail.com](mailto:simona.vargiu.malta@gmail.com)
|
| 133 |
+
|
| 134 |
+
---
|
| 135 |
+
|
| 136 |
+
**Status**: Validated on production LLM (Tinker/Llama-3.2-1B) and standard benchmarks (GLUE MRPC). Ready for research collaboration and testing.
|