Update unified_lora.py
Browse files- unified_lora.py +17 -0
unified_lora.py
CHANGED
|
@@ -1,4 +1,21 @@
|
|
| 1 |
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
Unified-LoRA Controller
|
| 3 |
========================
|
| 4 |
Adaptive per-layer rank controller for LoRA fine-tuning.
|
|
|
|
| 1 |
"""
|
| 2 |
+
Legacy Adaptive LoRA (Gradient-based)
|
| 3 |
+
====================================
|
| 4 |
+
|
| 5 |
+
Early experimental version of adaptive LoRA using gradient-based rank updates.
|
| 6 |
+
|
| 7 |
+
This approach adjusts rank per-layer based on gradient norm dynamics.
|
| 8 |
+
However, it suffers from instability and does not provide consistent benefits.
|
| 9 |
+
|
| 10 |
+
Replaced by:
|
| 11 |
+
- NestedLoRA (shared orbital architecture)
|
| 12 |
+
- OrbitalController (stress-based closed-loop control)
|
| 13 |
+
|
| 14 |
+
This file is kept for reference only.
|
| 15 |
+
|
| 16 |
+
Status: deprecated / legacy
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
Unified-LoRA Controller
|
| 20 |
========================
|
| 21 |
Adaptive per-layer rank controller for LoRA fine-tuning.
|