Simo76 commited on
Commit
dd090d6
·
1 Parent(s): 10cb430

Revise README with updated findings and details

Browse files

Updated performance observations, technical details, and installation instructions in README.

Files changed (1) hide show
  1. README.md +18 -32
README.md CHANGED
@@ -38,18 +38,18 @@ This experiment illustrates the behavior of **φ(t)** under induced instability;
38
 
39
  ### 📊 Standard Benchmark (GLUE MRPC + DistilBERT)
40
 
41
- Performance parity with baseline LoRA:
42
 
43
  | Method | F1 | Accuracy | φ final | Mode |
44
  |--------|-----|----------|---------|------|
45
  | Baseline LoRA | 0.785 | 0.646 | - | - |
46
  | Unified LoRA | 0.785 | 0.646 | 0.367 | 1 |
47
 
48
- **Key finding**: Zero degradation with adaptive control active
 
49
 
50
  ## Quick Start
51
 
52
- ```python
53
  from unified_lora import UnifiedController
54
 
55
  # Initialize controller
@@ -64,71 +64,57 @@ controller = UnifiedController(
64
  for step, batch in enumerate(train_loader):
65
  outputs = model(**batch)
66
  loss = outputs.loss
67
-
68
  # Update controller and get adaptive LR
69
  new_lr = controller.update(loss.item())
70
-
71
  # Apply new learning rate
72
  for g in optimizer.param_groups:
73
  g['lr'] = new_lr
74
-
75
  # Standard backprop
76
  loss.backward()
77
  optimizer.step()
78
  optimizer.zero_grad()
79
- ```
80
 
81
  ## Technical Details
82
 
83
  ### FSM State Transitions
84
 
85
- ```
86
- φ < 0.3 → Mode 0 (Single) LR = 5e-5
87
- φ < 0.7 → Mode 1 (Multi) LR = 3e-5
88
- φ ≥ 0.7 → Mode 2 (Mirror) LR = 1e-5
89
- ```
90
 
91
  ### Stress Signal Computation
92
 
93
- The metrics **C** (conflict), **E** (error), and **S** (stability) are normalized to [0,1] to maintain φ(t) well-conditioned:
94
 
95
- ```python
96
- E_smooth = β * E_smooth + (1 - β) * loss
97
- D = E_smooth / (1 + E_smooth) # Normalize to [0,1]
98
- φ = (1 - α) * φ + α * D # EMA update
99
- ```
100
 
101
- This normalization ensures stable FSM transitions and prevents numerical instabilities during training.
102
 
103
  ## Installation
104
 
105
- ```bash
106
  pip install transformers peft torch
107
- ```
108
 
109
  ## Citation
110
 
111
- If you use Unified LoRA in your research, please cite:
112
-
113
- ```bibtex
114
  @software{unified_lora_2025,
115
  author = {Simona Vargiu},
116
  title = {Unified LoRA: Adaptive Parameter-Efficient Fine-Tuning},
117
  year = {2025},
118
  url = {https://github.com/Sva76/Unified-LoRA}
119
  }
120
- ```
121
 
122
  ## License
123
 
124
- Apache License 2.0 - see [LICENSE](LICENSE) for details.
125
 
126
  ## Contact
127
 
128
- **Simona Vargiu** (Independent Researcher)
129
-
130
- For collaboration inquiries: [simona.vargiu.malta@gmail.com](mailto:simona.vargiu.malta@gmail.com)
131
-
132
- ---
133
 
134
- **Status**: Validated on production LLM (Tinker/Llama-3.2-1B) and standard benchmarks (GLUE MRPC). Ready for research collaboration and testing.
 
38
 
39
  ### 📊 Standard Benchmark (GLUE MRPC + DistilBERT)
40
 
41
+ Observed comparable performance to baseline LoRA on GLUE MRPC under the reported configuration:
42
 
43
  | Method | F1 | Accuracy | φ final | Mode |
44
  |--------|-----|----------|---------|------|
45
  | Baseline LoRA | 0.785 | 0.646 | - | - |
46
  | Unified LoRA | 0.785 | 0.646 | 0.367 | 1 |
47
 
48
+ Key observation (this run):
49
+ No performance degradation observed with adaptive control active in the reported setup.
50
 
51
  ## Quick Start
52
 
 
53
  from unified_lora import UnifiedController
54
 
55
  # Initialize controller
 
64
  for step, batch in enumerate(train_loader):
65
  outputs = model(**batch)
66
  loss = outputs.loss
67
+
68
  # Update controller and get adaptive LR
69
  new_lr = controller.update(loss.item())
70
+
71
  # Apply new learning rate
72
  for g in optimizer.param_groups:
73
  g['lr'] = new_lr
74
+
75
  # Standard backprop
76
  loss.backward()
77
  optimizer.step()
78
  optimizer.zero_grad()
 
79
 
80
  ## Technical Details
81
 
82
  ### FSM State Transitions
83
 
84
+ φ < 0.3 → Mode 0 (Single) LR = 5e-5
85
+ φ < 0.7 → Mode 1 (Multi) LR = 3e-5
86
+ φ 0.7 → Mode 2 (Mirror) LR = 1e-5
 
 
87
 
88
  ### Stress Signal Computation
89
 
90
+ The metrics C (conflict), E (error), and S (stability) are normalized to [0,1] to keep φ(t) well-conditioned:
91
 
92
+ E_smooth = β * E_smooth + (1 - β) * loss
93
+ D = E_smooth / (1 + E_smooth)
94
+ φ = (1 - α) * φ + α * D
 
 
95
 
96
+ This normalization supports stable FSM transitions and reduces numerical sensitivity during training.
97
 
98
  ## Installation
99
 
 
100
  pip install transformers peft torch
 
101
 
102
  ## Citation
103
 
 
 
 
104
  @software{unified_lora_2025,
105
  author = {Simona Vargiu},
106
  title = {Unified LoRA: Adaptive Parameter-Efficient Fine-Tuning},
107
  year = {2025},
108
  url = {https://github.com/Sva76/Unified-LoRA}
109
  }
 
110
 
111
  ## License
112
 
113
+ Apache License 2.0 see LICENSE for details.
114
 
115
  ## Contact
116
 
117
+ Simona Vargiu (Independent Researcher)
118
+ For collaboration inquiries: simona.vargiu.malta@gmail.com
 
 
 
119
 
120
+ Status: Demonstrated on (1) a Tinker Llama-3.2-1B run showing adaptive stress → recovery behavior and (2) the GLUE MRPC benchmark (DistilBERT) with comparable F1/Accuracy to baseline LoRA under the reported configuration. Broader benchmarks and statistical evaluation across seeds and tasks are ongoing. Open to research collaboration and extended evaluation.