docs/architecture.md

Architecture — Nested Orbital LoRA


Core idea: dynamic rank control via stress-driven orbital transitions with weight persistence (no cold start).



Problem: cold start on rank transitions


Standard multi-rank LoRA keeps separate adapters per rank:


r=4, r=8, r=16 → independent weights


Switching rank causes partial cold restarts → performance drop.



Solution: Nested LoRA (one adapter, multiple ranks)


Single adapter at max rank:


A(16, d), B(d, 16)


Active rank is obtained by slicing:




r=4  → A[:4, :],  B[:, :4]


r=8  → A[:8, :],  B[:, :8]


r=16 → full matrix




r4 ⊂ r8 ⊂ r16



Lower ranks reuse trained weights → no cold start.



Scaling


To keep output magnitude consistent:


scale = max_rank / max(r, 1)
scale = min(scale, 4.0)  # optional clamp




Orbital Controller (no thresholds)


Dynamic trajectory instead of static FSM:




Ascend → stress detected → increase rank


Hold → oscillation → stay


Descend → stable → decrease rank




Uses a stack to ensure symmetric return.



Stress signal


φ(t) = |loss - EMA(loss)| + 2.0 × max(0, loss - prev_loss)


Auto-calibrated thresholds:


t_stress = μ + 0.7σ

t_stable = max(μ - 0.3σ, 0)


Robust stats can be used to reduce noise.



Why it matters




avoids cold starts across rank changes


adapts capacity in real-time


works in black-box settings


O(1) overhead





Comparison




Property
Standard LoRA
AdaLoRA
Orbital LoRA




Rank control
Fixed
SVD
Stress


Control type
None
Open
Closed-loop


Transition cost
N/A
High
O(1)


Architecture
Single
Pruned
Nested


Black-box
Yes
No
Yes