aifeifei798 commited on
Commit
970db23
·
verified ·
1 Parent(s): c66f72f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +26 -3
README.md CHANGED
@@ -79,6 +79,18 @@ To understand *why* **Fragmented Training (FT)** accelerates inference and sharp
79
 
80
  ---
81
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  ### 📊 Empirical Proof: The 3-Phase Denoising Mechanism
83
 
84
  The layer-by-layer audit revealed a clear **3-Phase Transformer Dynamics** introduced by the FT LoRA:
@@ -95,7 +107,7 @@ The layer-by-layer audit revealed a clear **3-Phase Transformer Dynamics** intro
95
 
96
  #### 1. Phase 1: Middle-Layer Noise Pruning (L-10 to L-14)
97
  * **Observed Phenomenon:** A continuous 5-layer cluster of **Amplified Destructive Interference ($\cos < 0$)**.
98
- * **Key Metric:** At **L-12**, $\cos(h, \Delta h)$ drops from `-0.1229` (Base) to **`-0.1680`** (FT), a net change of **$\Delta \cos = -0.0452$**.
99
  * **Mechanism:** The FT LoRA acts as a **subtractive denoising filter**, actively injecting anti-phase vectors to cancel out scrambled token noise introduced in early layers.
100
 
101
  #### 2. Phase 2: Off-Target Path Suppression (L-19)
@@ -105,14 +117,14 @@ The layer-by-layer audit revealed a clear **3-Phase Transformer Dynamics** intro
105
 
106
  #### 3. Phase 3: Frictionless Target Acceleration (L-23 to L-33)
107
  * **Observed Phenomenon:** Decisive positive alignment toward the target token.
108
- * **Key Metric:** At **L-33**, cosine alignment reaches **$0.5018$** in the FT model (vs. $0.4352$ in Base).
109
  * **Mechanism:** Because noise was pruned in middle layers, the deep layers experience zero-friction acceleration toward the target token. This drives **Confidence Sharpening** and explains the **29.61% inference speedup**.
110
 
111
  ---
112
 
113
  ### 📈 Layer-by-Layer Audit Data (Selected Key Layers)
114
 
115
- | Layer | Base Model $\cos(h, \Delta h)$ | **FT Model $\cos(h, \Delta h)$** | $\Delta \cos$ (Diff) | Tensor Interference Dynamics |
116
  | :--- | :--- | :--- | :--- | :--- |
117
  | **L-10** | `-0.3186` (Subtractive) | **`-0.3297`** (Subtractive) | `-0.0111` | ⬇️ Amplified Subtraction |
118
  | **L-11** | `-0.1584` (Subtractive) | **`-0.1797`** (Subtractive) | `-0.0213` | ⬇️ Amplified Subtraction |
@@ -122,6 +134,8 @@ The layer-by-layer audit revealed a clear **3-Phase Transformer Dynamics** intro
122
  | **L-19** | `+0.1779` (Additive) | **`+0.0425`** (Additive) | **`-0.1353`** | ⬇️ Massive Noise Path Suppression |
123
  | **L-33** | `+0.4352` (Additive) | **`+0.5018`** (Additive) | **`+0.0665`** | 🚀 Frictionless Target Push |
124
 
 
 
125
  ### 🛠️ Reproduce the Audit Locally
126
 
127
  You can verify these exact residual stream vector measurements using our open-source PyTorch audit script:
@@ -131,6 +145,15 @@ You can verify these exact residual stream vector measurements using our open-so
131
  python compare_interference.py
132
  ```
133
 
 
 
 
 
 
 
 
 
 
134
  ---
135
 
136
  ## 📄 Abstract
 
79
 
80
  ---
81
 
82
+ ## 🔬 Mechanistic Verification: Residual Stream Vector Interference
83
+
84
+ > **"We didn't just observe the 30% speedup—we measured the exact layer-by-layer tensor dynamics that caused it."**
85
+
86
+ To understand *why* **Fragmented Training (FT)** accelerates inference and sharpens logic, we registered PyTorch forward hooks across all 36 layers of `Qwen3-4B`. We audited the residual stream dynamics:
87
+
88
+ $$h_{l+1} = h_l + \Delta h_l$$
89
+
90
+ We measured the **Cosine Similarity** $\cos(h\_l, \Delta h\_l)$ between the accumulated hidden state $h\_l$ and the layer-wise update $\Delta h\_l$.
91
+
92
+ ---
93
+
94
  ### 📊 Empirical Proof: The 3-Phase Denoising Mechanism
95
 
96
  The layer-by-layer audit revealed a clear **3-Phase Transformer Dynamics** introduced by the FT LoRA:
 
107
 
108
  #### 1. Phase 1: Middle-Layer Noise Pruning (L-10 to L-14)
109
  * **Observed Phenomenon:** A continuous 5-layer cluster of **Amplified Destructive Interference ($\cos < 0$)**.
110
+ * **Key Metric:** At **L-12**, $\cos(h\_l, \Delta h\_l)$ drops from `-0.1229` (Base) to **`-0.1680`** (FT), a net change of **$\Delta \cos = -0.0452$**.
111
  * **Mechanism:** The FT LoRA acts as a **subtractive denoising filter**, actively injecting anti-phase vectors to cancel out scrambled token noise introduced in early layers.
112
 
113
  #### 2. Phase 2: Off-Target Path Suppression (L-19)
 
117
 
118
  #### 3. Phase 3: Frictionless Target Acceleration (L-23 to L-33)
119
  * **Observed Phenomenon:** Decisive positive alignment toward the target token.
120
+ * **Key Metric:** At **L-33**, cosine alignment reaches **`0.5018`** in the FT model (vs. `0.4352` in Base).
121
  * **Mechanism:** Because noise was pruned in middle layers, the deep layers experience zero-friction acceleration toward the target token. This drives **Confidence Sharpening** and explains the **29.61% inference speedup**.
122
 
123
  ---
124
 
125
  ### 📈 Layer-by-Layer Audit Data (Selected Key Layers)
126
 
127
+ | Layer | Base Model `cos(h, Δh)` | **FT Model `cos(h, Δh)`** | cos` (Diff) | Tensor Interference Dynamics |
128
  | :--- | :--- | :--- | :--- | :--- |
129
  | **L-10** | `-0.3186` (Subtractive) | **`-0.3297`** (Subtractive) | `-0.0111` | ⬇️ Amplified Subtraction |
130
  | **L-11** | `-0.1584` (Subtractive) | **`-0.1797`** (Subtractive) | `-0.0213` | ⬇️ Amplified Subtraction |
 
134
  | **L-19** | `+0.1779` (Additive) | **`+0.0425`** (Additive) | **`-0.1353`** | ⬇️ Massive Noise Path Suppression |
135
  | **L-33** | `+0.4352` (Additive) | **`+0.5018`** (Additive) | **`+0.0665`** | 🚀 Frictionless Target Push |
136
 
137
+ ---
138
+
139
  ### 🛠️ Reproduce the Audit Locally
140
 
141
  You can verify these exact residual stream vector measurements using our open-source PyTorch audit script:
 
145
  python compare_interference.py
146
  ```
147
 
148
+ ```python
149
+ # Core measurement logic in compare_interference.py:
150
+ delta_h = h_next - h_l # Layer update vector
151
+ cos_sim = F.cosine_similarity(h_l, delta_h, dim=0).item()
152
+
153
+ # cos_sim < 0 => Destructive Interference (Subtractive Denoising)
154
+ # cos_sim > 0 => Constructive Alignment (Target Stacking)
155
+ ```
156
+
157
  ---
158
 
159
  ## 📄 Abstract