Update README.md
Browse files
README.md
CHANGED
|
@@ -18,6 +18,46 @@ Until now this has proven valuable, and now that it's become a hinderance a new
|
|
| 18 |
# Experiment 1: Retune AdamW Directly
|
| 19 |
I'll attempt to tweak AdamW specifically to not destroy the geometric shape, disabling weight_decay from this point onward.
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
---------------------------------------------------------------------------------------------------
|
| 23 |
# Large Experimental Conclusion 1:
|
|
|
|
| 18 |
# Experiment 1: Retune AdamW Directly
|
| 19 |
I'll attempt to tweak AdamW specifically to not destroy the geometric shape, disabling weight_decay from this point onward.
|
| 20 |
|
| 21 |
+
The outcomes have shown there isn't much beyond tuning specifics to tweak this particular classifier. Introducing more anchors helps, and with that the dims can be reduced.
|
| 22 |
+
Essentially the anchors are capacity tuning forks in this variation rather than utilities, which is fine. We can allocate them to a student.
|
| 23 |
+
|
| 24 |
+
```
|
| 25 |
+
=================================================================
|
| 26 |
+
SWEEP RESULTS
|
| 27 |
+
=================================================================
|
| 28 |
+
|
| 29 |
+
Config v_acc t_acc gap cv Δcv eq_std poly curve star struct
|
| 30 |
+
------------------------------------------------------------------------------------------
|
| 31 |
+
raw_adam 0.617 0.681 +0.064 1.3917 +1.1917 0.4075 0.39 0.75 0.86 0.61
|
| 32 |
+
proven 0.722 0.706 -0.016 1.3629 +1.1629 0.4157 0.45 0.99 0.93 0.71
|
| 33 |
+
+spread 0.669 0.686 +0.017 1.4491 +1.2491 0.4212 0.41 0.98 0.71 0.72
|
| 34 |
+
+entropy 0.674 0.711 +0.037 1.4945 +1.2945 0.4237 0.42 0.97 0.70 0.74
|
| 35 |
+
+ortho 0.695 0.690 -0.005 1.3454 +1.1454 0.4171 0.40 0.99 0.85 0.72
|
| 36 |
+
+cluster 0.701 0.717 +0.016 1.3034 +1.1034 0.4131 0.44 0.93 0.91 0.70
|
| 37 |
+
+drift 0.709 0.698 -0.012 1.3480 +1.1480 0.4134 0.41 1.00 0.91 0.72
|
| 38 |
+
+spr+ort 0.723 0.698 -0.025 1.3881 +1.1881 0.4224 0.46 0.97 0.94 0.71
|
| 39 |
+
+all_micro 0.694 0.700 +0.007 1.5181 +1.3181 0.4077 0.40 0.97 0.85 0.72
|
| 40 |
+
|
| 41 |
+
Best accuracy: +spr+ort (val_acc=0.723)
|
| 42 |
+
Best structure: +entropy (struct=0.737)
|
| 43 |
+
Closest to CV=0.2: +cluster (cv=1.3034, Δ=+1.1034)
|
| 44 |
+
Most equidistant: raw_adam (equi_std=0.4075)
|
| 45 |
+
Most stable CV: raw_adam (cv_std=0.1734)
|
| 46 |
+
|
| 47 |
+
=================================================================
|
| 48 |
+
DONE
|
| 49 |
+
=================================================================
|
| 50 |
+
```
|
| 51 |
+
The outcomes show that we can definitely impact the outcome and the deviation of the system will conform to an entirely new spectrum of CV currently unoccupied.
|
| 52 |
+
|
| 53 |
+
There's a lot to unpack here, and I think the biggest most critical piece to unpack is a hyperparameter about controlling where on the latent spectrum
|
| 54 |
+
you want your model's continuum to exist within.
|
| 55 |
+
|
| 56 |
+
# Experiment 2: Teacher/Student hierarchy
|
| 57 |
+
I've ran plenty of genetic experiments, a single student anchored from a teacher should provide a more robust sweep
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
|
| 61 |
|
| 62 |
---------------------------------------------------------------------------------------------------
|
| 63 |
# Large Experimental Conclusion 1:
|