Update README.md
Browse files
README.md
CHANGED
|
@@ -128,23 +128,35 @@ SVD information with the attention alignment spectrum.
|
|
| 128 |
|
| 129 |
# What Needs To Work
|
| 130 |
**Using MLP will reach fair accuracy and not use CONV or TRANSFORMERS.**
|
|
|
|
| 131 |
I have seen **around 60% on cifar100** with no traditional encoders, but the system was crutching the M_path to fill the gaps after enough epochs of the SVD path.
|
| 132 |
This structure is under the microscope now.
|
| 133 |
|
| 134 |
Instability allows SGD optimization to heavily benefit some image tasks while it fails completely on text tasks.
|
| 135 |
|
| 136 |
**Out Projection SUVt tokens are iffy**
|
|
|
|
| 137 |
The out projection is an MLP multiscale projection that took a while to set up, and it produces approximate transformer QKV with useful SUVt tokens downstream.
|
| 138 |
|
| 139 |
**Many activations corrupt geometry**
|
|
|
|
| 140 |
They are in there for experimentation. Feel free to experiment.
|
| 141 |
|
| 142 |
**without the expanded triton core spectrum larger systems suffer with triton**
|
|
|
|
| 143 |
Claude code is having trouble with this one as a full task, I'll need to build it in pieces. I've had OpenClaw working on it but the outcome
|
| 144 |
isn't looking good. The 4x4 and 5x4 won't converge, while the 6x6 crashes the system entirely instead of building it.
|
| 145 |
|
| 146 |
I'll need to wait for a fix for claude code, this is a known issue apparently.
|
| 147 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
|
| 149 |
## Additionally
|
| 150 |
|
|
|
|
| 128 |
|
| 129 |
# What Needs To Work
|
| 130 |
**Using MLP will reach fair accuracy and not use CONV or TRANSFORMERS.**
|
| 131 |
+
|
| 132 |
I have seen **around 60% on cifar100** with no traditional encoders, but the system was crutching the M_path to fill the gaps after enough epochs of the SVD path.
|
| 133 |
This structure is under the microscope now.
|
| 134 |
|
| 135 |
Instability allows SGD optimization to heavily benefit some image tasks while it fails completely on text tasks.
|
| 136 |
|
| 137 |
**Out Projection SUVt tokens are iffy**
|
| 138 |
+
|
| 139 |
The out projection is an MLP multiscale projection that took a while to set up, and it produces approximate transformer QKV with useful SUVt tokens downstream.
|
| 140 |
|
| 141 |
**Many activations corrupt geometry**
|
| 142 |
+
|
| 143 |
They are in there for experimentation. Feel free to experiment.
|
| 144 |
|
| 145 |
**without the expanded triton core spectrum larger systems suffer with triton**
|
| 146 |
+
|
| 147 |
Claude code is having trouble with this one as a full task, I'll need to build it in pieces. I've had OpenClaw working on it but the outcome
|
| 148 |
isn't looking good. The 4x4 and 5x4 won't converge, while the 6x6 crashes the system entirely instead of building it.
|
| 149 |
|
| 150 |
I'll need to wait for a fix for claude code, this is a known issue apparently.
|
| 151 |
|
| 152 |
+
**Magnitudes are WILDLY hard to control**
|
| 153 |
+
|
| 154 |
+
High magnitudes and complexity associations with those must be controlled. The vocabulary spectrum of a token is widely diverse, so the noise generated
|
| 155 |
+
from large and small magnitudes is very difficult to curate for. I'm debugging this task as I progress, and the dominant shape learning corruption is
|
| 156 |
+
magnitude differentiation currently.
|
| 157 |
+
|
| 158 |
+
I have some potentials and this is my current direction.
|
| 159 |
+
|
| 160 |
|
| 161 |
## Additionally
|
| 162 |
|