Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,13 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
---
|
| 4 |
+
This is a repo specifically dedicated to analyzing embedding spaces for perfect geometric symmetrical representation.
|
| 5 |
+
|
| 6 |
+
We're going to get to the bottom of why embeddings are so symmetrically sample-sturdy and can be assessed with CV deterministically on random,
|
| 7 |
+
and with that compare the deconstructive nature of collapsing embeddings into more unified compacted spaces for MHA heads.
|
| 8 |
+
|
| 9 |
+
The bands show; D16 and D38 are directly CV sample-capable for volume validity, 64 dim being the utmost upper bounds for distance before
|
| 10 |
+
full degredation and breakdown using LINALG.det
|
| 11 |
+
|
| 12 |
+
Generally the rule of thumb for MHA head counts is 64 dims per head and scale from there. I'm thinking there may be a direct causal response
|
| 13 |
+
to this matmul scaling principle and the rules contained within pytorch in general.
|