Update README.md
Browse files
README.md
CHANGED
|
@@ -4,6 +4,9 @@ datasets:
|
|
| 4 |
language:
|
| 5 |
- en
|
| 6 |
---
|
|
|
|
|
|
|
|
|
|
| 7 |
GLORT2 (GLORT2 Low Rank Transformer Transformer) is a transformer model where every single linear layer is another smaller transformer model. I combined qkv into one operation which means one transformer instead of 3 to save on parameters, I played w using a transformer on the embeddings but it wasnt .. great, it's 768 dim 10 layers w/ 384 dim 1 layer as the replacements for linear layers (besides embed and lm head)
|
| 8 |
|
| 9 |
also sorry I just realized theres some residual from where I copied the model code from in my own projects that includes some "expanded lm head size" stuff just ignore that if you're looking at the config and code this isn't a serious project so I don't care too much that it's there
|
|
|
|
| 4 |
language:
|
| 5 |
- en
|
| 6 |
---
|
| 7 |
+
|
| 8 |
+
# broken, let me reimplement and train
|
| 9 |
+
|
| 10 |
GLORT2 (GLORT2 Low Rank Transformer Transformer) is a transformer model where every single linear layer is another smaller transformer model. I combined qkv into one operation which means one transformer instead of 3 to save on parameters, I played w using a transformer on the embeddings but it wasnt .. great, it's 768 dim 10 layers w/ 384 dim 1 layer as the replacements for linear layers (besides embed and lm head)
|
| 11 |
|
| 12 |
also sorry I just realized theres some residual from where I copied the model code from in my own projects that includes some "expanded lm head size" stuff just ignore that if you're looking at the config and code this isn't a serious project so I don't care too much that it's there
|