added expansion image
Browse files
README.md
CHANGED
|
@@ -57,9 +57,9 @@ Getting pros of both searches made sense and that gave rise to interest in learn
|
|
| 57 |
|
| 58 |
4. What a Sparse model learns ?
|
| 59 |
|
| 60 |
-
The model learns to project it's learned dense representations over a MLM head to give a vocabulary distribution.
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
</details>
|
| 65 |
|
|
|
|
| 57 |
|
| 58 |
4. What a Sparse model learns ?
|
| 59 |
|
| 60 |
+
The model learns to project it's learned dense representations over a MLM head to give a vocabulary distribution. Which is just to say the model can do automatic token expansion. (Image courtesy of pinecone)
|
| 61 |
|
| 62 |
+
<img src="./expansion.png" width=650 height=500/>
|
| 63 |
|
| 64 |
</details>
|
| 65 |
|