Update README.md
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ Developed by Deep Learning Efficiency Research (DLER) team at NVIDIA Research.
|
|
| 12 |
- Fuse attention heads and SSM heads within the same layer, offering parallel and complementary processing of the same inputs
|
| 13 |
|
| 14 |
<div align="center">
|
| 15 |
-
<img src="https://huggingface.co/nvidia/Hymba-1.5B/resolve/main/images/module.png" alt="Hymba Module" width="600">
|
| 16 |
</div>
|
| 17 |
|
| 18 |
- Introduce meta tokens that are prepended to the input sequences and interact with all subsequent tokens, thus storing important information and alleviating the burden of "forced-to-attend" in attention
|
|
@@ -20,7 +20,7 @@ Developed by Deep Learning Efficiency Research (DLER) team at NVIDIA Research.
|
|
| 20 |
- Integrate with cross-layer KV sharing and global-local attention to further boost memory and computation efficiency
|
| 21 |
|
| 22 |
<div align="center">
|
| 23 |
-
<img src="https://huggingface.co/nvidia/Hymba-1.5B/resolve/main/images/macro_arch.png" alt="Hymba Model" width="600">
|
| 24 |
</div>
|
| 25 |
|
| 26 |
|
|
@@ -32,7 +32,7 @@ Developed by Deep Learning Efficiency Research (DLER) team at NVIDIA Research.
|
|
| 32 |
</div>
|
| 33 |
|
| 34 |
|
| 35 |
-
- Hymba-1.5B-Instruct: Outperform
|
| 36 |
|
| 37 |
|
| 38 |
<div align="center">
|
|
|
|
| 12 |
- Fuse attention heads and SSM heads within the same layer, offering parallel and complementary processing of the same inputs
|
| 13 |
|
| 14 |
<div align="center">
|
| 15 |
+
<img src="https://huggingface.co/nvidia/Hymba-1.5B-Instruct/resolve/main/images/module.png" alt="Hymba Module" width="600">
|
| 16 |
</div>
|
| 17 |
|
| 18 |
- Introduce meta tokens that are prepended to the input sequences and interact with all subsequent tokens, thus storing important information and alleviating the burden of "forced-to-attend" in attention
|
|
|
|
| 20 |
- Integrate with cross-layer KV sharing and global-local attention to further boost memory and computation efficiency
|
| 21 |
|
| 22 |
<div align="center">
|
| 23 |
+
<img src="https://huggingface.co/nvidia/Hymba-1.5B-Instruct/resolve/main/images/macro_arch.png" alt="Hymba Model" width="600">
|
| 24 |
</div>
|
| 25 |
|
| 26 |
|
|
|
|
| 32 |
</div>
|
| 33 |
|
| 34 |
|
| 35 |
+
- Hymba-1.5B-Instruct: Outperform SOTA small LMs.
|
| 36 |
|
| 37 |
|
| 38 |
<div align="center">
|