Chat-UniVi
/

MoE-Plus-Plus-7B

@@ -9,13 +9,6 @@ license: apache-2.0
 ## ⚡ Overview
 We introduce three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively. Moreover, we leverage gating residuals, enabling each token to consider the pathway taken in the previous layer when selecting the appropriate experts.
-<div align=center>
-<img src="figures/fig2.png" width="800px">
-</div>
-<div align=center>
-<img src="figures/fig3.png" width="800px">
-</div>
 ### Download URL
 <div align=center>
@@ -31,42 +24,19 @@ We introduce three types of zero-computation experts: the zero expert, copy expe
 ### 💡 Low Computing Overhead
 For an MoE++ model, its computational complexity is always **less than** that of MoE models with the same number of parameters.
-<div align=center>
-<img src="figures/fig4.png" width="800px">
-</div>
 ### 🔥 High Performance & High Throughput
 Extensive experimental results demonstrate that MoE++ achieves better performance while delivering 1.1~2.1x expert forward throughput compared to a vanilla MoE model of the same size, which lays a solid foundation for developing advanced and efficient MoE-related models.
-<div align=center>
-<img src="figures/fig7.png" width="800px">
-</div>
 ### 🤗 Deployment Friendly
 Given that zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU, eliminating the significant communication overhead and expert load imbalance associated with FFN experts distributed across different GPUs.
-## 🚀 Main Results
-### Comparisons between MoE++ and Vanilla MoE Models
-<div align=center>
-<img src="figures/fig9.png" width="800px">
-</div>
-### Comparisons to LLMs of Equivalent Activated Parameters
-<div align=center>
-<img src="figures/fig10.png" width="800px">
-</div>
 ## 😍 Why is MoE++ better than MoE?
 ### Flexible Computation Allocation
 MoE++ allows simple tokens to utilize fewer FFN experts, freeing up more FFN experts to focus on challenging tokens. This results in both **Reduced Computation** and **Enhanced Performance**.
-<div align=center>
-<img src="figures/fig5.png" width="800px">
-</div>
 * Verbs tend to activate a large number of FFN experts. For example, the verb "touch" activates an average of 1.77 FFN experts across all layers, approaching the upper limit of 2. This likely occurs because verbs often convey rich semantic information and frequently interact with nouns to form more complex semantic structures.
 * Nouns typically activate a moderate number of FFN experts, with most nouns averaging between 1.5 and 1.7 FFN expert activations.
 * Simple tokens with little semantic tend to activate a small number of FFN experts. For example, word fragments, such as "pper" and "ather", usually activate fewer than 1.5 FFN experts.
@@ -76,10 +46,6 @@ These findings confirm that MoE++ allows simple tokens to utilize fewer FFN expe
 ### Stable Routing
 Gating residuals effectively establish connections between different MoE++ layers and reduce the variance of routing scores. Meanwhile, the gating residuals do not change the mean and range of values of the routing scores. Consequently, gating residuals contribute to the stable routing of heterogeneous expert architectures in MoE++.
-<div align=center>
-<img src="figures/fig6.png" width="800px">
-</div>
 ## 🤖 API for Base Model Inference
 If you want to load the model from the model hub on Hugging Face or on local, you can use the following code snippets.

 ## ⚡ Overview
 We introduce three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively. Moreover, we leverage gating residuals, enabling each token to consider the pathway taken in the previous layer when selecting the appropriate experts.
 ### Download URL
 <div align=center>
 ### 💡 Low Computing Overhead
 For an MoE++ model, its computational complexity is always **less than** that of MoE models with the same number of parameters.
 ### 🔥 High Performance & High Throughput
 Extensive experimental results demonstrate that MoE++ achieves better performance while delivering 1.1~2.1x expert forward throughput compared to a vanilla MoE model of the same size, which lays a solid foundation for developing advanced and efficient MoE-related models.
 ### 🤗 Deployment Friendly
 Given that zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU, eliminating the significant communication overhead and expert load imbalance associated with FFN experts distributed across different GPUs.
 ## 😍 Why is MoE++ better than MoE?
 ### Flexible Computation Allocation
 MoE++ allows simple tokens to utilize fewer FFN experts, freeing up more FFN experts to focus on challenging tokens. This results in both **Reduced Computation** and **Enhanced Performance**.
 * Verbs tend to activate a large number of FFN experts. For example, the verb "touch" activates an average of 1.77 FFN experts across all layers, approaching the upper limit of 2. This likely occurs because verbs often convey rich semantic information and frequently interact with nouns to form more complex semantic structures.
 * Nouns typically activate a moderate number of FFN experts, with most nouns averaging between 1.5 and 1.7 FFN expert activations.
 * Simple tokens with little semantic tend to activate a small number of FFN experts. For example, word fragments, such as "pper" and "ather", usually activate fewer than 1.5 FFN experts.
 ### Stable Routing
 Gating residuals effectively establish connections between different MoE++ layers and reduce the variance of routing scores. Meanwhile, the gating residuals do not change the mean and range of values of the routing scores. Consequently, gating residuals contribute to the stable routing of heterogeneous expert architectures in MoE++.
 ## 🤖 API for Base Model Inference
 If you want to load the model from the model hub on Hugging Face or on local, you can use the following code snippets.