| | --- |
| | license: apache-2.0 |
| | --- |
| | |
| | Base Model: Just Merged ~ No Training Gates After Merge |
| |
|
| | ### Model Overview |
| |
|
| | I have developed a Mixture of Experts (MoE) architecture with two always-active experts designed to work together for Python instruction tuning. Each expert possesses a distinct skill: |
| |
|
| | - **Expert 1**: Specializes in generating Mermaid diagrams, primarily from Python code, which requires a deep understanding of code structures and logic. |
| | - **Expert 2**: Focuses on strict context obedience, ensuring that the model only generates outputs based on the provided instructions. |
| |
|
| | ### Why Always-Active MoE is Optimal |
| |
|
| | In this model, both experts are always active for each token, allowing them to complement each other: |
| |
|
| | - **Expert 1’s competence in Python structures** enhances the model's ability to generate correct and structured Python code. |
| | - **Expert 2’s context obedience** ensures that the output remains aligned with the user’s instructions, preventing unnecessary or irrelevant outputs, such as Mermaid diagrams, unless explicitly requested. |
| |
|
| | This setup allows me to efficiently train the model for Python instruction following. By leveraging both experts simultaneously, I ensure that the model generates syntactically correct Python code while strictly adhering to user prompts. |
| |
|
| |
|
