| license: apache-2.0 | |
| tags: | |
| - prime-rl | |
| - moe | |
| - test-model | |
| library_name: transformers | |
| <div align="center"> | |
| <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/61e020e4a343274bb132e138/H2mcdPRWtl4iKLd-OYYBc.jpeg" width="200"/> | |
| </div> | |
| # glm4-moe-tiny | |
| A small (~543M parameter) GLM-4 MoE model for testing only. It is generally compatible with vLLM and HuggingFace Transformers but is meant to be used with [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl). | |
| Fine-tuned on [PrimeIntellect/Reverse-Text-SFT](https://huggingface.co/datasets/PrimeIntellect/Reverse-Text-SFT) to provide a non-trivial distribution for KL divergence during RL. | |
| ## Quick Start | |
| ```bash | |
| uv run rl @ configs/ci/integration/rl_moe/glm4_moe.toml | |
| ``` | |
| See the [Testing MoE at Small Scale](https://github.com/PrimeIntellect-ai/prime-rl/blob/main/docs/testing-moe-at-small-scale.md) guide for full instructions. | |
| ## Model Details | |
| | Parameter | Value | | |
| |-----------|-------| | |
| | Hidden size | 1024 | | |
| | Layers | 24 | | |
| | Experts | 8 | | |
| | Active experts | 4 | | |
| | Parameters | ~543M | | |
| ## Links | |
| - [prime-rl](https://github.com/PrimeIntellect-ai/prime-rl) - RL training framework | |
| - [PrimeIntellect](https://www.primeintellect.ai/) - Building infrastructure for decentralized AI | |