| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - mixture-of-experts | |
| - moe | |
| - corm | |
| # CoRM | |
| Checkpoints for the paper [Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts](https://arxiv.org/abs/2609.01100). | |
| | Model | Active Params | Total Params | Routing | Link | | |
| |---|---|---|---|---| | |
| | CoRM-182M | 182M | 777M | Top-1 | [link](https://huggingface.co/ilsp/CoRM-182M-top1) | | |
| | CoRM-182M | 266M | 777M | Top-2 | [link](https://huggingface.co/ilsp/CoRM-182M-top2) | | |
| | CoRM-469M | 469M | 2.58B | Top-1 | [link](https://huggingface.co/ilsp/CoRM-469M-top1) | | |
| ## Usage | |
| These models use custom modeling code, so `trust_remote_code=True` is required. | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_id = "ilsp/CoRM-182M-top1" | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True) | |
| inputs = tokenizer("The capital of Greece is", return_tensors="pt") | |
| outputs = model.generate(**inputs, max_new_tokens=32) | |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) | |
| ``` | |
| ## Citation | |
| <!-- TODO: add BibTeX. --> | |