| --- |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - en |
| base_model: |
| - HuggingFaceTB/SmolLM2-360M-Instruct |
| - prithivMLmods/SmolLM2-CoT-360M |
| - summerstars/SolaraV2-coder-0517 |
| - Fu01978/SmolLM2-360M-Instruct-Heretic |
| pipeline_tag: text-generation |
| tags: |
| - mixture-of-experts |
| - moe |
| - mergekit |
| - smollm2 |
| - instruct |
| - reasoning |
| - code |
| - math |
| - creative |
| - merge |
| --- |
| |
| # SmolMoE-4x360M-Instruct |
|
|
| A Mixture-of-Experts model |
| built by merging four SmolLM2-360M fine-tunes |
| using [mergekit](https://github.com/arcee-ai/mergekit). |
| Each expert specializes |
| in a distinct domain, |
| with 2 experts active per token |
| (~720M active parameters |
| per forward pass |
| out of ~1.4B total). |
|
|
| ## Experts |
| |
| | # | Model | Specialization | |
| |---|-------|---------------| |
| | E0 | [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) | General knowledge, factual Q&A | |
| | E1 | [prithivMLmods/SmolLM2-CoT-360M](https://huggingface.co/prithivMLmods/SmolLM2-CoT-360M) | Chain-of-thought reasoning, logic | |
| | E2 | [summerstars/SolaraV2-coder-0517](https://huggingface.co/summerstars/SolaraV2-coder-0517) | Code generation, mathematics | |
| | E3 | [Fu01978/SmolLM2-360M-Instruct-Heretic](https://huggingface.co/Fu01978/SmolLM2-360M-Instruct-Heretic) | Creative writing, expressive language | |
|
|
| ## Architecture |
| |
| - **Base architecture:** Mixtral-style MoE (via mergekit) |
| - **Total experts:** 4 |
| - **Active experts per token:** 2 |
| - **Gate mode:** `hidden` (router trained on real hidden states), with subsequent router fine-tuning |
| - **Active parameters per token:** ~720M |
|
|
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "Fu01978/SmolMoE-4x360M-Instruct", |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| trust_remote_code=True, |
| ) |
| tokenizer = AutoTokenizer.from_pretrained("Fu01978/SmolMoE-4x360M-Instruct") |
| |
| messages = [{"role": "user", "content": "Implement a binary search in Python."}] |
| formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(formatted, return_tensors="pt").to(model.device) |
| |
| output = model.generate(**inputs, max_new_tokens=256, temperature=0.2, do_sample=True) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Limitations |
|
|
| - General factual accuracy |
| is imperfect — the model can hallucinate |
| details on knowledge questions |
| - At 360M per expert, |
| complex multi-step reasoning |
| has limits |
| - E0 (General) |
| is the weakest |
| router target |
| due to weight similarity |
| with E3 (Heretic), |
| which is a direct fine-tune |
| of the same base |
|
|
| ## Created With |
| |
| - [mergekit](https://github.com/arcee-ai/mergekit) |
| — MoE construction |
| - Kaggle Dual T4 GPUs |
