---
base_model:
- LiquidAI/LFM2-1.2B
tags:
- text-generation-inference
- transformers
- unsloth
- lfm2
license: apache-2.0
language:
- en
datasets:
- MegaScience/MegaScience
---


# 🧪 MegaSciMoE‑1.2B

<img src="banner.png" width="800" />

**A lightweight open MoE language model focused on science reasoning and instruction-following tasks.**
Fine-tuned using Unsloth on top of LiquidAI’s LFM2‑1.2B architecture.



---

## 🔬 Model Overview

* **Model Type:** Sparse Mixture-of-Experts (MoE)
* **Base:** LiquidAI LFM2‑1.2B
* **Fine-tuned with:** [Unsloth](https://unsloth.ai) LoRA
* **Focus Areas:** Physics, Chemistry, Biology, Scientific QA
* **Parameter Efficiency:** Only 2 experts active (\~250M active params)
* **Design Goal:** Run efficiently on consumer GPUs and edge devices

---

## 🧠 Capabilities

* Respond to structured science-related instructions
* Generate concise, factual explanations in STEM topics
* Follow multiturn question-answer patterns
* Summarize technical descriptions in plain language

---

## 🚀 Example Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("yasserrmd/MegaSciMoE-1.2B", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("yasserrmd/MegaSciMoE-1.2B")

prompt = "What is the difference between an ionic and covalent bond?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## 📚 Training Data

MegaSciMoE-1.2B was fine-tuned on a blend of:

* Multiturn scientific dialogues
* Instruction-based question answering
* Simulated domain-specific prompts in physics, chemistry, and biology
* Scientific corpus extracted from open datasets

---

## 📎 Sample Prompts

```text
Q: What is a neutron made of?
A: A neutron is composed of three quarks (two down quarks and one up quark)...

---

Q: Define osmosis in simple terms.
A: Osmosis is the movement of water across a membrane...

---

Q: What is the role of mitochondria?
A: Mitochondria are the powerhouses of the cell...
```

---

## 🧪 Benchmark Intent

Planned evaluations on:

* ARC-Challenge
* PubMedQA
* ScienceQA
  (*Note: No math-based datasets included in this build*)

---

## ⚙️ Technical Details

* LoRA fine-tuning with Unsloth on 1.2B MoE model
* Memory-efficient inference (fits on T4, L4, RTX 3060, even CPU)
* GGUF and FP16 variants to be released

---

## 📄 License

Apache 2.0 — free to use and modify with attribution. Ensure upstream license compatibility before commercial use.

---

## 🤝 Acknowledgements

* Thanks to [Unsloth](https://unsloth.ai), [LiquidAI](https://huggingface.co/liquidrock), and [Hugging Face](https://huggingface.co) for enabling this build
* Visuals rendered with scientific prompts and surreal-futuristic style




[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)