🧪 FrankenMoE — Proof of Concept (NOT production)

This is a technical experiment, not a useful model.

⚠️ Important Warning

This repository documents a proof-of-concept MoE pipeline. The model quality is NOT good — it produces incoherent / random outputs because:

1. The router uses (no training)
2. Experts were fine-tuned with only ~5K samples each
3. Base model is only Qwen2.5-1.5B-Instruct

Do NOT use this model for anything serious. It exists purely to demonstrate that the FrankenMoE pipeline can be built end-to-end.

What We Actually Built

A working MoE pipeline from dense LoRA experts → GGUF:

Qwen2.5-1.5B-Instruct (base)
  ├── Expert 0: Coding (LoRA fine-tuned)
  ├── Expert 1: Math (LoRA fine-tuned)  
  └── Shared Expert: Base model

Key Technical Discoveries

Discovery	Detail
mergekit 0.1.4 bug	param incompatible with transformers >= 4.40 — must patch
QwenMoE requirements	Exactly 1 shared expert + 2^n routed experts (2, 4, 8)
Tied embeddings fix	Qwen2.5 uses tied embeddings → must clone → before GGUF conversion, set
LoRA must be merged	Adapters must be before MoE assembly

Repository Structure

📦 frankenmoe_moe_v2-F16.gguf  — MoE GGUF (fixed, has output.weight)
📁 moe_full/                   — Full safetensors model
📁 coding/ math/ chat/         — Individual dense experts (LoRA + GGUF)
📄 FrankenMoE_Academic_Paper.pdf — Research paper
🐍 simple_router.py            — Keyword-based router (functional alternative)

Quick Test

wget https://huggingface.co/hotdogs/frankenmoe/resolve/main/frankenmoe_moe_v2-F16.gguf
llama-cli -m frankenmoe_moe_v2-F16.gguf -p "Write a Python function"
# Output: Random/incoherent — this is expected! See warning above.

Future: Real Model

The pipeline will be re-run with:

• Larger base model (Qwen2.5-7B/14B)
• Trained router (classification loss)
• More training data per domain
• 4 experts for proper 2^n routing

Stay tuned — the real model is coming.

---

Built by UKA 🇹🇭 | May 2026