Akahsizrr/Mini-Whale-1-12B

Mini Whale 1 12B

A 12B-parameter Mixture-of-Experts model that fuses Qwen3-4B (host) with 260 coding experts extracted from DeepSeek-V4-Flash, fine-tuned with QLoRA and optimized for local inference on consumer GPUs (RTX 3060 12GB).

---

Table of Contents

1. What is this model?
2. Architecture (A to Z)
3. Training
4. Local Inference Guide
5. DSpark Speculative Decoding
6. Performance
7. File Structure
8. Citation

---

What is this model?

Mini Whale 1 12B is a fusion architecture that combines:

• Host model: Qwen3-4B (36 layers, hidden dim 2560, BF16) — handles language understanding, reasoning, and general text generation
• Coding experts: 260 SwiGLU experts from DeepSeek-V4-Flash (each ~25M params, hidden dim 4096) — specialized for code generation
• Bridge layers: Low-rank adapters (rank 7) that translate between Qwen3's 2560-dim representation space and DeepSeek's 4096-dim expert space
• Router: sqrt-softplus scoring with top-2 expert selection per token

The result is a model that thinks like Qwen3 but codes like DeepSeek — all while fitting in 12GB VRAM with 4-bit quantization.

---

Architecture (A to Z)

Overview

Input Tokens
    │
    ▼
┌─────────────────────────────────────────┐
│  Qwen3-4B Host (36 layers, hidden=2560) │
│                                         │
│  For each layer:                        │
│  1. Self-Attention (SDPA, 32 heads)     │
│  2. Host MLP (SwiGLU, 9728 intermediate)│
│  3. [Augmented layers only]:            │
│     a. Bridge In (2560 → 4096)          │
│     b. Router (sqrt-softplus, top-2)    │
│     c. Expert SwiGLU (2 of N experts)   │
│     d. Bridge Out (4096 → 2560)         │
│     e. RMSNorm + Sigmoid Gate           │
│     f. Residual Repair (rank-7)         │
│     g. Residual Addition                │
│  4. RMSNorm                             │
└─────────────────────────────────────────┘
    │
    ▼
LM Head (tied embeddings, 151936 vocab)
    │
    ▼
Output Logits → Token

Layer Structure (36 layers)

All 36 layers are augmented (have coding experts), but the number of experts per layer varies:

Layers	Expert Count	Description
0-8	1-8	Early layers: fewer experts, basic coding patterns
9	1	Minimal augmentation (dense-like)
10-28	2-13	Middle layers: moderate expert density
29-35	8-25	Deep layers: heavy expert density for complex code

Total experts across all layers: 303 expert instances (260 unique experts, some shared across layers).

Expert Computation

Each expert is a standard SwiGLU MLP:

expert_output = down_proj(
    clamp(silu(gate_proj(x)) * up_proj(x), -10.0, 10.0)
)

The clamp(-10, 10) is critical — it matches DeepSeek-V4-Flash's native swiglu_limit and prevents activation explosion across layers.

Bridge Layers

The bridge is the key innovation. It translates between two different representation spaces:

• bridge_in: Linear(2560, 4096) — maps Qwen3 hidden states to DeepSeek expert space
• bridge_out: Linear(4096, 2560) — maps expert outputs back to Qwen3 space

Both are low-rank (rank 7) to keep parameter count low and prevent the experts from overwhelming the host signal.

Router

logits = router.gate(hidden_states)                    # (tokens, num_experts)
scores = sqrt(clamp(softplus(logits), min=1e-6))      # sqrt-softplus scoring
topk_weights, topk_indices = scores.topk(2, dim=-1)   # top-2 selection
topk_weights = topk_weights / topk_weights.sum(dim=-1) # normalize

The sqrt-softplus scoring is smoother than softmax and prevents expert collapse (one expert dominating all tokens).

Residual Safety

The coding delta is gated and clamped to prevent it from disrupting the host's representation:

gate = sigmoid(coding_gate)          # learnable scalar, init -2.0 → ~12% gate
coding_delta = bridge_out(expert_output)
coding_delta = coding_norm(coding_delta)
coding_delta = coding_delta * gate

# Clamp to max_delta_ratio of host signal norm
scale = min(1.0, (host_norm * max_delta_ratio) / delta_norm)
coding_delta = coding_delta * scale

# Repair (rank-7 residual correction)
repair_delta = repair_up(repair_down(host_hidden))

# Final residual
output = host_hidden + coding_delta + repair_delta

This ensures the coding experts contribute to the output without dominating it. The host model's reasoning ability is preserved.

Key Config

Parameter	Value
Host hidden size	2560
Host intermediate size	9728
Host layers	36
Host attention heads	32 (Q) / 8 (KV)
Head dim	128
Expert hidden size	4096
Expert intermediate size	2048
Top-k experts	2
Bridge rank	7
Vocab size	151,936
Max position	40,960
RoPE theta	1,000,000
Total params	~12B
Quantized size (4-bit)	23.4 GB (BF16) / 8.9 GB (4-bit)

---

Training

Stage 1: Bridge + Router Training (QLoRA)

• Method: QLoRA (4-bit NF4 host + trainable LoRA adapters on attention + bridge layers)
• Trainable params: 19.5M (bridge_in, bridge_out, router.gate, repair, q/k/v/o_proj LoRA)
• Dataset: Coding instruction dataset (Python, JavaScript, TypeScript)
• Steps: 500
• Loss: 0.62 → 0.19
• Optimizer: AdamW, lr=2e-4, cosine schedule
• Hardware: RTX 3060 12GB (local)

Stage 2: Merge

LoRA adapters were merged into the base model using streaming tensor-by-tensor merge:

• 216 LoRA modules merged (B = A @ B for each adapter)
• 144 checkpoint overrides applied
• 1285 tensors copied unchanged
• Output: 23.4 GB BF16 model (5 shards)

Critical Runtime Fixes

Two fixes are required at inference time (applied automatically by the model code):

1. SwiGLU clamping (limit=10.0): DeepSeek-V4-Flash uses swiglu_limit=10.0 to clamp expert intermediate activations. Without this, outlier values grow exponentially across layers.

2. Router stability: softplus is clamped to min=1e-6 before sqrt() to prevent NaN gradients.

---

Local Inference Guide

Requirements

Component	Minimum	Recommended
GPU	RTX 3060 12GB	RTX 4070 16GB+
RAM	16GB	32GB
Python	3.10+	3.11+
CUDA	12.1+	12.6+
VRAM (4-bit)	8.9 GB	—
VRAM (4-bit + DSpark)	10.9 GB	—

Installation

# Create virtual environment
python -m venv fuse2-venv
fuse2-venv\Scripts\activate

# Install dependencies
pip install torch --index-url https://download.pytorch.org/whl/cu126
pip install transformers==5.14.1 bitsandbytes==0.49 peft==0.20 accelerate
pip install safetensors

Download the Model

pip install huggingface_hub
huggingface-cli download Akahsizrr/Mini-Whale-1-12B --local-dir ./mini-whale-1-12b

Quick Start: One-Liner 4-bit Loading (~5 tok/s)

The model has all runtime fixes baked into the model code (fuse2_model.py):

• SwiGLU clamping (limit=10.0) — built into SwiGLUExpert.forward
• Router stability (softplus clamp) — built into Fuse2Router.forward
• Meta param auto-init + norm dtype fix — handled by from_pretrained override

So you can load and generate with zero post-load patching:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL = "Akahsizrr/Mini-Whale-1-12B"

# 4-bit NF4 quantization (matches config.json quantization_config)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load model — runtime fixes applied automatically by from_pretrained override
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    trust_remote_code=True,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="sdpa",
)
model.eval()

# Tokenizer
tok = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)

# Generate
messages = [{"role": "user", "content": "Write a Python fizzbuzz."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tok(text, return_tensors="pt").input_ids.to(model.device)

with torch.inference_mode():
    out = model.generate(input_ids, max_new_tokens=512, do_sample=False,
                         repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

That's it — no manual SwiGLU patching, no meta device fixes, no norm dtype casting. The model code handles all of it.

Quick Start: Manual 4-bit Loading (~5 tok/s)

For more control over the loading process (e.g., selective quantization of attention vs MLP layers):

import torch, json, os, sys, gc
from transformers import AutoConfig, AutoTokenizer
from accelerate import init_empty_weights
from safetensors import safe_open
import bitsandbytes as bnb

MODEL_PATH = "./mini-whale-1-12b"
DEVICE = "cuda:0"
sys.path.insert(0, MODEL_PATH)
import fuse2_model_local

# Load config with SDPA attention
config = AutoConfig.from_pretrained(MODEL_PATH, trust_remote_code=True)
config._attn_implementation = "sdpa"

# Create model on meta device (saves RAM)
with init_empty_weights():
    model = fuse2_model_local.Fuse2ForCausalLM(config)

# Load weights with 4-bit quantization
quant_suffixes = ("q_proj.weight", "k_proj.weight", "v_proj.weight", "o_proj.weight",
                  "gate_proj.weight", "up_proj.weight", "down_proj.weight")

with open(f"{MODEL_PATH}/model.safetensors.index.json") as f:
    index = json.load(f)
weight_map = index["weight_map"]

def nav(model, key):
    parts = key.split(".")
    obj = model
    for p in parts[:-1]:
        obj = obj[int(p)] if p.isdigit() else getattr(obj, p)
    return obj, parts[-1]

def find_linear(model, key):
    parts = key.split(".")
    obj = model
    for p in parts[:-2]:
        obj = obj[int(p)] if p.isdigit() else getattr(obj, p)
    return obj, parts[-2]

param_names = set(dict(model.named_parameters()).keys())
replaced = {}

for shard_name in sorted(set(weight_map.values())):
    with safe_open(os.path.join(MODEL_PATH, shard_name), framework="pt", device="cpu") as f:
        for key in [k for k, v in weight_map.items() if v == shard_name]:
            if key not in param_names:
                continue
            tensor = f.get_tensor(key)
            if any(key.endswith(s) for s in quant_suffixes):
                owner, attr = find_linear(model, key)
                old = getattr(owner, attr)
                new = bnb.nn.Linear4bit(old.in_features, old.out_features,
                                        bias=False, quant_type="nf4",
                                        compute_dtype=torch.bfloat16, device=DEVICE)
                new.weight = bnb.nn.Params4bit(tensor.to(torch.bfloat16),
                                               requires_grad=False, quant_type="nf4").cuda(0)
                setattr(owner, attr, new)
            else:
                parent, pname = nav(model, key)
                parent._parameters[pname] = torch.nn.Parameter(
                    tensor.to(torch.bfloat16).to(DEVICE), requires_grad=False)
            del tensor
    gc.collect(); torch.cuda.empty_cache()

# Fix tied embeddings
if model.lm_head.weight.device.type == 'meta':
    model.lm_head.weight = torch.nn.Parameter(
        model.model.embed_tokens.weight.data.clone(), requires_grad=False)

# Apply runtime fixes (SwiGLU clamp + router stability)
from fuse2_model_local import Fuse2AugmentedLayer
import torch.nn.functional as F

for layer in model.model.layers:
    if not isinstance(layer, Fuse2AugmentedLayer):
        continue
    if hasattr(layer, 'coding_gate') and layer.coding_gate.device.type == 'meta':
        layer.coding_gate = torch.nn.Parameter(torch.tensor(-2.0, device=DEVICE))
    if hasattr(layer, 'coding_norm') and layer.coding_norm.weight.device.type == 'meta':
        layer.coding_norm = torch.nn.RMSNorm(
            layer.coding_norm.weight.shape[0], eps=1e-6).to(DEVICE)
    experts = getattr(layer, "experts", None)
    if experts:
        for expert in experts:
            gp, up, dp = expert.gate_proj, expert.up_proj, expert.down_proj
            def make_fwd(g, u, d, lim=10.0):
                def forward(x):
                    return d(torch.clamp(F.silu(g(x)) * u(x), -lim, lim))
                return forward
            expert.forward = make_fwd(gp, up, dp)
    router = getattr(layer, "router", None)
    if router:
        gate, top_k = router.gate, router.top_k
        def make_router(g, tk):
            def forward(h):
                logits = g(h)
                scores = torch.clamp(F.softplus(logits), min=1e-6).sqrt()
                w, idx = scores.topk(tk, dim=-1)
                return w / (w.sum(dim=-1, keepdim=True) + 1e-8), idx, logits
            return forward
        router.forward = make_router(gate, top_k)

model.set_coding_enabled(True)
model.to(DEVICE)
model.eval()

# Fix norm dtypes (float32 → bfloat16 for fused kernels)
for module in model.modules():
    if hasattr(module, 'weight') and hasattr(module, 'eps'):
        if module.weight.dtype == torch.float32:
            module.weight.data = module.weight.data.to(torch.bfloat16)

# Tokenizer
tok = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)

# Generate
messages = [{"role": "user", "content": "Write a Python fizzbuzz."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tok(text, return_tensors="pt").input_ids.to(DEVICE)

with torch.inference_mode():
    out = model.generate(input_ids, max_new_tokens=512, do_sample=False,
                         repetition_penalty=1.3)
print(tok.decode(out[0], skip_special_tokens=True))

DSpark Speculative Decoding (~10 tok/s)

For 2x speedup, use DSpark speculative decoding. This requires the drafter model:

# The drafter is included in the model directory
# It's a 5-layer Qwen3 model that predicts 7 tokens at once

See the fuse2_dspark_fast.py script (included in this repo) for the full implementation.

VRAM Breakdown

Configuration	VRAM (allocated)	VRAM (reserved)	Speed
4-bit target only	8.9 GB	9.4 GB	~5 tok/s
4-bit target + BF16 drafter	11.7 GB	12.2 GB	OVERFLOW
4-bit target + 4-bit drafter	10.3 GB	10.8 GB	~8 tok/s
BF16-attn target + 4-bit drafter (shared embed)	10.9 GB	11.4 GB	~10 tok/s

Critical: On RTX 3060 12GB, Windows reserves ~0.5GB for display. If reserved VRAM exceeds ~11.5GB, CUDA silently spills to system RAM via PCIe, causing a 50x slowdown. Always check torch.cuda.memory_reserved(0).

---

DSpark Speculative Decoding

This model supports DSpark (Draft Speculative) decoding for 2x speedup:

How it works

1. Drafter (5-layer Qwen3, 2560 hidden): Predicts 7 tokens in a single forward pass using masked attention + target hidden states
2. Target (12B Fuse-2): Verifies all 7 tokens in ONE forward pass
3. Greedy verification: Accept the longest prefix where drafter matches target's argmax
4. Bonus token: Target always produces 1 bonus token (even if all 7 are rejected)

Key optimizations

• Forward hooks on 5 target layers (not output_hidden_states=True on all 36)
• Single target forward per block (no sequential fallback)
• 4-bit drafter to save VRAM
• Shared embedding between target and drafter (saves 0.8 GB)
• Sliding window KV cache (512 tokens) for constant speed on long sequences

Acceptance rate

• Short prompts: 3.0-3.4/7 (43-49%)
• Long prompts: 2.3-2.5/7 (33-36%)
• Code patterns: up to 4.5/7 (64%)

---

Performance

Speed (RTX 3060 12GB, 4-bit NF4)

Mode	Speed (short)	Speed (long, 512+ tok)	VRAM
Basic generation	5.5 tok/s	5.5 tok/s	8.9 GB
DSpark speculative	10.0 tok/s	8.5 tok/s	10.9 GB

Quality

The model produces:

• Reasoning: Qwen3-quality chain-of-thought (the host handles this)
• Code: DeepSeek-quality code generation (the experts handle this)
• Mixed: Seamless switching between reasoning and code

Example output

Prompt: "Write a Python function to check if a number is prime."

Output (excerpt):

Okay, I need to write a Python function to check if a number is prime.
Let me think about how to approach this.

First, a prime number is a number greater than 1 that has no divisors
other than 1 and itself...

1. Check if the number is less than 2 → return False.
2. Check if the number is 2 → return True
3. Check if the number is even → if yes, return False
4. Iterate from 3 to sqrt(n), stepping by 2
5. For each i, check if n is divisible by i
6. If none divide n, return True

---

File Structure

mini-whale-1-12b/
├── config.json                      # Model configuration
├── generation_config.json           # Generation defaults
├── model-00001-of-00005.safetensors # Shard 1 (5.1 GB)
├── model-00002-of-00005.safetensors # Shard 2 (5.1 GB)
├── model-00003-of-00005.safetensors # Shard 3 (5.1 GB)
├── model-00004-of-00005.safetensors # Shard 4 (5.1 GB)
├── model-00005-of-00005.safetensors # Shard 5 (3.5 GB)
├── model.safetensors.index.json     # Shard index
├── fuse2_model.py                   # Model code (HF auto_map)
├── fuse2_model_local.py             # Model code (local import)
├── tokenizer.json                   # Tokenizer
├── tokenizer_config.json            # Tokenizer config
├── vocab.json                       # Vocabulary
├── merges.txt                       # BPE merges
└── README.md                        # This file

---

Citation

@misc{miniwhale1,
  title={Mini Whale 1 12B: Fusing Qwen3-4B with DeepSeek Coding Experts},
  author={Vasko Djack},
  year={2026},
  url={https://huggingface.co/Akahsizrr/Mini-Whale-1-12B}
}

---

License

Apache 2.0 — see LICENSE file for details.

Acknowledgments

• Qwen3-4B by Alibaba/Qwen Team — host model
• DeepSeek-V4-Flash by DeepSeek AI — coding experts
• DSpark speculative decoding framework
• bitsandbytes for 4-bit NF4 quantization