Akahsizrr/mini-deepseek-v4-flash

Mini-Deepseek v4 Flash

Mini-Deepseek v4 Flash is a custom 12.4B-parameter hybrid Mixture-of-Experts language model for coding, software engineering, and technical reasoning. It combines a Qwen3-4B host with coding-specialized experts extracted from DeepSeek V4 Flash and connected through learned representation bridges and per-layer routers.

The model is designed for practical coding assistance: code generation, debugging, refactoring, architecture discussions, algorithm implementation, web development, and technical explanation. It supports long-context Transformers inference and has been tested with a native KV-cache path, vLLM's Transformers backend, bitsandbytes quantization, and an MLX adapter.

Important: This is research software with custom remote model code. Review fuse2_model.py before enabling trust_remote_code=True in an untrusted environment.

What is the model?

Mini-Deepseek v4 Flash is a structural fusion model rather than a conventional dense fine-tune or a pure distillation student.

• Host: Qwen3-4B, 36 decoder layers, hidden size 2560.
• Coding path: 260 coding-specialized DeepSeek V4 Flash SwiGLU experts.
• Expert space: hidden size 4096, intermediate size 2048.
• Routing: per-layer sqrt-softplus router with top-k expert selection.
• Bridges: learned 2560 -> 4096 and 4096 -> 2560 projections.
• Repair path: low-rank residual repair projection.
• Stored size: approximately 12.4B parameters for the assembled BF16 checkpoint.
• Active compute: approximately 5-6B parameters per token, depending on routing.
• Context: 40,960 positions in the source configuration; deployment profiles may select a lower operational limit for memory and latency.

The host and extracted experts remain separate architectural components inside each augmented layer. Coding experts are selected per token; the dense host remains the general-language backbone.

Quick start with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Akahsizrr/Fuse-1-Lite"
tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="merged-v2-full", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    subfolder="merged-v2-full",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

messages = [{"role": "user", "content": "Implement a rate-limited async HTTP client in Python."}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        use_cache=True,
        temperature=0.4,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use enable_thinking=False for direct code artifacts. The model's custom forward path now preserves past_key_values and uses the native Transformers cache object.

Quantized variants

The repository contains validated variants under quantized/:

Variant	Format	Intended runtime	Status
fuse2-4bit-bnb	bitsandbytes NF4	Transformers + CUDA	Reload and generation validated on A100
fuse2-4bit-fp4-bnb	bitsandbytes FP4	Transformers + CUDA	Reload and generation validated on A100
fuse2-8bit-bnb	bitsandbytes INT8	Transformers + CUDA	Reload and generation validated on A100
fuse2-fp8-e4m3fn	FP8 E4M3FN weight-only	Custom Transformers runtime	Reload and generation validated on A100

The quantized variants retain the Fuse-2 expert scaling and SwiGLU stability safeguards. They are not interchangeable with standard Qwen or Llama checkpoints.

vLLM support

The BF16 assembled checkpoint is compatible with vLLM's Transformers modeling backend:

vllm serve Akahsizrr/Fuse-1-Lite/merged-v2-full \
  --trust-remote-code \
  --model-impl transformers \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85

The vLLM integration uses a proper Fuse2Model decoder mapping, native vLLM KV-cache allocation, and custom-layer compatibility aliases. The bitsandbytes 8-bit artifact is intended for the Transformers runtime; direct vLLM loading of that artifact is not claimed as validated.

MLX support

microscope/fuse2_mlx.py provides the MLX model adapter and microscope/fuse2_mlx_loader.py provides the loader shim. The adapter includes per-layer expert routing and MLX KV-cache support. Cache parity was validated on a compact Fuse-2 test model. A full 12.4B Apple Silicon conversion should be benchmarked on the target Mac before production use.

Architecture and provenance

DeepSeek V4 Flash supplies the source coding experts. Qwen3-4B supplies the host representation, tokenizer-facing path, attention stack, and general-language behavior. Expert selection used activation profiling and causal ablation rather than activation frequency alone. The extraction artifacts and expert manifests are included in the release tree.

See TECHNICAL_REPORT.md for the complete architecture, compatibility, quantization, validation, and limitations report.

Limitations

• This is a research fusion architecture, not a standard dense Qwen checkpoint.
• The expert path adds substantial per-token work even when only a small top-k subset is selected.
• KV caching is now supported, but the custom expert computation remains more expensive than a native fused MoE kernel.
• vLLM support is through its Transformers backend rather than a dedicated native vLLM model implementation.
• MLX support is an adapter and requires full-device validation for the assembled checkpoint.
• GGUF and llama.cpp support are not included in this release.
• No new benchmark score is claimed here beyond the compatibility and reload tests documented in TECHNICAL_REPORT.md.

License

The fusion code and released derivative artifacts are published under the MIT license, subject to the licenses of the source models listed in the provenance section.