Clover 2 — white clover sculpture on a black background

Clover 2 is a Mixture of LoRA Experts language model built on a ~27.8B-parameter dense base using the Qwen3.8-27B architecture.

It pairs the base model with seven trained LoRA experts and a learned router. Rather than merging the experts into a fixed set of weights, Clover 2 decides which experts to use for each token, at inference time. When none is a good match, it can leave the base model's behavior unchanged.

One base model. Seven experts. No static merge or manual mode switching.

At a glance

Details
Base ~27.8B dense parameters (Qwen3.8-27B architecture)
Architecture Mixture of LoRA Experts (MoLE)
Experts 7 trained LoRA adapters, plus a null route
Routing Per token, at each layer; top-2 by default
Adapter targets gate_proj, up_proj, down_proj
Serving Custom vLLM integration
License Kitani-Clover License

Architecture

Each layer has a small trained router that evaluates the current token and chooses which expert contributions, if any, to apply. By default, the router selects the top two experts. A null route allows the model to use the unmodified base path when specialized behavior is unnecessary.

The experts are LoRA adapters, not seven full copies of the model. They modify the feed-forward projections (gate_proj, up_proj, and down_proj) and are selected dynamically during a forward pass.

This is different from merging adapters ahead of time: there is no fixed average of expert weights, and you do not need to pick a mode before sending a prompt.

Expert lineup

Slot Expert Focus
1 Roleplay Character consistency and creative dialogue
2 Clover / Assistant General-purpose instruction following and assistant behavior
3 Security Offensive and defensive security research
4 Math Multi-step reasoning and arithmetic
5 Long Document Long-context comprehension and recall
6 Efficiency Direct, concise responses
7 Knowledge Factual recall

Benchmarks

Numbers below are from direct testing, same prompts and sampling settings across models, not vendor-reported figures.

vs other models

Model Roleplay compliance Security compliance (offensive / defensive) AI-writing clichés (tells / 1k words, lower is better)
Clover 2 100% (n=12) 66.7% / 91.7% (n=12 each, security expert masked off, Clover 2's default) 1.53
Llama 4 Maverick 100% (n=12) 83.3% / 83.3% (n=12 each) 2.6
Mistral Large 4 100% (n=5)† 66.7%† / unavailable† unavailable†
Gemini 4 Carbon 91.7% (n=12) 33.3% / 66.7% (n=12 each) 1.35
GPT-4o 50% (n=12) 41.7% / 75% (n=12 each) 2.4 (n=28)‡

† Mistral Large 4 hit a persistent provider-side outage during testing. Missing numbers are marked unavailable, not scored as zero or estimated. ‡ Lower n than the other rows because some responses were excluded (judge/content-filter issues on specific prompts), not because they scored zero.

Security compliance here is measured against dual-use, educational/authorized-research framed prompts (e.g. "I'm red-teaming my own network, how do I..."), not unambiguous malicious requests. With the security expert enabled (not the shipped default), Clover 2 reaches 70.0% / 100.0% on the same offensive/defensive sets.

Clover 2 vs its own base model

Benchmark Clover 2 (routed) Base (no experts active) Delta
Math reasoning (MATH-500-style) 24.0% 8.0% +16.0pp
GSM-Hard 54.0% 36.0% +18.0pp
LongBench-v2 (long-document) 53.3% 46.7% +6.6pp
AI-writing clichés (tells / 1k words) 1.53 2.35 -0.82 (lower is better)
XSTest, safe prompts (refusal rate) 4.1% 18.1% -14.0pp (lower is better)
XSTest, unsafe prompts (refusal rate) 51.8% 79.7% -27.9pp

The last row is a real tradeoff worth being upfront about: routing also makes the model refuse fewer genuinely unsafe prompts, not only the safe ones the base model was over-refusing. That asymmetry is part of why the security expert specifically ships masked off by default rather than relying on general safety behavior alone.

What Clover 2 does well

  • Math and reasoning: In our evaluations, Clover 2 improves on the dense base across GSM8K, MATH-500, and GSM-Hard.
  • Long documents: Better long-context comprehension and retrieval, including gains on LongBench-v2.
  • Writing: Fewer stock phrases, clichés, and other repetitive AI-writing patterns in our direct evaluations.
  • Roleplay: More consistent character voices and fewer unnecessary interruptions during ordinary creative requests, including very long conversations.

These are comparisons against the underlying base model, not claims of superiority to every model of similar size. Numerical scores and evaluation settings should be supplied alongside public benchmark results.

Refusals and security

Clover 2 is intended to handle ordinary creative writing and roleplay without unnecessary refusals or out-of-character disclaimers.

The Security expert (slot 3) is masked by default in the serving configuration. Our evaluations found that enabling it increases compliance with offensive security-research requests, so it is not active in the default configuration. Operators conducting authorized security research can explicitly enable it.

Note: A default serving mask is not a restriction on access to a publicly distributed adapter. Anyone with the weights and a compatible runtime may be able to change that setting.

Repository layout

clover-2/
├── base/                 # Base model weights, tokenizer, and config
├── adapters/             # Seven standard PEFT LoRA adapters
│   └── <expert>/         # adapter_config.json + adapter_model.safetensors
├── router/               # trained_router_v2.pt
└── serving/
    ├── vllm_moelora.py   # vLLM integration
    └── serve_example.py  # Complete serving example

Each adapter uses the standard PEFT format. You can load an individual adapter independently if you prefer to merge or run a single specialization without the router.

Running Clover 2

Requirements: PyTorch, a compatible CUDA environment, and vLLM (tested with 0.31.0).

The quickest starting point is serving/serve_example.py, which demonstrates loading the base, attaching the LoRA routing system, loading the experts and router weights, and warming up the patched model.

The essential setup looks like this:

from vllm import LLM
import vllm_moelora as V

llm = LLM(
    model="./base",
    dtype="bfloat16",
    compilation_config={"mode": 0, "cudagraph_mode": "FULL"},
)

worker = llm.llm_engine.model_executor.driver_worker.worker
moles = V.attach(
    worker.model_runner.model,
    n_experts=7,
    rank=128,
    scaling=2.0,
    topk=2,
)

# Next: load all seven adapters with V.load_adapter(...), then load
# the trained router weights. See serving/serve_example.py for the
# complete sequence and paths.
worker.compile_or_warm_up_model()

This is an outline, not a standalone runnable script. Use the full example to load adapters and router weights before warmup. The worker access shown here depends on the tested vLLM internals.

Important: disable torch.compile

Do not enable vLLM's torch.compile mode with the current routing plugin.

vLLM can trace and compile the model during its initial warmup, before vllm_moelora.py patches the forward pass. If that happens, subsequent inference can reuse the old compiled graph and silently omit all LoRA contributions.

Set compilation_config={"mode": 0, "cudagraph_mode": "FULL"} to disable Torch compilation while retaining vLLM's CUDA graph capture. After loading and attaching the experts and router, call worker.compile_or_warm_up_model() so CUDA graphs capture the patched model.

serving/serve_example.py handles this ordering.

License

Clover 2 is distributed under the Kitani-Clover license. See Kitani-Clover License for the complete terms and conditions.


Built by Kitani · Model repository

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support