Merge-35B-A3B-Reasoning-v5-b1

Merge AI Lab · Mixture-of-Experts model

A LoRA supervised fine-tune of Unsloth's Qwen3.6-35B-A3B (Mixture-of-Experts, ~3B active parameters), trained locally on a single AMD Radeon 8060S in BF16 via Unsloth, at Merge.

This is one of two models we are publishing from the same internal programme. Its dense sibling, Merge-27B-MTP-Reasoning-v1, is the quality reference; this one is the throughput pick — noticeably faster per token at close to the same score. Both were developed against the same capability targets and scored on the same internal benchmark.

The finding worth sharing: the LoRA targets include the routed-expert FFNs. Earlier attempts (a2–a4) adapted only the attention and shared-expert projections — roughly 11% of the active FFN — and moved the benchmark essentially not at all (89 versus a base of 90 on the rubric of the time). Targeting the routed experts is what made the run work. If you are fine-tuning a sparse MoE and seeing a flat score, that is the first thing to check.

What it was built for

Both Merge models were developed and evaluated against one capability set, drawn from the work our internal agents actually do:

Capability	What that means here
Reasoning	Long-form deliberate reasoning with a real thinking budget, then a committed answer.
Agentic / tool calling	Multi-step tool use: calling, reading results back, and not re-calling a tool it has already answered from.
Coding	Modern web production — ReactJS, Next.js, TypeScript, PostgreSQL (schema, queries, typed server routes).
pt-PT creative copy	European Portuguese marketing and interface copy — pt-PT, not pt-BR — with structure discipline, not just translation.

The end purpose is internal: both models drive MCP servers and tool-using agents across our copy, design and implementation pipeline. They are meant to be the local model behind an agent that plans a page, writes the copy in pt-PT, and implements the component — not a general-purpose chat assistant. MoE routing makes this one the better fit where an agent loop has to run often and fast.

The SFT corpus itself is reasoning-trace distillation (below). The capability list above is what the model was selected and measured against, not a description of the training mix.

Out of scope: anything safety-critical. No safety tuning was added beyond the base model's.

Results

Scored on our private internal benchmark (described below). Both rows are the same quantization, the same harness, the same grader.

Model	Quant	Technical /100
unsloth/Qwen3.6-35B-A3B-MTP (base)	Q8_0	81
Merge-35B-A3B-Reasoning-v5-b1	Q8_0	92

+11 points over the base — the largest gain of any model in the programme, and the result that justified the routed-expert recipe. Two independent runs both scored 92.

Secondary metric	Score
Vision bonus	10 / 12
Tool-grounding bonus	14–15 / 19

For context, the dense 27B sibling scores 94 at Q4_K_M — two points ahead, at lower throughput.

About the benchmark

We are not publishing the benchmark, but here is what it is and is not.

It is an 8-task suite built from real briefs in our own production workflow — not synthetic puzzles and not sampled from any public set. Each task is graded against a hidden rubric with withheld answer keys, and the tasks cover exactly the capability list above: design decomposition, planning a content structure from a thin brief, writing and adapting frontend components, pt-PT localization quality, surgical CSS repair from visual intent, client-side React/Next interaction, and a typed Next.js server route backed by PostgreSQL.

Scoring:

• Technical /100 — seven tasks, raw scores summed and rescaled to 100. This is the ranking metric, and full credit is reachable without any tool use.
• Vision bonus /12 — one image-grounded task, scored separately so text-only models are not penalized.
• Tool-grounding bonus /19 — quality of tool use, awarded only against verifiable session-log evidence. Also separate from the ranking score.

Deliberately, the technical score judges output quality, not protocol compliance: a model is not punished on its main score for clumsy tool orchestration, because a bad plan executed cleanly is still a bad plan.

Honest limits, stated plainly:

• n = 2 runs, 8 tasks, single-operator rubric grading. The 2-point gap to the 27B's 94 is within plausible noise; do not read it as a settled ranking.
• Scores are not comparable to public leaderboards. Different tasks, different rubric, different grader.
• The two Merge models were scored at different sampling presets — this one at temp 0.6, the 27B at temp 0.1 — because each was run at the settings that suit it. The 94-vs-92 comparison is therefore not a clean like-for-like measurement. The 81→92 delta on this card is like-for-like: same preset, same quant, same harness.
• A pre-export capability gate scored 12/14, identical to the base — the gate was saturated and did not discriminate. It is not evidence of improvement.
• During benchmark runs this model triggered repeated-search warnings on three tasks; the 27B triggered none. Nothing was blocked, but this run was nudged and the 27B was not.

Training

Field	Value
Method	LoRA SFT (Unsloth) — 4-bit quantized base load + BF16 LoRA, deltas merged onto the BF16 base at export
Base	unsloth/Qwen3.6-35B-A3B @ 2ab40a9
Rank / alpha / dropout	16 / 16 / 0
Target modules	q_proj k_proj v_proj o_proj, in_proj_qkv in_proj_z in_proj_a in_proj_b out_proj, gate_proj up_proj down_proj
Target parameters	mlp.experts.gate_up_proj, mlp.experts.down_proj — the routed experts
Frozen	router (mlp.gate), MTP heads, vision tensors
LR / scheduler / warmup	2e-4 / linear / 5 steps
Weight decay / seed	0.001 / 3407
Epochs / steps	3 / ~404
Batch	1 × grad-accum 8
Context	8192, train_on_responses_only
Optimizer	adamw_8bit

Keeping the router frozen is deliberate: training the gate alongside the experts destabilised earlier runs. Adapt the experts, leave routing to the base.

Note on precision: the base was loaded 4-bit (bnb) during training — an implicit library default rather than a deliberate choice, discovered afterwards. The LoRA deltas were merged onto the full BF16 base at export, so the shipped weights are BF16-derived. The train/merge precision mismatch is disclosed here for reproducibility.

Training data

Same corpus as the 27B: 1,078 reasoning conversations from two public TeichAI datasets (Claude Opus 4.5, 250 examples; Opus 4.6, 887), plain messages turns, no tools, and no private, client or company data.

Provenance disclosure: these are distilled traces generated by Claude Opus models. Anyone redistributing or building on this model should check both the TeichAI dataset licenses and Anthropic's terms covering the use of model outputs.

Files

File	Size	Notes
model-0000{1..8}-of-00008.safetensors	68 GB	BF16 merged weights (HF format), MTP tensors included
Merge-35B-A3B-Reasoning-v5-b1.Q8_0.gguf	35.2 GB	the benchmarked build
mmproj-F32.gguf	861 MB	vision projector (required for image input)
lora/	3.5 GB	LoRA adapter, for re-merging onto the base

Q8_0 is the only quantization we ship, because it is the only one we benchmarked. For any other quant or runtime format, convert from the BF16 weights.

A note for anyone merging LoRAs into this architecture

The BF16 checkpoint here was rebuilt from the adapter, and doing so surfaced a trap worth passing on. PEFT's merge_and_unload() writes the routed experts de-fused — 256 separate mlp.experts.{i}.{gate,up,down}_proj.weight tensors per layer — while Qwen3_5MoeForConditionalGeneration expects them packed as experts.gate_up_proj (num_experts, 2*moe_intermediate, hidden) and experts.down_proj (num_experts, hidden, moe_intermediate), and defines an empty _checkpoint_conversion_mapping. Nothing reconciles the two.

A checkpoint saved that way loads with tens of thousands of unexpected keys and randomly-initialized experts — it will run and emit fluent nonsense. GGUF conversion is unaffected, which is how it can go unnoticed. The weights published here were re-fused into the packed layout and verified to load with zero missing, unexpected, or mismatched keys.

If you merge a LoRA into a Qwen3.5/3.6 MoE, check your loading report before you trust the result.

SHA-256 of the GGUF: b33f534225df8e748d5cdc23b3914f3c2bd356a66d0f35bfc6c6393183bbe21f

Embedded GGUF metadata: general.name = Merge-35B-A3B-Reasoning-v5-b1, general.architecture = qwen35moe, general.size_label = 256x2.6B.

The general.name field was corrected on 2026-08-05 (it previously read B1, an internal build label). Tensor data is unchanged; only the metadata block differs. This changed the file hash — the pre-correction file hashed 7834f0b186124563837cf8561cd93bd774ae7d64715ee75070bf1e00ac5e3996.

Sampling settings

Use these. They are the settings the 92/100 was produced at, and the temperature is what keeps this model out of the failure mode described below.

Parameter	Value
temperature	0.6
top_k	20
top_p	0.95
min_p	0
repeat_penalty	1.1
presence_penalty	off

Do not run this model at very low temperature. temp 0.1 with top_k 40 / top_p 0.9 / repeat_penalty 1.05 is the configuration that produced a runaway reasoning loop. At near-greedy settings this model can settle into a self-referential attractor and never close its </think> block. If you need determinism, use a fixed seed at temp 0.6 rather than lowering the temperature.

This is the main way it differs operationally from the dense 27B, which is stable at low temperature.

Note: the GGUF embeds temp 1.0, top_p 0.95, top_k 20. Set the temperature explicitly.

The 76/100 result, and why it is not the model

An earlier July 2026 evaluation of this same build scored 76/100, with one task at 0 from a runaway reasoning loop. This was a decoding-configuration failure, not a capability defect, and the settings above are the fix. We are documenting it rather than deleting it, because it is the most useful thing we learned about running MoE reasoning models locally.

A controlled replay reconstructed the exact 65k-token context at loop onset and ran 2 sampler configurations × 3 seeds:

Sampler config (as tested in the probe)	Result
temp 0.1, top_k 40, top_p 0.9, repeat_penalty 1.05	1 of 3 seeds looped and never closed </think>
temp 0.6, top_p 0.95, top_k 20, presence_penalty 1.0	6 of 6 productive — closed reasoning and emitted valid output

Temperature is the variable that mattered. The probe's recovery config differs slightly from the shipping preset above (it used presence_penalty 1.0; the shipping preset uses repeat_penalty 1.1 with presence off) — both sit at temp 0.6, which is the part that prevents the loop. At temp 0.6 the failure did not reproduce at all.

Usage

llama-server \
  -m Merge-35B-A3B-Reasoning-v5-b1.Q8_0.gguf \
  --mmproj mmproj-F32.gguf \
  --jinja --reasoning-format auto \
  --temp 0.6 --top-k 20 --top-p 0.95 --repeat-penalty 1.1 \
  -ngl 999 --flash-attn on

Limitations

• Sensitive to decoding settings — see above. This is the main operational caveat.
• Trained at 8k context; longer-context behaviour is the base model's.
• MoE routing was not tuned — the router is frozen, so expert selection is base behaviour.
• English and European Portuguese in practice.
• Two points behind the dense 27B on our benchmark. Pick this one for throughput, that one for the last bit of quality.
• No added safety tuning.

Licensing

Released under Apache 2.0, matching the base as published by Unsloth (unsloth/Qwen3.6-35B-A3B — apache-2.0, derived from Qwen/Qwen3.6-35B-A3B).

Two further conditions apply to anyone building on this model, and they are not ours to grant: the TeichAI dataset licenses, and Anthropic's terms on the use of Claude outputs — the training traces are Claude-distilled. Check both before redistributing.

Citation

@misc{merge35b_a3b_reasoning_v5b1,
  title  = {Merge-35B-A3B-Reasoning-v5-b1},
  author = {Mergeinto.digital},
  year   = {2026},
  note   = {Routed-expert LoRA SFT of Unsloth Qwen3.6-35B-A3B on distilled reasoning traces}
}