README.md

---
license: apache-2.0
base_model: stamsam/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-MLX-oQ4-MTP
base_model_relation: quantized
pipeline_tag: text-generation
language:
  - en
tags:
  - servicenow
  - itsm
  - csdm
  - delivery
  - gguf
  - llama.cpp
  - ollama
  - lm-studio
  - quantized
  - qwen3.5
  - moe
  - mixture-of-experts
---

# marvy-2-35B-MoE-GGUF

**GGUF quants of marvy-2, a 35B-A3B Mixture-of-Experts model fine-tuned for the
ServiceNow delivery lifecycle.** ~3B active parameters per token; runs on a
single consumer GPU or fast Apple Silicon thanks to the MoE sparsity.

GGUF quantizations for use with [llama.cpp](https://github.com/ggml-org/llama.cpp),
[Ollama](https://ollama.com), [LM Studio](https://lmstudio.ai), and compatible
runtimes.

> Released under **Apache-2.0**. Built with Qwen3.5 (Apache-2.0) via
> `unsloth/Qwen3.6-35B-A3B` and the Opus-distilled `stamsam/...MTP` base.

## Files

| File | Quant | Size | Use when |
|---|---|---|---|
| `marvy-2-35B-MoE-Q4_K_M.gguf` | Q4_K_M | ~20 GB | Default — best size/quality balance |
| `marvy-2-35B-MoE-Q8_0.gguf`   | Q8_0   | ~34 GB | Near-FP16 quality, more headroom |

## Architecture notes

This is a **hybrid SSM + MoE Transformer**:

- 40 layers, mixed SSM (Mamba-style) and grouped-query attention blocks
- 256 routed experts per MoE layer, 3B active per token (A3B)
- Shared expert per layer for common pathways
- `qwen3_5_moe` architecture in llama.cpp; needs a recent build
  (see "Supported runtimes" below)

The Multi-Token Prediction (MTP) head present in the base model was **not
included** in this GGUF — these quants are text-only causal LM. The base
model's MoE expert weights and SSM blocks are preserved.

## Quick start

### Ollama

```bash
ollama run hf.co/MainStack/marvy-2-35B-MoE-GGUF:Q4_K_M
```

### llama.cpp

```bash
llama-cli -hf MainStack/marvy-2-35B-MoE-GGUF:Q4_K_M \
  -p "Write a ServiceNow user story with acceptance criteria for P1 SLA escalation." \
  --temp 0.4 \
  -c 4096
```

### LM Studio

Search the model catalog for `marvy-2-35B-MoE-GGUF` and download, or load
the local `.gguf` file via "Open Model in Folder". LM Studio's OpenAI-compatible
server is at `http://localhost:1234/v1` by default.

## Supported runtimes

The `qwen3_5_moe` architecture (with mixed SSM layers and Multi-Token
Prediction in the base) is new. Verify your runtime supports it:

- **llama.cpp**: master branch as of mid-2026 (commits containing
  `Qwen3_5MoeForConditionalGeneration` registration in `conversion/qwen.py`).
  The Homebrew formula may lag — clone upstream if you hit
  "unknown architecture" errors.
- **Ollama**: ships its own llama.cpp; check that your Ollama version is
  recent.
- **LM Studio**: uses its own bundled runtime; recent versions support
  Qwen3.5 MoE.

## Trained on

- v1 corpus: ServiceNow delivery lifecycle artifacts (SOW, SDD, stories,
  acceptance criteria, value hypothesis, ...) — same data marvy-1 used.
- v2 corpus: extended to capability-to-epic mapping, mermaid diagram
  authoring, deployment package modeling, stakeholder mapping, story-to-UAT,
  and more. See `EVAL.md` in the repo root for per-task perplexity.

## How this GGUF was built

```
LoRA adapter (rank 32, 350 steps, attention-only Q/K/V/O)
    +
bf16 base   (unsloth/Qwen3.6-35B-A3B)
    │   mlx_lm fuse
merged-bf16/  (14 shards, 65 GB safetensors with mlx_lm switch_mlp naming)
    │   scripts/marvy-v2-rename-moe-tensors.py  (bridge to HF-canonical names)
merged-bf16-hf/  (15 shards, 65 GB; switch_mlp → experts.gate_up_proj packed)
    │   llama.cpp/convert_hf_to_gguf.py --no-mtp --outtype f16
marvy-2-35B-MoE-F16.gguf  (65 GB, 733 tensors)
    │   llama-quantize
marvy-2-35B-MoE-Q4_K_M.gguf  (20 GB, 4.6 BPW)
marvy-2-35B-MoE-Q8_0.gguf    (34 GB, 8.52 BPW)
```

End-to-end build script: `scripts/marvy-v2-35B-MoE-build-gguf.sh` (in the
source repo). The `switch_mlp → experts` rename is necessary because mlx_lm
fuse and llama.cpp's converter use different naming conventions for routed
MoE tensors.

## License

Apache-2.0 (inherits from Qwen). See `LICENSE` and `NOTICE` in the repo root.

## Citation

```bibtex
@misc{marvy-2-35B-MoE,
  title  = {marvy-2: A ServiceNow delivery-lifecycle MoE LLM},
  author = {MainStack},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/MainStack/marvy-2-35B-MoE-GGUF}}
}
```