File size: 3,092 Bytes
8c05887 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
language:
- en
license: apache-2.0
base_model:
- mindlab-research/Macaron-V1-Tall
tags:
- gguf
- llama.cpp
- qwen
- moe
- macaron
- mixture-of-lora
---
# Macaron-V1-Tall (GGUF Pre-Merged Specialists)
This repository contains the GGUF-quantized, pre-merged specialist models for **Macaron-V1-Tall**, optimized for local deployment on consumer hardware.
## 🏆 Acknowledgments and Credits
All foundational research, training, and architecture design belong to the original authors. This repository solely provides a community-driven quantization and deployment format.
* **Original Authors:** mindlab-research/Macaron-V1-Tall
* **Original Repository:** [mindlab-research/Macaron-V1-Tall](https://huggingface.co/mindlab-research/Macaron-V1-Tall)
* **Base Architecture:** Qwen 3.6 (35B Base + 4x 3.7B LoRA Specialists)
## 🧩 The Architectural Pivot (Why this repository exists)
The Macaron-V1-Tall architecture utilizes a Mixture-of-LoRA (MoE) pattern over a Grouped-Query Attention (GQA) base model. Currently, the standard `llama.cpp` ecosystem encounters tensor reshaping limitations (`NotImplementedError`) when attempting to dynamically apply these specific LoRA adapters at runtime.
To bypass this mathematical constraint and enable seamless local execution, this repository utilizes a **Pre-Merge Strategy**. Each LoRA specialist has been physically fused into the 35B base model using CPU-RAM computation (bypassing VRAM bottlenecks) prior to GGUF conversion.
**The Result:** Four independent, fully fused GGUF models. Instead of dynamically swapping LoRAs in VRAM, developers can leverage operating system Page Caching to swiftly alternate between these ~21GB files in RAM, enabling rapid intent-based routing without out-of-memory (OOM) errors.
## 📦 Available Specialists (Q4_K_M Quantization)
The model have been quantized to `Q4_K_M` to balance perplexity and memory footprint, reducing the required storage from ~70GB (FP16) to approximately **21GB per specialist**.
* `macaron-l2-coding-35b-q4_k_m.gguf` (Coding Specialist)
> **Note:** The experimental `mtp_num_hidden_layers` (Multi-Token Prediction) metadata has been sanitized from the config to ensure strict compatibility with the `llama.cpp` loader.
## 🚀 Deployment & Usage (llama.cpp)
These models are heavily optimized for hybrid VRAM/RAM offloading. If you are running a consumer GPU (e.g., 16GB VRAM) backed by substantial system RAM, you must carefully balance the layers to prevent VRAM saturation.
### Example of a Launch Command (`llama-server`)
The following command demonstrates how to load the model while explicitly offloading the heaviest MoE layers to the CPU, freeing up your VRAM for the KV cache and attention heads.
```bash
./llama-server \
-m "macaron-l2-coding-35b-q4_k_m.gguf" \
--host 127.0.0.1 \
--port 9091 \
-ngl 40 \
-ncmoe 10 \
--ctx-size 16384 \
-b 512 \
-fa on \
-ctk q4_0 \
-ctv q4_0 \
-t 16 \
--reasoning-preserve \
--no-context-shift \
-sps 0.0 \
--ctx-checkpoints 0 \
--models-max 1 \
--parallel 1 \
--no-mmap \
--metrics
|