This repository contains the GGUF-quantized, pre-merged specialist models for **Macaron-V1-Tall**, optimized for local deployment on consumer hardware.
## 🏆 Acknowledgments and Credits
All foundational research, training, and architecture design belong to the original authors. This repository solely provides a community-driven quantization and deployment format.
## 🧩 The Architectural Pivot (Why this repository exists)
The Macaron-V1-Tall architecture utilizes a Mixture-of-LoRA (MoE) pattern over a Grouped-Query Attention (GQA) base model. Currently, the standard `llama.cpp` ecosystem encounters tensor reshaping limitations (`NotImplementedError`) when attempting to dynamically apply these specific LoRA adapters at runtime.
To bypass this mathematical constraint and enable seamless local execution, this repository utilizes a **Pre-Merge Strategy**. Each LoRA specialist has been physically fused into the 35B base model using CPU-RAM computation (bypassing VRAM bottlenecks) prior to GGUF conversion.
**The Result:** Four independent, fully fused GGUF models. Instead of dynamically swapping LoRAs in VRAM, developers can leverage operating system Page Caching to swiftly alternate between these ~21GB files in RAM, enabling rapid intent-based routing without out-of-memory (OOM) errors.
## 📦 Available Specialists (Q4_K_M Quantization)
The model have been quantized to `Q4_K_M` to balance perplexity and memory footprint, reducing the required storage from ~70GB (FP16) to approximately **21GB per specialist**.
> **Note:** The experimental `mtp_num_hidden_layers` (Multi-Token Prediction) metadata has been sanitized from the config to ensure strict compatibility with the `llama.cpp` loader.
## 🚀 Deployment & Usage (llama.cpp)
These models are heavily optimized for hybrid VRAM/RAM offloading. If you are running a consumer GPU (e.g., 16GB VRAM) backed by substantial system RAM, you must carefully balance the layers to prevent VRAM saturation.
### Example of a Launch Command (`llama-server`)
The following command demonstrates how to load the model while explicitly offloading the heaviest MoE layers to the CPU, freeing up your VRAM for the KV cache and attention heads.
