Instructions to use mindlab-research/Macaron-V1-Tall with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mindlab-research/Macaron-V1-Tall with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mindlab-research/Macaron-V1-Tall")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("mindlab-research/Macaron-V1-Tall")
model = AutoModelForMultimodalLM.from_pretrained("mindlab-research/Macaron-V1-Tall", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mindlab-research/Macaron-V1-Tall with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mindlab-research/Macaron-V1-Tall"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mindlab-research/Macaron-V1-Tall",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mindlab-research/Macaron-V1-Tall

SGLang

How to use mindlab-research/Macaron-V1-Tall with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mindlab-research/Macaron-V1-Tall" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mindlab-research/Macaron-V1-Tall",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mindlab-research/Macaron-V1-Tall" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mindlab-research/Macaron-V1-Tall",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mindlab-research/Macaron-V1-Tall with Docker Model Runner:
```
docker model run hf.co/mindlab-research/Macaron-V1-Tall
```

MTP?

by Trilogix1 - opened 8 days ago

Discussion

Trilogix1

8 days ago

Any plan to mtp?

ilintar

7 days ago

@Trilogix1 I think you missed something :)

Trilogix1

7 days ago

•

edited 7 days ago

@ilintar

Good catch thanks (appreciated) , this happens when you have 100s of version. I used an old build to quantize :)

Hugston-Macaron-V1-Tall-Qx_K_M.gguf .... load_model: failed to create MTP context....

Edit: Transformers discard MTP during loading with the lora coding merged weights. It classify it as different (weight and loras not from same source). Even though it find the 19 layers of mtp is discarded them. If someone can make it work with MTP let me know.

ilintar

7 days ago

I wouldn't recommend running this on llama.cpp to be honest. Without the LoRA routing this is just a plain old Qwen3.6 35B-A3B. There's another model currently being ported with a similar architecture (Granite-Switch), so once that gets merged I think we can do this one more easily.

Trilogix1

7 days ago

Right, need to merge each loras and test for this one. I will definitely take a look on the granite-switch too.
Haven´t seen you playing with any mix or model preference. Curious to know what´s your top 5 models list under 400B, considering (architecture, intelligence, tps, --flags especially coding). What about abliteration method, what's your pick if you had to? We can exchange at least one "good" model.
You can reply at trilogic@hugston.com if is not public info.

ilintar

7 days ago

@Trilogix1 you can merge LoRAs, but that defeats the purpose. The whole idea of the router / switch is that you load the main model, then you apply the LoRAs "on the fly". If you merge all LoRAs, you basically end up with a bad 140B model. So you need some way to dynamically load the LoRAs.

Top 4 list under 400B: StepFun 3.7 Flash, MiniMax M3, Hy3, Qwen3.6 27B. Haven't yet tested Laguna properly.

Trilogix1

7 days ago

•

edited 6 days ago

Good model choice, missing still the diamond one.

I understand the purpose but a merge is running faster tps in our tests and you keep same size (it doesnt expand the base tensors but fill them with new training info so it cant possibly be 140b but same size and weight). Routering is useful when you change domain often (but we test and bench only 2 domains which is covered quite good with lora). Abliteration method?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment