Instructions to use soundTeam/MS3.2-24b-Angel_mlx-q8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use soundTeam/MS3.2-24b-Angel_mlx-q8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="soundTeam/MS3.2-24b-Angel_mlx-q8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("soundTeam/MS3.2-24b-Angel_mlx-q8")
model = AutoModelForImageTextToText.from_pretrained("soundTeam/MS3.2-24b-Angel_mlx-q8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use soundTeam/MS3.2-24b-Angel_mlx-q8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "soundTeam/MS3.2-24b-Angel_mlx-q8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soundTeam/MS3.2-24b-Angel_mlx-q8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/soundTeam/MS3.2-24b-Angel_mlx-q8

SGLang

How to use soundTeam/MS3.2-24b-Angel_mlx-q8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "soundTeam/MS3.2-24b-Angel_mlx-q8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soundTeam/MS3.2-24b-Angel_mlx-q8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "soundTeam/MS3.2-24b-Angel_mlx-q8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "soundTeam/MS3.2-24b-Angel_mlx-q8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Unsloth Studio

How to use soundTeam/MS3.2-24b-Angel_mlx-q8 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for soundTeam/MS3.2-24b-Angel_mlx-q8 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for soundTeam/MS3.2-24b-Angel_mlx-q8 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for soundTeam/MS3.2-24b-Angel_mlx-q8 to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="soundTeam/MS3.2-24b-Angel_mlx-q8",
    max_seq_length=2048,
)

Docker Model Runner
How to use soundTeam/MS3.2-24b-Angel_mlx-q8 with Docker Model Runner:
```
docker model run hf.co/soundTeam/MS3.2-24b-Angel_mlx-q8
```

MLX format for Angel 24b

Get em while they're hot.

This one is at Q8 quality. Vision stack appears to be mangled somewhat, sadly.

Angel 24b

Better to reign in Hell than serve in Heaven.

Overview

MS3.2-24b-Angel is a model finetuned from Mistral Small 3.2 for roleplaying, storywriting, and differently-flavored general instruct usecases.

Testing revealed strong prose and character portrayal for its class, rivalling the preferred 72B models of some testers.

Quantizations

EXL3:

Official EXL3 quants (thanks artus <3)

GGUF:

Official GGUF imatrix quants w/ mmproj (thanks artus, again <3)

MLX:

TODO! :3

Usage

Use Mistral v7 Tekken.
It is highly recommended (if your framework supports it) to use the official Mistral tokenization code instead of Huggingface's. This is possible in vLLM with --tokenizer-mode mistral.
Recommended samplers (from CURSE and corroborated by me, Fizz) are 1.2 temperature, 0.1 min_p, and 1.05 repetition penalty.
We recommend a system prompt, but its contents only faintly matter (I accidentally had an assistant system prompt during the entire time I was testing)

Training Process

The original model had its vision adapter removed for better optimization and easier usage in training frameworks
The model was then put through an SFT process (using Axolotl) on various sources of general instruct, storytelling, and RP data, which resulted in allura-forge/ms32-sft-merged.
Afterwards, the model was put through a KTO process (using Unsloth) on more focused storywriting and anti-slop data, as well as general instruction following and human preference, which resulted in the final checkpoints at allura-forge/ms32-final-TEXTONLY.
Finally, the vision tower was manually added back to the weights to continue to support multimodality.

Credits

Fizz - training and data wrangling
Artus (by proxy) & Bot - help with funding
CURSE - testing
Mango - testing, data, help with KTO configs
DoctorShotgun - making the original text-only model
Axolotl & Unsloth - creating the training frameworks used for parts of this finetune
Everyone in Allura - moral support, being cool
Vivziepop and co - Angel Dust

<3 love you all

Downloads last month: 2

Model tree for soundTeam/MS3.2-24b-Angel_mlx-q8

Base model

mistralai/Mistral-Small-3.1-24B-Base-2503

Finetuned

mistralai/Mistral-Small-3.2-24B-Instruct-2506

Finetuned

allura-org/MS3.2-24b-Angel

Quantized

(6)

this model

soundTeam
/

MS3.2-24b-Angel_mlx-q8