Instructions to use zerofata/MS3.2-PaintedFantasy-v2-24B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zerofata/MS3.2-PaintedFantasy-v2-24B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zerofata/MS3.2-PaintedFantasy-v2-24B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("zerofata/MS3.2-PaintedFantasy-v2-24B")
model = AutoModelForCausalLM.from_pretrained("zerofata/MS3.2-PaintedFantasy-v2-24B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zerofata/MS3.2-PaintedFantasy-v2-24B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zerofata/MS3.2-PaintedFantasy-v2-24B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerofata/MS3.2-PaintedFantasy-v2-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/zerofata/MS3.2-PaintedFantasy-v2-24B

SGLang

How to use zerofata/MS3.2-PaintedFantasy-v2-24B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zerofata/MS3.2-PaintedFantasy-v2-24B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerofata/MS3.2-PaintedFantasy-v2-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zerofata/MS3.2-PaintedFantasy-v2-24B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerofata/MS3.2-PaintedFantasy-v2-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use zerofata/MS3.2-PaintedFantasy-v2-24B with Docker Model Runner:
```
docker model run hf.co/zerofata/MS3.2-PaintedFantasy-v2-24B
```

Not bad at all, but...

by Hardeh - opened Aug 2, 2025

Discussion

Hardeh

Aug 2, 2025

This model performs pretty well, but it has a strange quirk of leaking "|" symbol randomly in the text. I'm using recommended temp, minP, topP and dry. But from time to time, model inserts "|" after each sentence.

zerofata

Owner Aug 2, 2025

Odd, I haven't noticed any leakage. What quant are you using to run the model? That's the first place I'd look for that sort of issue I think.

Might also be worth verifying you're using Mistral v7 Tekken template, and not accidentally running another template.

Hardeh

Aug 2, 2025

I'm using i1-Q5_K_M, and yes, using Mistral v7 template.

zerofata

Owner Aug 2, 2025

I'd still possibly give another quant a go.

https://huggingface.co/bartowski/zerofata_MS3.2-PaintedFantasy-v2-24B-GGUF

The only other thing I can think of is sampler settings, or something specific to the conversation. I gave the q5km quant from mradermacher a go for a few hours and wasn't able to reproduce the issue.

If you do keep getting the issue, a screenshot would be helpful.

neonr-0

Aug 5, 2025

Can confirm this, sometimes happen spontaneously to get "|" right at the end of sentence, on static Q6_K.
I fixed it with my implementation on python just removing when generations is done: OutText = OutText.replace("|", "") But this is more like temporary solution.

zerofata

Owner Aug 5, 2025

•

edited Aug 5, 2025

Hmm, that's two different quants and setups so I guess the issue has to be the model.

Thanks for the feedback, I'll keep an eye out for it.

zerofata

Owner Aug 5, 2025

•

edited Aug 5, 2025

Just a thought, what roleplay format do you use?

Actions: In plaintext
Dialogue: "In quotes"
Thoughts: In asterisks

Actions: In asterisks
Dialogue: "In quotes"
Thoughts: In backticks (or not at all)

The first set is what the model is trained on, but the second set is what tends to be common in RP. It's possible the format could be responsible for these tokens leaking, since it takes the model somewhat off distribution.

neonr-0

Aug 6, 2025

I used only first format:

Actions: In plaintext
Dialogue: "In quotes"
Thoughts: *In asterisks*

But at long conversations model prone to forget about *thoughts* or use them rarely. (I tried to force with OOC but it's like 30% success rate.) But when I add new character it does work fine for a while.
Other settings I use:

repetition_penalty: 1.0-1.05 
repeat_last_n: 2000
temperature: 0.8-1.2
top_p: 0.95
min_p:  0.05

Model works best at context up to ~10K, after that it's starting to lose "quality" and it's hard to get something from beginning of context. But maybe this is because of 6bit.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment