Instructions to use d-rang-d/MS3-RP-Broth-24B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use d-rang-d/MS3-RP-Broth-24B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="d-rang-d/MS3-RP-Broth-24B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("d-rang-d/MS3-RP-Broth-24B")
model = AutoModelForCausalLM.from_pretrained("d-rang-d/MS3-RP-Broth-24B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use d-rang-d/MS3-RP-Broth-24B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "d-rang-d/MS3-RP-Broth-24B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d-rang-d/MS3-RP-Broth-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/d-rang-d/MS3-RP-Broth-24B

SGLang

How to use d-rang-d/MS3-RP-Broth-24B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "d-rang-d/MS3-RP-Broth-24B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d-rang-d/MS3-RP-Broth-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "d-rang-d/MS3-RP-Broth-24B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d-rang-d/MS3-RP-Broth-24B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use d-rang-d/MS3-RP-Broth-24B with Docker Model Runner:
```
docker model run hf.co/d-rang-d/MS3-RP-Broth-24B
```

undownloadable

by mradermacher - opened Mar 3, 2025

Discussion

mradermacher

Mar 3, 2025

Hi, one or more files are undownloadable (likely due to a HF bug): model-00001-of-00010.safetensors

The only solution I know how to fix this is to delete and re-upload the files, or even the whole repository.

Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/0a/47/0a47f0613b0b71c1724baf6567497d040f010527c511ce1848acb493c6af5f84/049938712487903faf3344ec405e0516885d4b84826d1fad0b6943bb8599cba4?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00010.safetensors%3B+filename%3D%22model-00001-of-00010.safetensors%22%3B&Expires=1741010447&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc0MTAxMDQ0N319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzBhLzQ3LzBhNDdmMDYxM2IwYjcxYzE3MjRiYWY2NTY3NDk3ZDA0MGYwMTA1MjdjNTExY2UxODQ4YWNiNDkzYzZhZjVmODQvMDQ5OTM4NzEyNDg3OTAzZmFmMzM0NGVjNDA1ZTA1MTY4ODVkNGI4NDgyNmQxZmFkMGI2OTQzYmI4NTk5Y2JhND9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=GOAyntakCH-ODHyBDT36WD474~UHjwNoh-te~i3XLcBW-NI8N1BqrBXi1GSyepdwXdXXB92rUKjszqNGx3sl2KffnmbgJ9n7Qe4zGEuREGdvwocRd4DX8UMzjAaSHTn2XNsQqeSxWgzHIIPME2qObCTY3fia-CDVabWYxcUVJLY-ipy5Gd1o3-2mqcBRYWrF012UWGP3xHS1AUe7JdtE~uT6GIZyOcbWHYKmxrIIGUFOKsi-0qbxJxq3px~qONjdBhb7rCrsjT64m-mHMQ1t0m9p5P-LVskUQfBgmieIrNdR3WJaaLyQTnEBfE4qbO1oZEbV2RO6cFYWpDHFRWbh~Q__&Key-Pair-Id=K24J24Z295AEI9

Nohobby

D-rang-d labs org Mar 3, 2025

Hi!
This is the MS-RP-whole model that you've already quanted. I just duplicated it from mergekit-community to my org so it can have some info in the model card. The error is to be expected since I assume the files in the repo are just pointers with no actual content.
Thanks for letting me know, and sorry for the confusion. I'll add the link to the actual model files in the readme.

mradermacher

Mar 3, 2025

•

edited Mar 3, 2025

That is good to know - but depending on how you duplicated it, the files should work. In fact, I don't think hf would allow to have lfs pointer files without them pointing at actual storage, but that might have been the case. It's also an interesting data point - I regularly stumble over undownloadable files, even when people have uploaded files manually - but then it's usually not just the first lfs file, but some random file(s).

Anyway, thanks for this clarification - I'll not be waiting for the files to appear then :)

mradermacher changed discussion status to closed Mar 3, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment