Instructions to use martinsu/tildeopen-30b-mu-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use martinsu/tildeopen-30b-mu-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="martinsu/tildeopen-30b-mu-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("martinsu/tildeopen-30b-mu-instruct")
model = AutoModelForCausalLM.from_pretrained("martinsu/tildeopen-30b-mu-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use martinsu/tildeopen-30b-mu-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "martinsu/tildeopen-30b-mu-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "martinsu/tildeopen-30b-mu-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/martinsu/tildeopen-30b-mu-instruct

SGLang

How to use martinsu/tildeopen-30b-mu-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "martinsu/tildeopen-30b-mu-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "martinsu/tildeopen-30b-mu-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "martinsu/tildeopen-30b-mu-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "martinsu/tildeopen-30b-mu-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use martinsu/tildeopen-30b-mu-instruct with Docker Model Runner:
```
docker model run hf.co/martinsu/tildeopen-30b-mu-instruct
```

gguf

by KnutJaegersberg - opened Feb 16

Discussion

KnutJaegersberg

Feb 16

would help distribution to make a gguf version as well. I tried with the gguf my repo tool but it throws an exception. thought it would just work as it's a llama.

martinsu

Owner Feb 23

•

edited Feb 23

Since this fine tune is classic one - no new added layers etc, its 1:1 original(except weights and tokenizer content + embeddings), i need to look how unsloth did it, or maybe we can ask them politely to convert this fine tune. Also i have new RAG finetuned version of this model, ill upload when ill have spare time, its way better at following instructions.

maigonis

Mar 6

I doubt they will add this to their catalog, but can assist whit converting.

Im still waiting for something smaller, running 30B dense model on consumer hardware is not a easy task, MoE sure.

martinsu

Owner Mar 7

I run this model in workflow/agentic environment where humans are waiting for output on VLLM 4xA40 GPU node. So thats not a consumer hardware.

Yes, pushing consumer hardware limits on these models are quite an adventure alas.

gieza

Apr 17

I have found that conversion to gguf when using llamma.cpp convert_hf_to_gguf.py is failing due to unrecognized 'no' in Readme.md

language: [en, de, fr, es, it, pt, nl, pl, lv, et, lt, cs, sk, ro, bg, sl, hr, sv, da, fi, hu, uk, ru, zh, hi, ja, ko, el, no]

Probably should be:

language: [en, de, fr, es, it, pt, nl, pl, lv, et, lt, cs, sk, ro, bg, sl, hr, sv, da, fi, hu, uk, ru, zh, hi, ja, ko, el, nb]

martinsu

Owner Apr 17

I have found that conversion to gguf when using llamma.cpp convert_hf_to_gguf.py is failing due to unrecognized 'no' in Readme.md
language: [en, de, fr, es, it, pt, nl, pl, lv, et, lt, cs, sk, ro, bg, sl, hr, sv, da, fi, hu, uk, ru, zh, hi, ja, ko, el, no]
Probably should be:
language: [en, de, fr, es, it, pt, nl, pl, lv, et, lt, cs, sk, ro, bg, sl, hr, sv, da, fi, hu, uk, ru, zh, hi, ja, ko, el, nb]

Thank You! Easy fix!

KnutJaegersberg

Apr 19

Cool. That made it work I guess now downloading a guff

https://huggingface.co/KnutJaegersberg/tildeopen-30b-mu-instruct-Q8_0-GGUF
https://huggingface.co/KnutJaegersberg/tildeopen-30b-mu-instruct-Q4_K_M-GGUF

KnutJaegersberg changed discussion status to closed Apr 19

martinsu

Owner Apr 20

•

edited Apr 20

I will perhaps check out how tokenizer behaves in this conversion - it uses slow one or something else, since even if its broken, it sort of predicts somewhat plausible, but degraded output tokens.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment