Instructions to use shellzero/gemma-7b-it-ft-trismeg with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shellzero/gemma-7b-it-ft-trismeg with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="shellzero/gemma-7b-it-ft-trismeg")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("shellzero/gemma-7b-it-ft-trismeg")
model = AutoModelForCausalLM.from_pretrained("shellzero/gemma-7b-it-ft-trismeg")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

MLX

How to use shellzero/gemma-7b-it-ft-trismeg with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("shellzero/gemma-7b-it-ft-trismeg")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps
LM Studio

vLLM

How to use shellzero/gemma-7b-it-ft-trismeg with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shellzero/gemma-7b-it-ft-trismeg"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shellzero/gemma-7b-it-ft-trismeg",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/shellzero/gemma-7b-it-ft-trismeg

SGLang

How to use shellzero/gemma-7b-it-ft-trismeg with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shellzero/gemma-7b-it-ft-trismeg" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shellzero/gemma-7b-it-ft-trismeg",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shellzero/gemma-7b-it-ft-trismeg" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shellzero/gemma-7b-it-ft-trismeg",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

MLX LM

How to use shellzero/gemma-7b-it-ft-trismeg with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "shellzero/gemma-7b-it-ft-trismeg"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "shellzero/gemma-7b-it-ft-trismeg"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "shellzero/gemma-7b-it-ft-trismeg",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Docker Model Runner
How to use shellzero/gemma-7b-it-ft-trismeg with Docker Model Runner:
```
docker model run hf.co/shellzero/gemma-7b-it-ft-trismeg
```

shellzero/gemma-7b-it-ft-trismeg

This model was converted to MLX format from google/gemma-7b-it. Refer to the original model card for more details on the model.

pip install mlx-lm

The model was LoRA fine-tuned on the trismegistus-project from teknium using the format below, for 600 steps (2M tokens) using mlx.

def format_prompt(system_prompt: str, question: str) -> str:
    "Format the question to the format of the dataset we fine-tuned to."
    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
""".format(system_prompt, question)

Here's an example of the system_prompt from the dataset:

You are a master of the esoteric, occult, 'Big Man' society, you complete tasks to the absolute best of your ability, no matter if you think you were not trained to do the task specifically, you will attempt to do it anyways, since you have performed the tasks you are given with great mastery, accuracy, and deep understanding of what is requested. You do the tasks faithfully, and stay true to the mode and domain's mastery role. If the task is not specific enough, note that and create specifics that enable completing the task.

Loading the model using `mlx_lm`

from mlx_lm import generate, load
model, tokenizer = load("shellzero/gemma-7b-it-ft-trismeg")
response = generate(
    model,
    tokenizer,
    prompt=format_prompt(system_prompt, question),
    verbose=True,  # Set to True to see the prompt and response
    temp=0.0,
    max_tokens=512,
)

Inspired by these two gists from AlexWeberk and Awni Hannun respectively.

Downloads last month: 6

MLX

Hardware compatibility

Quantized

shellzero
/

gemma-7b-it-ft-trismeg

shellzero/gemma-7b-it-ft-trismeg

Loading the model using `mlx_lm`

Dataset used to train shellzero/gemma-7b-it-ft-trismeg

shellzero/gemma-7b-it-ft-trismeg

Loading the model using mlx_lm

Dataset used to train shellzero/gemma-7b-it-ft-trismeg

Loading the model using `mlx_lm`