Instructions to use MuXodious/GLM-4.7-Flash-absolute-heresy with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MuXodious/GLM-4.7-Flash-absolute-heresy with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MuXodious/GLM-4.7-Flash-absolute-heresy")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MuXodious/GLM-4.7-Flash-absolute-heresy")
model = AutoModelForCausalLM.from_pretrained("MuXodious/GLM-4.7-Flash-absolute-heresy", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MuXodious/GLM-4.7-Flash-absolute-heresy with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MuXodious/GLM-4.7-Flash-absolute-heresy"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/GLM-4.7-Flash-absolute-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/MuXodious/GLM-4.7-Flash-absolute-heresy

SGLang

How to use MuXodious/GLM-4.7-Flash-absolute-heresy with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MuXodious/GLM-4.7-Flash-absolute-heresy" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/GLM-4.7-Flash-absolute-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MuXodious/GLM-4.7-Flash-absolute-heresy" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/GLM-4.7-Flash-absolute-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use MuXodious/GLM-4.7-Flash-absolute-heresy with Docker Model Runner:
```
docker model run hf.co/MuXodious/GLM-4.7-Flash-absolute-heresy
```

Heretic Options?

by darkc0de - opened Feb 15

Discussion

darkc0de

Feb 15

Hey MuXodious!
I'm curious to know your preferred heretic options your using since v1.2.0

heretic --model USER/REPO
heretic --model USER/REPO --orthogonalize-direction
heretic --model USER/REPO --orthogonalize-direction --row-normalization full
heretic --model USER/REPO --orthogonalize-direction --row-normalization full --winsorization-quantile 0.995

Something different?

MuXodious

Owner Feb 15

Salut, the creator of the cult classic Xortron, darkc0de!

I almost exclusively use heretic --orthogonalize-direction --row-normalization full --winsorization-quantile 0.995 USER/REPO, adjusting windsor to a higher or lower value to see if it helps.

darkc0de

Feb 16

Check out the results of this run,

heretic --orthogonalize-direction --row-normalization full --winsorization-quantile 0.995 --model unsloth/GLM-4.7-Flash

MuXodious

Owner Feb 16

•

edited Feb 16

Now, those are what I refer to as *impotent heresy. Although, I also used the same set of arguments for GLM 4.7, it still remains a tad unique case. Its reasoning throws off the refusal detection mechanism and, as a result, the ablation optimisation algorithm. So, you effectively end up with a low initial refusal count and an impotent ablation. You should also keep in mind that not all models refuse the same. The default set of refusal markers is adequate, but not enough for all cases. Here's an example. You may have to study the model behaviours a bit to pinpoint its unique markers.

Is there a particular reason to use the unsloth version?

darkc0de

Feb 16

Over time I've grown to appreciate the small fixes unsloth applies to some models. Their quants top notch too

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment