Instructions to use ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit")
model = AutoModelForCausalLM.from_pretrained("ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit

SGLang

How to use ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit with Docker Model Runner:
```
docker model run hf.co/ISTA-DASLab/Llama-3.3-70B-Instruct-HIGGS-4bit
```

running model issue

by 0kd637 - opened May 2, 2025

Discussion

0kd637

May 2, 2025

I suggest I insatlled all required packages and running this script:
"
import torch
from transformers.utils.quantization_config import HiggsConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline, utils

model_path = ".../Llama-3.3-70B-Instruct-HIGGS-4bit"

model = AutoModelForCausalLM.from_pretrained(
model_path,
quantization_config=HiggsConfig(bits=4)
device_map="auto",
)

model = torch.compile(model)
tokenizer = AutoTokenizer.from_pretrained(model_path)
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)

print("Chat with your local model! Type 'exit' to quit.")
while True:
user_input = input("You: ")
if user_input.lower() in ["exit", "quit"]:
break

You may customize prompt formatting depending on the model’s instruction tuning

prompt = f"user\n{user_input}\nmodel\n"

output = generator(prompt, max_new_tokens=256, do_sample=True, temperature=0.7)
print("LLM:", output[0]["generated_text"][len(prompt):].strip())
"
gets me error:
"

This IS expected if you are initializing LlamaForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing LlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
File "/home/user/src/envTorch/run_my.py", line 20, in
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/src/envTorch/transformers/src/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/src/envTorch/transformers/src/transformers/modeling_utils.py", line 280, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/src/envTorch/transformers/src/transformers/modeling_utils.py", line 4595, in from_pretrained
hf_quantizer.postprocess_model(model, config=config)
File "/home/user/src/envTorch/transformers/src/transformers/quantizers/base.py", line 238, in postprocess_model
return self._process_model_after_weight_loading(model, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/src/envTorch/transformers/src/transformers/quantizers/quantizer_higgs.py", line 153, in _process_model_after_weight_loading
module.tune_metadata = TuneMetaData.from_dict(self.quantization_config.tune_metadata[name])

KeyError: 'model.layers.0.self_attn.q_proj'
"

i tried to overcome it with different "tune_metadata" settings tricks, but no luck.

Can u please consider what's wrong in my script?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment