Instructions to use ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit")
model = AutoModelForCausalLM.from_pretrained("ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit

SGLang

How to use ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit with Docker Model Runner:
```
docker model run hf.co/ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit
```

This model has been quantized using GPTQModel.

bits: 4
group_size: 128
desc_act: true
static_groups: false
sym: true
lm_head: false
damp_percent: 0.01
true_sequential: true
model_name_or_path: ""
model_file_base_name: "model"
quant_method: "gptq"
checkpoint_format: "gptq"
meta：
- quantizer: "gptqmodel:0.9.9-dev0"

Example:

from transformers import AutoTokenizer
from gptqmodel import GPTQModel

model_name = "ModelCloud/Meta-Llama-3.1-405B-Instruct-gptq-4bit"

prompt = [{"role": "user", "content": "I am in Shanghai, preparing to visit the natural history museum. Can you tell me the best way to"}]

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = GPTQModel.from_quantized(model_name)

input_tensor = tokenizer.apply_chat_template(prompt, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_ids=input_tensor.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)

print(result)

lm-eval benchmark

|                 Tasks                 |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|---------------------------------------|------:|------|-----:|----------|---|-----:|---|-----:|
|arc_challenge                          |      1|none  |     0|acc       |↑  |0.5990|±  |0.0143|
|                                       |       |none  |     0|acc_norm  |↑  |0.6425|±  |0.0140|
|arc_easy                               |      1|none  |     0|acc       |↑  |0.8645|±  |0.0070|
|                                       |       |none  |     0|acc_norm  |↑  |0.8359|±  |0.0076|
|boolq                                  |      2|none  |     0|acc       |↑  |0.8820|±  |0.0056|
|hellaswag                              |      1|none  |     0|acc       |↑  |0.6560|±  |0.0047|
|                                       |       |none  |     0|acc_norm  |↑  |0.8446|±  |0.0036|
|lambada_openai                         |      1|none  |     0|acc       |↑  |0.7252|±  |0.0062|
|                                       |       |none  |     0|perplexity|↓  |3.5576|±  |0.0880|
|mmlu                                   |      1|none  |      |acc       |↑  |0.8245|±  |0.0031|
| - humanities                          |      1|none  |      |acc       |↑  |0.7892|±  |0.0058|
|  - formal_logic                       |      0|none  |     0|acc       |↑  |0.6349|±  |0.0431|
|  - high_school_european_history       |      0|none  |     0|acc       |↑  |0.8667|±  |0.0265|
|  - high_school_us_history             |      0|none  |     0|acc       |↑  |0.9314|±  |0.0177|
|  - high_school_world_history          |      0|none  |     0|acc       |↑  |0.9367|±  |0.0158|
|  - international_law                  |      0|none  |     0|acc       |↑  |0.9091|±  |0.0262|
|  - jurisprudence                      |      0|none  |     0|acc       |↑  |0.8796|±  |0.0315|
|  - logical_fallacies                  |      0|none  |     0|acc       |↑  |0.8834|±  |0.0252|
|  - moral_disputes                     |      0|none  |     0|acc       |↑  |0.8295|±  |0.0202|
|  - moral_scenarios                    |      0|none  |     0|acc       |↑  |0.7888|±  |0.0137|
|  - philosophy                         |      0|none  |     0|acc       |↑  |0.8521|±  |0.0202|
|  - prehistory                         |      0|none  |     0|acc       |↑  |0.8735|±  |0.0185|
|  - professional_law                   |      0|none  |     0|acc       |↑  |0.6760|±  |0.0120|
|  - world_religions                    |      0|none  |     0|acc       |↑  |0.8830|±  |0.0246|
| - other                               |      1|none  |      |acc       |↑  |0.8539|±  |0.0060|
|  - business_ethics                    |      0|none  |     0|acc       |↑  |0.8100|±  |0.0394|
|  - clinical_knowledge                 |      0|none  |     0|acc       |↑  |0.8679|±  |0.0208|
|  - college_medicine                   |      0|none  |     0|acc       |↑  |0.7688|±  |0.0321|
|  - global_facts                       |      0|none  |     0|acc       |↑  |0.7000|±  |0.0461|
|  - human_aging                        |      0|none  |     0|acc       |↑  |0.8341|±  |0.0250|
|  - management                         |      0|none  |     0|acc       |↑  |0.8932|±  |0.0306|
|  - marketing                          |      0|none  |     0|acc       |↑  |0.9444|±  |0.0150|
|  - medical_genetics                   |      0|none  |     0|acc       |↑  |0.9300|±  |0.0256|
|  - miscellaneous                      |      0|none  |     0|acc       |↑  |0.9425|±  |0.0083|
|  - nutrition                          |      0|none  |     0|acc       |↑  |0.8987|±  |0.0173|
|  - professional_accounting            |      0|none  |     0|acc       |↑  |0.6773|±  |0.0279|
|  - professional_medicine              |      0|none  |     0|acc       |↑  |0.9228|±  |0.0162|
|  - virology                           |      0|none  |     0|acc       |↑  |0.5542|±  |0.0387|
| - social sciences                     |      1|none  |      |acc       |↑  |0.8833|±  |0.0057|
|  - econometrics                       |      0|none  |     0|acc       |↑  |0.7193|±  |0.0423|
|  - high_school_geography              |      0|none  |     0|acc       |↑  |0.9394|±  |0.0170|
|  - high_school_government_and_politics|      0|none  |     0|acc       |↑  |0.9741|±  |0.0115|
|  - high_school_macroeconomics         |      0|none  |     0|acc       |↑  |0.8615|±  |0.0175|
|  - high_school_microeconomics         |      0|none  |     0|acc       |↑  |0.9412|±  |0.0153|
|  - high_school_psychology             |      0|none  |     0|acc       |↑  |0.9321|±  |0.0108|
|  - human_sexuality                    |      0|none  |     0|acc       |↑  |0.8550|±  |0.0309|
|  - professional_psychology            |      0|none  |     0|acc       |↑  |0.8497|±  |0.0145|
|  - public_relations                   |      0|none  |     0|acc       |↑  |0.7636|±  |0.0407|
|  - security_studies                   |      0|none  |     0|acc       |↑  |0.8163|±  |0.0248|
|  - sociology                          |      0|none  |     0|acc       |↑  |0.9204|±  |0.0191|
|  - us_foreign_policy                  |      0|none  |     0|acc       |↑  |0.9300|±  |0.0256|
| - stem                                |      1|none  |      |acc       |↑  |0.7907|±  |0.0070|
|  - abstract_algebra                   |      0|none  |     0|acc       |↑  |0.5800|±  |0.0496|
|  - anatomy                            |      0|none  |     0|acc       |↑  |0.8296|±  |0.0325|
|  - astronomy                          |      0|none  |     0|acc       |↑  |0.9145|±  |0.0228|
|  - college_biology                    |      0|none  |     0|acc       |↑  |0.9236|±  |0.0222|
|  - college_chemistry                  |      0|none  |     0|acc       |↑  |0.5800|±  |0.0496|
|  - college_computer_science           |      0|none  |     0|acc       |↑  |0.7300|±  |0.0446|
|  - college_mathematics                |      0|none  |     0|acc       |↑  |0.5800|±  |0.0496|
|  - college_physics                    |      0|none  |     0|acc       |↑  |0.7157|±  |0.0449|
|  - computer_security                  |      0|none  |     0|acc       |↑  |0.8000|±  |0.0402|
|  - conceptual_physics                 |      0|none  |     0|acc       |↑  |0.8383|±  |0.0241|
|  - electrical_engineering             |      0|none  |     0|acc       |↑  |0.7931|±  |0.0338|
|  - elementary_mathematics             |      0|none  |     0|acc       |↑  |0.8730|±  |0.0171|
|  - high_school_biology                |      0|none  |     0|acc       |↑  |0.9161|±  |0.0158|
|  - high_school_chemistry              |      0|none  |     0|acc       |↑  |0.7685|±  |0.0297|
|  - high_school_computer_science       |      0|none  |     0|acc       |↑  |0.9600|±  |0.0197|
|  - high_school_mathematics            |      0|none  |     0|acc       |↑  |0.6556|±  |0.0290|
|  - high_school_physics                |      0|none  |     0|acc       |↑  |0.7086|±  |0.0371|
|  - high_school_statistics             |      0|none  |     0|acc       |↑  |0.7778|±  |0.0284|
|  - machine_learning                   |      0|none  |     0|acc       |↑  |0.7054|±  |0.0433|
|openbookqa                             |      1|none  |     0|acc       |↑  |0.3300|±  |0.0210|
|                                       |       |none  |     0|acc_norm  |↑  |0.4420|±  |0.0222|
|piqa                                   |      1|none  |     0|acc       |↑  |0.8188|±  |0.0090|
|                                       |       |none  |     0|acc_norm  |↑  |0.8308|±  |0.0087|
|rte                                    |      1|none  |     0|acc       |↑  |0.7220|±  |0.0270|
|truthfulqa_mc1                         |      2|none  |     0|acc       |↑  |0.4333|±  |0.0173|
|winogrande                             |      1|none  |     0|acc       |↑  |0.7656|±  |0.0119|

|      Groups      |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|------|------|------|---|-----:|---|-----:|
|mmlu              |      1|none  |      |acc   |↑  |0.8245|±  |0.0031|
| - humanities     |      1|none  |      |acc   |↑  |0.7892|±  |0.0058|
| - other          |      1|none  |      |acc   |↑  |0.8539|±  |0.0060|
| - social sciences|      1|none  |      |acc   |↑  |0.8833|±  |0.0057|
| - stem           |      1|none  |      |acc   |↑  |0.7907|±  |0.0070|

Downloads last month: 4

Safetensors

Model size

410B params

Tensor type

BF16

I32