Instructions to use ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16")
model = AutoModelForCausalLM.from_pretrained("ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16

SGLang

How to use ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16 with Docker Model Runner:
```
docker model run hf.co/ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16
```

Repetitive generation without additional EOS token

by amrothemich - opened Jun 24, 2024

Discussion

amrothemich

Jun 24, 2024

Hi! The generation_config supplied will generate indefinitely in a chat setting and repeat itself, because '<|end_of_text|>' is rarely generated. It should work better if <|eot_id|> is added, which is generated at the end of every chat response.

Here's the config that meta supplies to cover both cases:

{
  "bos_token_id": 128000,
  "eos_token_id": [128001, 128009],
  "do_sample": true,
  "temperature": 0.6,
  "max_length": 4096,
  "top_p": 0.9,
  "transformers_version": "4.40.0.dev0"
}

https://huggingface.co/ISTA-DASLab/Meta-Llama-3-70B-Instruct-AQLM-2Bit-1x16/blob/main/generation_config.json

BlackSamorez

IST Austria Distributed Algorithms and Systems Lab org Jun 24, 2024

Isn't <|eot_id|> set as eos_token already? Look here.

amrothemich

Jun 25, 2024

My understanding (guess work, haven't looked at the code/documentation) is that the generation config separately specifies the eos token so it knows when to stop generation. And in the generation_config for this model, it's specified as 128001, which is never really generated. Tokenizer has the real EOS token so it knows what to append to a tokenized sequence, but generation needs to have the more "intermediate" stop token to indicate the end of a particular response (but not necessarily the end of the whole conversation).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment