Instructions to use meta-llama/Meta-Llama-3-8B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Meta-Llama-3-8B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Meta-Llama-3-8B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-llama/Meta-Llama-3-8B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Meta-Llama-3-8B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/meta-llama/Meta-Llama-3-8B-Instruct

SGLang

How to use meta-llama/Meta-Llama-3-8B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Meta-Llama-3-8B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Meta-Llama-3-8B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use meta-llama/Meta-Llama-3-8B-Instruct with Docker Model Runner:
```
docker model run hf.co/meta-llama/Meta-Llama-3-8B-Instruct
```

Requesting access

#10

by deleted - opened Apr 18, 2024

Discussion

deleted

Apr 18, 2024

I requested access both here in hf and at meta 2 ~hrs ago - also got the email from meta - but I still see the:
You need to agree to share your contact information to access this model
with below:
Your request to access this repo has been successfully submitted, and is pending a review from the repo's authors.

Is it normal and takes some time or am I missing something?

aviator1994

Apr 18, 2024

•

edited Apr 18, 2024

I've been waiting as well. Seems like it's normal, but I'm not sure how long it usually takes.

linlp22

Apr 19, 2024

Anyone success？

dmitrykvant

Apr 19, 2024

deleted

Apr 19, 2024

clarklikepc2100

Apr 21, 2024

plus one

outmanzhao

Apr 22, 2024

plus one

par4dox

Apr 22, 2024

bonni93

Apr 22, 2024

Still had no access after a few days - I ended up deleting my hf account and recreating it.
With the new account, I then first requested access from Meta and then here at hf.
I got access here eventually after a few minutes.
Try also your institutional email, it might work better.

JoKoSt

Apr 23, 2024

After requesting the llama3 model I got a Mail approving acces for "meta-llama/Meta-Llama-Guard-2-8B" instead.. I also requested permission at meta directly (which are approved as well) but when trying to download the model via HF i still get "Repo model meta-llama/Meta-Llama-3-8B-Instruct is gated. You must be authenticated to access it"
Maybe this helps to narrow down the error here...

darsh-essential

Apr 23, 2024

I requested llama3 model too, but I got an email approving access for "meta-llama/Meta-LLama-Guard-2-8B" instead.

tsudhakar112

Apr 25, 2024

•

edited Apr 25, 2024

while requesting for LLama model , it is giving download link in mail . but how can we access it by huggingface ? , in huggingface still showing no permission.

vinooj

Apr 28, 2024

•

edited Apr 28, 2024

( Resolved) I'm new to hugging face. And I'm wondering how do I request access to meta-llama/Meta-Llama-3-8B-Instruct? Ans: Click the Requesting Access. It took me a while to figure out :-)

kysc

Apr 29, 2024

I requested Llama3 using my personal email and got an email for "meta-llama/Meta-LLama-Guard-2-8B" within 30min. But it's actually the access for "meta-llama/Meta-Llama-3-8B-Instruct"