Instructions to use aws-neuron/CodeLlama-7b-hf-neuron-24xlarge with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/CodeLlama-7b-hf-neuron-24xlarge with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/CodeLlama-7b-hf-neuron-24xlarge")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/CodeLlama-7b-hf-neuron-24xlarge")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/CodeLlama-7b-hf-neuron-24xlarge")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use aws-neuron/CodeLlama-7b-hf-neuron-24xlarge with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/CodeLlama-7b-hf-neuron-24xlarge

SGLang

How to use aws-neuron/CodeLlama-7b-hf-neuron-24xlarge with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-24xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/CodeLlama-7b-hf-neuron-24xlarge with Docker Model Runner:
```
docker model run hf.co/aws-neuron/CodeLlama-7b-hf-neuron-24xlarge
```

jburtoft commited on Dec 28, 2023

Commit

9cf0443

1 Parent(s): 62d1d6a

2af91306f926c3e84b6effb1fcb8d9bcfe67dfce5560bffa96cf3a1ef76c24da

Browse files

Files changed (26) hide show

checkpoint/pytorch_model.bin/p258.model.layers.28.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p259.model.layers.28.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p26.model.layers.2.input_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p260.model.layers.28.input_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p261.model.layers.28.post_attention_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p262.model.layers.29.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p263.model.layers.29.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p264.model.layers.29.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p265.model.layers.29.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p269.model.layers.29.input_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p278.model.layers.30.input_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight +0 -0
checkpoint/pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight +3 -0

checkpoint/pytorch_model.bin/p258.model.layers.28.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b4552bbcd35a668e0d4bdec61b60f69ab1ca2919ca32511566a372101b4f5331
+size 180355958

checkpoint/pytorch_model.bin/p259.model.layers.28.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3c2e9f3e4f77188bcffb41c16edc71f347c1e2c067bbb8174c041f3e52fd88cd
+size 180355964

checkpoint/pytorch_model.bin/p26.model.layers.2.input_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p260.model.layers.28.input_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p261.model.layers.28.post_attention_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p262.model.layers.29.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be570d179045ba344f6f45258576733aa7df187d5557db973f4b11a8b3205599
+size 67109765

checkpoint/pytorch_model.bin/p263.model.layers.29.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bf7ece8858c13644ea983c8927d352a451dccd05d930a8f7d5be0773e7d5ff8a
+size 67109765

checkpoint/pytorch_model.bin/p264.model.layers.29.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:19ece12e729ef23392525eda319fc8da3381d3992def627c7d756fb66ce3f1f7
+size 67109765

checkpoint/pytorch_model.bin/p265.model.layers.29.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2caeb8dfca0167ec1e99f919466a78811a6f700babe681f50bd5ab15d1d214b
+size 67109765

checkpoint/pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c0bfb0cb047752f9218ede87e5cba4cf1b012261c387ebfa7aa9c0c9692ce931
+size 180355964

checkpoint/pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d380266d4ff687d39cddb2a16571c11c10420dcb5a97aaca7db6d7984b8253a9
+size 180355958

checkpoint/pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1ea294126a615fdfaae76ed77b677a8723a95b1725af73c66119f1a0d0c3be1f
+size 180355964

checkpoint/pytorch_model.bin/p269.model.layers.29.input_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0bceb618e8d770d9725c673d3df6b05f0cc797c3a8b9266a4f1703813e174bd7
+size 67109765

checkpoint/pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cd45e6fd42c91b11e30b2567c76986ca8a5fe55ae3dd37eaf26edc84a40b0d6c
+size 67109765

checkpoint/pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:02848d1ab5e3ea17743f74270e6f87c0a30f962328fa44011cbc65611bb77725
+size 67109765

checkpoint/pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:921c53531f90eb583c04bf5d24379b0beb14eabf746c76d4f5eb9d09d40379d6
+size 67109765

checkpoint/pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9971731580a321abb137729a8898e777ec3c923ef9d4025bd22a4d66abbb4c73
+size 180355964

checkpoint/pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06dbbec996653277d2817ae47ce3a7577fbabca1d4ff0bb08b94fd77951ac2eb
+size 180355958

checkpoint/pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bac43ce16ef6720fdac5ad736c64f7612671ceff45a63cc49c7fc6fa2148d90a
+size 180355964

checkpoint/pytorch_model.bin/p278.model.layers.30.input_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight ADDED Viewed

Binary file (17.3 kB). View file

checkpoint/pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2b525a770150317174677492109084700667780f95f42670fc8f4a80bedcda12
+size 67109759

checkpoint/pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:af8615d6b7c2b6793bd83d369d133efb798e8b642127e39d43f6779973abb52f
+size 67109765