Instructions to use aws-neuron/CodeLlama-7b-hf-neuron-8xlarge with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/CodeLlama-7b-hf-neuron-8xlarge with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/CodeLlama-7b-hf-neuron-8xlarge")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/CodeLlama-7b-hf-neuron-8xlarge")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/CodeLlama-7b-hf-neuron-8xlarge")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/CodeLlama-7b-hf-neuron-8xlarge with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/CodeLlama-7b-hf-neuron-8xlarge

SGLang

How to use aws-neuron/CodeLlama-7b-hf-neuron-8xlarge with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/CodeLlama-7b-hf-neuron-8xlarge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/CodeLlama-7b-hf-neuron-8xlarge with Docker Model Runner:
```
docker model run hf.co/aws-neuron/CodeLlama-7b-hf-neuron-8xlarge
```

CodeLlama-7b-hf-neuron-8xlarge / .gitattributes

jburtoft

Upload folder using huggingface_hub (#4)

e5bff58 verified over 2 years ago

Raw

History Blame Contribute Delete

2.94 kB

	*.7z filter=lfs diff=lfs merge=lfs -text
	*.arrow filter=lfs diff=lfs merge=lfs -text
	*.bin filter=lfs diff=lfs merge=lfs -text
	*.bz2 filter=lfs diff=lfs merge=lfs -text
	*.ckpt filter=lfs diff=lfs merge=lfs -text
	*.ftz filter=lfs diff=lfs merge=lfs -text
	*.gz filter=lfs diff=lfs merge=lfs -text
	*.h5 filter=lfs diff=lfs merge=lfs -text
	*.joblib filter=lfs diff=lfs merge=lfs -text
	.lfs. filter=lfs diff=lfs merge=lfs -text
	*.mlmodel filter=lfs diff=lfs merge=lfs -text
	*.model filter=lfs diff=lfs merge=lfs -text
	*.msgpack filter=lfs diff=lfs merge=lfs -text
	*.npy filter=lfs diff=lfs merge=lfs -text
	*.npz filter=lfs diff=lfs merge=lfs -text
	*.onnx filter=lfs diff=lfs merge=lfs -text
	*.ot filter=lfs diff=lfs merge=lfs -text
	*.parquet filter=lfs diff=lfs merge=lfs -text
	*.pb filter=lfs diff=lfs merge=lfs -text
	*.pickle filter=lfs diff=lfs merge=lfs -text
	*.pkl filter=lfs diff=lfs merge=lfs -text
	*.pt filter=lfs diff=lfs merge=lfs -text
	*.pth filter=lfs diff=lfs merge=lfs -text
	*.rar filter=lfs diff=lfs merge=lfs -text
	*.safetensors filter=lfs diff=lfs merge=lfs -text
	saved_model/*/ filter=lfs diff=lfs merge=lfs -text
	.tar. filter=lfs diff=lfs merge=lfs -text
	*.tar filter=lfs diff=lfs merge=lfs -text
	*.tflite filter=lfs diff=lfs merge=lfs -text
	*.tgz filter=lfs diff=lfs merge=lfs -text
	*.wasm filter=lfs diff=lfs merge=lfs -text
	*.xz filter=lfs diff=lfs merge=lfs -text
	*.zip filter=lfs diff=lfs merge=lfs -text
	*.zst filter=lfs diff=lfs merge=lfs -text
	tfevents filter=lfs diff=lfs merge=lfs -text
	compiled/13f69de924bfe5a0f650.neff filter=lfs diff=lfs merge=lfs -text
	compiled/24a88b503bb13eebc633.neff filter=lfs diff=lfs merge=lfs -text
	compiled/4d315c567ee64f5c3c52.neff filter=lfs diff=lfs merge=lfs -text
	compiled/91d9e9131e21917fb676.neff filter=lfs diff=lfs merge=lfs -text
	compiled/a2673088cd059202b6c2.neff filter=lfs diff=lfs merge=lfs -text
	compiled/b480b1deeb6e4c40c9d4.neff filter=lfs diff=lfs merge=lfs -text
	compiled/b59dca76c353105042b2.neff filter=lfs diff=lfs merge=lfs -text
	compiled/d9a9b3aab1095d21dc94.neff filter=lfs diff=lfs merge=lfs -text
	compiled/e6af685a35c321de6779.neff filter=lfs diff=lfs merge=lfs -text
	compiled/ee57f53ab745d1cbe07e.neff filter=lfs diff=lfs merge=lfs -text
	compiled/22c9e8c61931750704e4.neff filter=lfs diff=lfs merge=lfs -text
	compiled/6e28cdc209d01cf80749.neff filter=lfs diff=lfs merge=lfs -text
	compiled/79e2d4e60d54467c6d8c.neff filter=lfs diff=lfs merge=lfs -text
	compiled/89ecf163d454a64c0c00.neff filter=lfs diff=lfs merge=lfs -text
	compiled/a433e75bc0664453b177.neff filter=lfs diff=lfs merge=lfs -text
	compiled/be670fb677a32755b5fc.neff filter=lfs diff=lfs merge=lfs -text
	compiled/bed5751f3dea8390f0f6.neff filter=lfs diff=lfs merge=lfs -text
	compiled/d4a2099a3da7ae458edf.neff filter=lfs diff=lfs merge=lfs -text
	compiled/e2ed77001ff9f87f5f60.neff filter=lfs diff=lfs merge=lfs -text
	compiled/e497ac4eb1b05d80f918.neff filter=lfs diff=lfs merge=lfs -text