Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

7467c3d

verified ·

1 Parent(s): 45af329

aaeb82add04961495089fc545338f1c056af544f0a24e53effb9f6b94679a446

Browse files

Files changed (19) hide show

checkpoint/pytorch_model.bin/p6.model.layers.0.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p60.model.layers.6.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p61.model.layers.6.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p62.model.layers.6.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p63.model.layers.6.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p64.model.layers.7.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p65.model.layers.7.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p66.model.layers.7.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p67.model.layers.7.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p68.model.layers.7.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p69.model.layers.7.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p7.model.layers.0.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p70.model.layers.7.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p71.model.layers.7.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p72.model.layers.7.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p73.model.layers.8.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p74.model.layers.8.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p75.model.layers.8.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p76.model.layers.8.self_attn.o_proj.weight +3 -0

checkpoint/pytorch_model.bin/p6.model.layers.0.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:02362172434e5ecf478cb93802e22dd6962ff11ac70480f5b812e0e0d7551b9d
+size 234881901

checkpoint/pytorch_model.bin/p60.model.layers.6.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8a217eefe81f1a4c1f0f36cc762c7e5311c2033b89f3d26fa2e7c4a128d9eecd
+size 234881904

checkpoint/pytorch_model.bin/p61.model.layers.6.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aaae9691a3abf40b370ae1c0a941a737389e3732cf33489b961221ab1c936711
+size 234881910

checkpoint/pytorch_model.bin/p62.model.layers.6.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:657b81cc492deb34c056c3e23e8475858f13b1744ec1c916a3b7cdb5ab8bd43a
+size 17276

checkpoint/pytorch_model.bin/p63.model.layers.6.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b345d5243120dd196ac41fa1208570666530f6d72546f8b208b197919cbd9f70
+size 17303

checkpoint/pytorch_model.bin/p64.model.layers.7.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:36b7f320d15d2370c80503ee851ae4c6cc83b0778bedb4661bea208564ee66f8
+size 67109759

checkpoint/pytorch_model.bin/p65.model.layers.7.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:46889cb72ae338fcb0f3f42d5d535caf10da28380848241990ade7c4e789b723
+size 16778111

checkpoint/pytorch_model.bin/p66.model.layers.7.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:673161507654166d3f551813a8642060af854caa58bda6a33837fbd937e0262d
+size 16778111

checkpoint/pytorch_model.bin/p67.model.layers.7.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d007af3ec3ccbe67911ebff3bb689d7c676da6aa141275964c704a3785852026
+size 67109759

checkpoint/pytorch_model.bin/p68.model.layers.7.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:50a8f33dc2c5a96d71622087d629402cc86269b2daeb2e7a6494bd7caf0135fe
+size 234881910

checkpoint/pytorch_model.bin/p69.model.layers.7.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52dd5115cbfb30721420558b19ec6f3461de6a221dbf0391f4e8e85df7f6ba86
+size 234881904

checkpoint/pytorch_model.bin/p7.model.layers.0.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:56f34724aae4adbffe0b5bec9ae3685e045d3123b6abcc2423da2676ecd514ea
+size 234881907

checkpoint/pytorch_model.bin/p70.model.layers.7.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f888c34e57183a757b7bbf406a6598c4beaf4e24688f28208bba22fe2244bdc5
+size 234881910

checkpoint/pytorch_model.bin/p71.model.layers.7.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f1eaf89e7f0dedb6666cf0e805c01459e1631c80dd775a772f97403f07c597cb
+size 17276

checkpoint/pytorch_model.bin/p72.model.layers.7.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:380dadfcedfd45cd38ae6161068b4de678db51db8272a28cebbcb9d198ec52b3
+size 17303

checkpoint/pytorch_model.bin/p73.model.layers.8.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6335526771d89b1b9a1368bd8631dfc452365e5bdc594094cf4aa55c1e1f70f9
+size 67109759

checkpoint/pytorch_model.bin/p74.model.layers.8.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:764cdadb009fe397da298bb0ce78e26b6af19a74a58c2d77664792b01d427e47
+size 16778111

checkpoint/pytorch_model.bin/p75.model.layers.8.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fc646c54daf7cf25258ed497affecd064507bb77760b6d85fbe5daf4ec0b8bbe
+size 16778111

checkpoint/pytorch_model.bin/p76.model.layers.8.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d9905dd0a8ca84f3c40081021e7efe08ba64f2e213f68d22f1e06e220ecf5409
+size 67109759