Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

4596d1a

verified ·

1 Parent(s): 9504409

a96e90a1dbbc338a4e66b273d4873e5167379a3263bf76104d43a985649d8259

Browse files

Files changed (21) hide show

checkpoint/pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p269.model.layers.29.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p278.model.layers.30.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p281.model.layers.31.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p282.model.layers.31.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p283.model.layers.31.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p284.model.layers.31.mlp.gate_proj.weight +3 -0

checkpoint/pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3aa202889a2cfc15d140cb437a0b754e74d3bc4693a88cc44e1115a21f54bb8c
+size 234881916

checkpoint/pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:00258529afc816221cb5aa6dc971ee3420e1b1b3f949cc1da6e4f5e36cf9a29a
+size 234881910

checkpoint/pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a74d9de5acde4ddb836cdc8f560d9922477bb72a55890c3a26c272572ed66fc6
+size 234881916

checkpoint/pytorch_model.bin/p269.model.layers.29.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd73a34710c73d3c4416a73f14cf2300c5b1235061d2ba015305bd49ca2798ca
+size 17282

checkpoint/pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aae15eff10955f27823a51d6a036be0074dbad879defd61baf9a86233c3f76de
+size 17303

checkpoint/pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fd62c553a51635c19aeca64a64738b118aee0989704b4ab2de6bd4d01ceb6d48
+size 17309

checkpoint/pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4fceec2734a4ecad40e44693fbe1958dbd1ac495f260d58728862aa0cb4c0f45
+size 67109765

checkpoint/pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8da159abbc568b21aa49bbc2cb9a3779b2bc036c40555d5f4e91dd6bb02d224d
+size 16778117

checkpoint/pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:37658ab507a0b47e2095c4ff3f0f747022952a9f7cea10dd7df458db290dd065
+size 16778117

checkpoint/pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:22e2644ff0505b2887a21480d068a728ed0d256864158720bdd5febc44cdb85a
+size 67109765

checkpoint/pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd733d0d0b3c1029ed1c32d2bf90c475fc853d9d006cccd2ff9f49dfb09ce7da
+size 234881916

checkpoint/pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9180678a8901411040962be2882fdc8d7153cb23196a6a80737dc477146bc0c5
+size 234881910

checkpoint/pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b70009436ef8bc3dee70632d0af693cdafd1f6e2a4625513e556991519cd63c0
+size 234881916

checkpoint/pytorch_model.bin/p278.model.layers.30.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c219d2f7e1fb928e4619a952b206e9481dc2913b3022e3e9ffdecf64614a04c5
+size 17282

checkpoint/pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1e27232ef2a91d730e2fddd95379a2c850ee590c2c850e26c8ca34074b3aa06f
+size 17309

checkpoint/pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:23b7f1eda78c73ba9a662c85bd3594c03ea4c60c4fd42f7038f6045d03f4dd89
+size 67109759

checkpoint/pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ec5b7a4b3cf70541d4de3d103a52aa79348d33a65d3a4be6d3a56f95fb03491b
+size 67109765

checkpoint/pytorch_model.bin/p281.model.layers.31.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e4cf8a4691b6416e872dc8b2e0076b6d4022ab42a030e3c981616e84368f08b2
+size 16778117

checkpoint/pytorch_model.bin/p282.model.layers.31.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3f6f3370f3f258c00e59c9455dd843a513e6a17b14433630201ab9e93af1e242
+size 16778117

checkpoint/pytorch_model.bin/p283.model.layers.31.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:18f2add2d0f5399998362e4a9ac71748a313d042c71f164c01f5d3069ef35906
+size 67109765

checkpoint/pytorch_model.bin/p284.model.layers.31.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:79a348b7e31ea6fed0bc1f5646540055df050b89d11f7c531a947f10883110e8
+size 234881916