Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

050652e

verified ·

1 Parent(s): 4596d1a

67c2d0c8bca4f0af46228fa2df85d4a9ad8e898e40b1c55adc326ca5e1f9ae80

Browse files

Files changed (22) hide show

checkpoint/pytorch_model.bin/p285.model.layers.31.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p286.model.layers.31.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p287.model.layers.31.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p288.model.layers.31.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p289.model.layers.32.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p29.model.layers.3.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p290.model.layers.32.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p291.model.layers.32.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p292.model.layers.32.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p293.model.layers.32.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p294.model.layers.32.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p295.model.layers.32.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p296.model.layers.32.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p297.model.layers.32.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p298.model.layers.33.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p299.model.layers.33.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p3.model.layers.0.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p30.model.layers.3.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p300.model.layers.33.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p301.model.layers.33.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p302.model.layers.33.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p303.model.layers.33.mlp.up_proj.weight +3 -0

checkpoint/pytorch_model.bin/p285.model.layers.31.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c1574a84df1999e82b414ccfed7c8b150a74cec1fa1255a0526983e4d4386b52
+size 234881910

checkpoint/pytorch_model.bin/p286.model.layers.31.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f36bff75c356fccb8967faf960ebd191009cf866a0cca9a4001e116974379d71
+size 234881916

checkpoint/pytorch_model.bin/p287.model.layers.31.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:030e8990d217f545e5fa73c3165bfe3dbad6495ca703d7eb02ee2f8fda6de6fd
+size 17282

checkpoint/pytorch_model.bin/p288.model.layers.31.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:675c230e666bedab4987bc8d9ad64ac06fd92e8f6c37673109c3c47133ce6a51
+size 17309

checkpoint/pytorch_model.bin/p289.model.layers.32.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6887a9ce24efc793920eab1a213ca8ef611420c8a91b964a16b3302844091b0b
+size 67109765

checkpoint/pytorch_model.bin/p29.model.layers.3.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:db9b127ecb1f1e591fa980e9f7bc61ce667769be4b4b0caac202a153132d832e
+size 16778111

checkpoint/pytorch_model.bin/p290.model.layers.32.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b89c2aeac84378696b27fc81bd726905e36ae957f35c8772567515fda9fa4c3b
+size 16778117

checkpoint/pytorch_model.bin/p291.model.layers.32.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4be18e17c6f3598488c6175026c53a2bfdb0216a594bc06508cc42f2bc9f9c2d
+size 16778117

checkpoint/pytorch_model.bin/p292.model.layers.32.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:33b8364bd2602419272a37eaec54bc7cb08e883554fd5de84dfb58eec9c13f62
+size 67109765

checkpoint/pytorch_model.bin/p293.model.layers.32.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4a15e1f0b0be92b02c68e231d066d9ff6555c532093741e3b2072d7addfb2870
+size 234881916

checkpoint/pytorch_model.bin/p294.model.layers.32.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4c66ecaf5f5ff6446bf237ca630974dbcc779dc4f6ee5dda1495d52ff006afa3
+size 234881910

checkpoint/pytorch_model.bin/p295.model.layers.32.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:97dc5fca1fb458597edaedacf22aa9827d839c84727cefd15c053128857e3746
+size 234881916

checkpoint/pytorch_model.bin/p296.model.layers.32.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:beb4afab90823a3394c2a550a461ccc42ab11e3e983c4d4f0a52fb2b9e6668b0
+size 17282

checkpoint/pytorch_model.bin/p297.model.layers.32.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:89a66b11610d33d5c8bb17578a813b882c5a2e204345015f62f46a90114b2a11
+size 17309

checkpoint/pytorch_model.bin/p298.model.layers.33.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0b0b68f667813c0dd0e1717ffd0b7c5563170812dde115b1a7ced80d4297ed6d
+size 67109765

checkpoint/pytorch_model.bin/p299.model.layers.33.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e54c8c0e203a45de9f763c8663c38751c963a9b1a27fe735a3a25704b716b195
+size 16778117

checkpoint/pytorch_model.bin/p3.model.layers.0.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:703ea404a6b3b6009a8600f6c7e9226e402fb8c4e3cfaba05c9efaed659f6762
+size 16778108

checkpoint/pytorch_model.bin/p30.model.layers.3.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:293d65ffb50e0e5395f9a6f2a1a2da1b913bab105fc3de01ced6b70a89bed1d2
+size 16778111

checkpoint/pytorch_model.bin/p300.model.layers.33.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ff7a93156dc27953c2de1c66d10d345ebcf44441045884f2e236d67c9f9234f7
+size 16778117

checkpoint/pytorch_model.bin/p301.model.layers.33.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9611dd102c3c0e3a3021e98e4d03d5af57f4fbb1cff1beb461c9640e75637b96
+size 67109765

checkpoint/pytorch_model.bin/p302.model.layers.33.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:be6501f6c715c7bb9f85d15bb70e54a2ab809e1a1110484dca4fa5262cfbf1f7
+size 234881916

checkpoint/pytorch_model.bin/p303.model.layers.33.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:88a4a438b0d2d1cafe785ef3daba2b0878d4a845ffd024013fabc5f3ff961ac6
+size 234881910