Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

dac7362

verified ·

1 Parent(s): 055f4bf

f88a38b11ffcb39d1f076005f722ed3bc81dd19c93a42780110e9be9446f184c

Browse files

Files changed (17) hide show

checkpoint/pytorch_model.bin/p419.model.layers.46.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p42.model.layers.4.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p420.model.layers.46.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p421.model.layers.46.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p422.model.layers.46.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p423.model.layers.46.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p424.model.layers.47.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p425.model.layers.47.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p426.model.layers.47.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p427.model.layers.47.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p428.model.layers.47.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p429.model.layers.47.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p43.model.layers.4.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p430.model.layers.47.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p431.model.layers.47.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p432.model.layers.47.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p433.model.norm.weight +3 -0

checkpoint/pytorch_model.bin/p419.model.layers.46.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a29b4ce8c013946ddc69e0b0071d8ceae7430dc1ca759d1296e71c60e3d96122
+size 234881916

checkpoint/pytorch_model.bin/p42.model.layers.4.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:04eaeb372ed089ddd1d98904f379c1d8c1204d6d5e3cdfbf15272e6a9fb7bf63
+size 234881904

checkpoint/pytorch_model.bin/p420.model.layers.46.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6144376f49da63fcf0a398e1c88ac184e249143fef36c9b1df2675cc01971054
+size 234881910

checkpoint/pytorch_model.bin/p421.model.layers.46.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6feaa6b854a4476996e351bf52dcc769b9cbe450648ef810aedd11aa63a6d510
+size 234881916

checkpoint/pytorch_model.bin/p422.model.layers.46.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5270e8c918913467738031abea340644a4276e19fae1cc32e984d4877f04c653
+size 17282

checkpoint/pytorch_model.bin/p423.model.layers.46.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b8db7dc05de1c86df1cd1483398caa722b84883d9fb58c8d4a53950f16bb3cbc
+size 17309

checkpoint/pytorch_model.bin/p424.model.layers.47.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2b3e1026693cff343a13ddbbdaaf34bc601eac4a5817578b9026e02fc23ecf60
+size 67109765

checkpoint/pytorch_model.bin/p425.model.layers.47.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b6408a794a7d681b4008064758e27ab842de74c465e1f24d443cb778f896ecb2
+size 16778117

checkpoint/pytorch_model.bin/p426.model.layers.47.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e00ca39d58a75b57306070736638c82fad24cf30abcec853912e2be4af1dc9b4
+size 16778117

checkpoint/pytorch_model.bin/p427.model.layers.47.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a167a58e17163f5e7d3128bd74e7c3e91a092aaf04dd9a940a6228cb0b7fb450
+size 67109765

checkpoint/pytorch_model.bin/p428.model.layers.47.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:164537ce4a0da5b950d3ed75a34c9710e3b37f10168a139850e5a851e3ba331a
+size 234881916

checkpoint/pytorch_model.bin/p429.model.layers.47.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4f24059e38bbe9a0ef385ee8c9e19469115b69526dcc591b9f3f339026423d12
+size 234881910

checkpoint/pytorch_model.bin/p43.model.layers.4.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c839eaf529453b2b2d0fd475a9ab8be0e6bfc313e8bfb72cb3dab400fdb2f9e8
+size 234881910

checkpoint/pytorch_model.bin/p430.model.layers.47.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a773374c1893bf3d102993af5bfecd0ff9a5731589e4a2c76dd2c5b861a69aa1
+size 234881916

checkpoint/pytorch_model.bin/p431.model.layers.47.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e369f8daeb6ae0de680a97f10c4cae6a482df5fe89e02185480a6b58c2c5103f
+size 17282

checkpoint/pytorch_model.bin/p432.model.layers.47.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9b536d9c4ab9f37570b0bc8bbba0d40925c02fcaa312c564eb97e1ae8b0152b7
+size 17309

checkpoint/pytorch_model.bin/p433.model.norm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d21513bf4cdf37c6eb77471e46783e6d537960a81e64125734b8d5dcc0cb3849
+size 17155