Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

ffdb2e8

verified ·

1 Parent(s): 050652e

b103556ce7f2bcb66146e0f9f4048f151c32e65544b0e2fd199d6fd382f079a8

Browse files

Files changed (20) hide show

checkpoint/pytorch_model.bin/p304.model.layers.33.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p305.model.layers.33.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p306.model.layers.33.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p307.model.layers.34.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p308.model.layers.34.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p309.model.layers.34.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p31.model.layers.3.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p310.model.layers.34.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p311.model.layers.34.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p312.model.layers.34.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p313.model.layers.34.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p314.model.layers.34.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p315.model.layers.34.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p316.model.layers.35.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p317.model.layers.35.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p318.model.layers.35.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p319.model.layers.35.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p32.model.layers.3.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p320.model.layers.35.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p321.model.layers.35.mlp.up_proj.weight +3 -0

checkpoint/pytorch_model.bin/p304.model.layers.33.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0e30d9d7a5454131d7db9477d73635c67e5662e067825c69ad6b47f68f347e98
+size 234881916

checkpoint/pytorch_model.bin/p305.model.layers.33.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d50e6054f345370b7f3ab26352361247ebc9944d82e9f8154b66cb4ff1f0b836
+size 17282

checkpoint/pytorch_model.bin/p306.model.layers.33.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1fc43df646e63bc578abc58b71dbf22ae7982037259b62445ac85a9b047abf08
+size 17309

checkpoint/pytorch_model.bin/p307.model.layers.34.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c7c5bd4e55b679279aaa6e9d4d6b2137f7499935efe33872d2bcef83b4807e79
+size 67109765

checkpoint/pytorch_model.bin/p308.model.layers.34.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ebb01cf98487003a441235deb6e46ebd69931b9d77f5820ac8f7695a3edfc602
+size 16778117

checkpoint/pytorch_model.bin/p309.model.layers.34.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:844e37f2a1eaac6ceae4dbcb8192a7d69863aa72fffcaa3f43d99b39f04c325e
+size 16778117

checkpoint/pytorch_model.bin/p31.model.layers.3.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0fb43cc400c821b5bab96624e1923a07c8ccd2aefba03ce55b8c1a3644768290
+size 67109759

checkpoint/pytorch_model.bin/p310.model.layers.34.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a8500431f6590baa7c60a7ce9fe58217d2296c553aef667aec9f110839a1307e
+size 67109765

checkpoint/pytorch_model.bin/p311.model.layers.34.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2b3553ff0ec2552582a634e0a09423ddf34677ebda42c756cfd33243411d13fc
+size 234881916

checkpoint/pytorch_model.bin/p312.model.layers.34.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:da26ce41a49e723695d5cb3087a72cb486b8ca44bd0e35f72538588ab5cdd341
+size 234881910

checkpoint/pytorch_model.bin/p313.model.layers.34.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5fef3cc31619967763c94d6be4037f2bcc776616ccf71c8e3834793a1a36637a
+size 234881916

checkpoint/pytorch_model.bin/p314.model.layers.34.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2485a2dadc623b2cbe16a4b3f5ccfa5624497f0f9cd7f1ea1b73d560a97c5c6
+size 17282

checkpoint/pytorch_model.bin/p315.model.layers.34.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:84f27768ffceed00fa60a95f69d0d57345b958dc1af381ca58d77ac1cf9570de
+size 17309

checkpoint/pytorch_model.bin/p316.model.layers.35.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5771d626f2faf60f8d9e765a13e8c517ba48582cd17304c816dad4d9b89edc50
+size 67109765

checkpoint/pytorch_model.bin/p317.model.layers.35.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6210ceb4cd938b99858c6dd492a7cbfa97aea3350460b5eabe02a88f3ac2485a
+size 16778117

checkpoint/pytorch_model.bin/p318.model.layers.35.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2e05d25e6348f4a6ebdecb3ee214c6ea380d96bb4de82c6f1b11584e831bac35
+size 16778117

checkpoint/pytorch_model.bin/p319.model.layers.35.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f1f5caaf84b6050b89ef56f83e2d3b7811d8b834a270aaed82ab8a0c6e9dd2b5
+size 67109765

checkpoint/pytorch_model.bin/p32.model.layers.3.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6c80c7696f0eab54a1980dfe46acd74992750c7c3b861e1b63f32b139b0b10ec
+size 234881910

checkpoint/pytorch_model.bin/p320.model.layers.35.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:15552a880b795e947783a2ae58ccc79109bc587e78800a25751e996087753b0e
+size 234881916

checkpoint/pytorch_model.bin/p321.model.layers.35.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6f48de180f5b5aa03ef43394cf542438a7a28d3b5fc66b6ae71b1dc2cdeb7ba5
+size 234881910