Instructions to use aws-neuron/Mistral-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use aws-neuron/Mistral-neuron with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="aws-neuron/Mistral-neuron")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("aws-neuron/Mistral-neuron") model = AutoModelForCausalLM.from_pretrained("aws-neuron/Mistral-neuron") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use aws-neuron/Mistral-neuron with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aws-neuron/Mistral-neuron" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/aws-neuron/Mistral-neuron
- SGLang
How to use aws-neuron/Mistral-neuron with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "aws-neuron/Mistral-neuron" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "aws-neuron/Mistral-neuron" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use aws-neuron/Mistral-neuron with Docker Model Runner:
docker model run hf.co/aws-neuron/Mistral-neuron
12921ed9c4993dff71badf93c660e917ff8aaea69efcaeb860854f5b2a0fbbb0
Browse files- pytorch_model.bin/p114.model.layers.12.mlp.up_proj.weight +3 -0
- pytorch_model.bin/p115.model.layers.12.mlp.down_proj.weight +3 -0
- pytorch_model.bin/p116.model.layers.12.input_layernorm.weight +3 -0
- pytorch_model.bin/p117.model.layers.12.post_attention_layernorm.weight +3 -0
- pytorch_model.bin/p118.model.layers.13.self_attn.q_proj.weight +3 -0
- pytorch_model.bin/p119.model.layers.13.self_attn.k_proj.weight +3 -0
- pytorch_model.bin/p12.model.layers.1.self_attn.v_proj.weight +3 -0
- pytorch_model.bin/p120.model.layers.13.self_attn.v_proj.weight +3 -0
- pytorch_model.bin/p121.model.layers.13.self_attn.o_proj.weight +3 -0
- pytorch_model.bin/p122.model.layers.13.mlp.gate_proj.weight +3 -0
- pytorch_model.bin/p123.model.layers.13.mlp.up_proj.weight +3 -0
- pytorch_model.bin/p124.model.layers.13.mlp.down_proj.weight +3 -0
- pytorch_model.bin/p125.model.layers.13.input_layernorm.weight +3 -0
- pytorch_model.bin/p126.model.layers.13.post_attention_layernorm.weight +3 -0
- pytorch_model.bin/p127.model.layers.14.self_attn.q_proj.weight +3 -0
- pytorch_model.bin/p128.model.layers.14.self_attn.k_proj.weight +3 -0
- pytorch_model.bin/p129.model.layers.14.self_attn.v_proj.weight +3 -0
- pytorch_model.bin/p13.model.layers.1.self_attn.o_proj.weight +3 -0
- pytorch_model.bin/p130.model.layers.14.self_attn.o_proj.weight +3 -0
- pytorch_model.bin/p131.model.layers.14.mlp.gate_proj.weight +3 -0
- pytorch_model.bin/p132.model.layers.14.mlp.up_proj.weight +3 -0
pytorch_model.bin/p114.model.layers.12.mlp.up_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:aadd92e6355c9c2f4eb5ebd633bec4c74242c370f306ae16149ea46aaac68f35
|
| 3 |
+
size 234881910
|
pytorch_model.bin/p115.model.layers.12.mlp.down_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3a1bbc55083cd7c7732c9221263b36b77aef795cb4243c2b073437ef706985ab
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p116.model.layers.12.input_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f881aeb8485027090bcaefb4c03ff5896639745a7b97e38cfaae94751bdd2139
|
| 3 |
+
size 17282
|
pytorch_model.bin/p117.model.layers.12.post_attention_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c7df644a404ab468910f051b564e0bcce1f6fb36242d92579d2a7d7e425b460d
|
| 3 |
+
size 17309
|
pytorch_model.bin/p118.model.layers.13.self_attn.q_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e53f7d641dd1751356ce3d6b41945e58fa05ea0fc8b52b914942203fc1f901ad
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p119.model.layers.13.self_attn.k_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:77de0c211f8b8ca66ed57238d283d0ff451532e8e2fe14fc48e996b8191c3426
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p12.model.layers.1.self_attn.v_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6c23f11e6df663d5cd8593557616c770c3003d883d29157664cb108c48acdb88
|
| 3 |
+
size 16778111
|
pytorch_model.bin/p120.model.layers.13.self_attn.v_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:578cf4f64a492a8cb27bf1069accddb68e7403143def15c66b4a5828d5187165
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p121.model.layers.13.self_attn.o_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:044674f18267c4243e49b849f0144f652a5f634331f5f84bea2841899a20405d
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p122.model.layers.13.mlp.gate_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:39a6410bfb512bd41a2bd77d9e97af71e1f1e2215af851d56b0dfcd0a6ac213d
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p123.model.layers.13.mlp.up_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bd9b832c494b1565b02907ef2ab646fcda1d22b890c5c17d284318eecf980a82
|
| 3 |
+
size 234881910
|
pytorch_model.bin/p124.model.layers.13.mlp.down_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2b41b4bbdfa27f1e4610fff6c7479bf34ee83c9d7cecce088721e770f59e00db
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p125.model.layers.13.input_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ca452ddba55968dc8178f804b22a66a3e0671a4522f7827cb3f780a6f0a06ef9
|
| 3 |
+
size 17282
|
pytorch_model.bin/p126.model.layers.13.post_attention_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e2c85f6dec02423e682c65eaec188843147e8b781f1aaa9ad9328d27ec38b258
|
| 3 |
+
size 17309
|
pytorch_model.bin/p127.model.layers.14.self_attn.q_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:22a217e37b4620aee88423fb13bda6b9d386f5d3975fb1b0364866a435ff54ce
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p128.model.layers.14.self_attn.k_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8f9ce0c56e6db46d98ead9e421e8446bd13dc6752c8b578076857dbf6aa09c8a
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p129.model.layers.14.self_attn.v_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2587c54ce24ce5b2b7abe43be5d69f2bc235785f35a1c9c5bf930ed13a9b7c69
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p13.model.layers.1.self_attn.o_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:605a356d1cb9a03e5d27d7c9af7d4525d4752e2b7c2156f41ea3c12a49b442c3
|
| 3 |
+
size 67109759
|
pytorch_model.bin/p130.model.layers.14.self_attn.o_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eb475fcb012508d2caa09109c5cceffc6d32a2dc10746e5042fb98a377925a8c
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p131.model.layers.14.mlp.gate_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d4a4a4ba1e207ceacb9297b751f12e67abed2791669d896dd4986983ecf5cd58
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p132.model.layers.14.mlp.up_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88f96731920a93fa27bd7a7313aadddad810307f33d455c9f2ccebb2102ab8d3
|
| 3 |
+
size 234881910
|