Instructions to use aws-neuron/Mistral-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use aws-neuron/Mistral-neuron with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="aws-neuron/Mistral-neuron")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("aws-neuron/Mistral-neuron") model = AutoModelForCausalLM.from_pretrained("aws-neuron/Mistral-neuron") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use aws-neuron/Mistral-neuron with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aws-neuron/Mistral-neuron" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/aws-neuron/Mistral-neuron
- SGLang
How to use aws-neuron/Mistral-neuron with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "aws-neuron/Mistral-neuron" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "aws-neuron/Mistral-neuron" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aws-neuron/Mistral-neuron", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use aws-neuron/Mistral-neuron with Docker Model Runner:
docker model run hf.co/aws-neuron/Mistral-neuron
71b32bdc64207dd142f307ba1d680f6f6dece41fc1e5dae40dbc1883a010a157
Browse files- pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight +3 -0
- pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight +3 -0
- pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight +3 -0
- pytorch_model.bin/p269.model.layers.29.input_layernorm.weight +3 -0
- pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight +3 -0
- pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight +3 -0
- pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight +3 -0
- pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight +3 -0
- pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight +3 -0
- pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight +3 -0
- pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight +3 -0
- pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight +3 -0
- pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight +3 -0
- pytorch_model.bin/p278.model.layers.30.input_layernorm.weight +3 -0
- pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight +3 -0
- pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight +3 -0
- pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight +3 -0
- pytorch_model.bin/p281.model.layers.31.self_attn.k_proj.weight +3 -0
- pytorch_model.bin/p282.model.layers.31.self_attn.v_proj.weight +3 -0
- pytorch_model.bin/p283.model.layers.31.self_attn.o_proj.weight +3 -0
- pytorch_model.bin/p284.model.layers.31.mlp.gate_proj.weight +3 -0
pytorch_model.bin/p266.model.layers.29.mlp.gate_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:84114cb3a2db716bf1d4a781bf5abd60259f1eedfba6dd0b4620dbe164893628
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p267.model.layers.29.mlp.up_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:89dfa180d10f7a06ed1ef1a64f03f820c2f6c564ed2aeb27e1cedbc27eac27fe
|
| 3 |
+
size 234881910
|
pytorch_model.bin/p268.model.layers.29.mlp.down_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e253af6809450e9f8537d0666e586fe7eedce466c5e8cb8a47a9648fe30c94dc
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p269.model.layers.29.input_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1b5dd8f975895e8fab7a96040a51e66fd816cb381fbca84cf927556d2c0a9c25
|
| 3 |
+
size 17282
|
pytorch_model.bin/p27.model.layers.2.post_attention_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9c9949a1572beb8a90639cad31a266183e59db76a3960d4b03be1938ed67cbd6
|
| 3 |
+
size 17303
|
pytorch_model.bin/p270.model.layers.29.post_attention_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:888051513cda0b5048eac04d784750a3d113b162f138bfbfe46e895a315d9897
|
| 3 |
+
size 17309
|
pytorch_model.bin/p271.model.layers.30.self_attn.q_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:be13a5e0e59fa2f10d25a874b36ba5d3d2b3af00237250b243ae7776387acc42
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p272.model.layers.30.self_attn.k_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:82e75bca821bc96137e4c6cb8ee89e0c44ddcf84c257af7d83b5514b831d9966
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p273.model.layers.30.self_attn.v_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cf480f8bd1daa070a39d021240d4e26a2e5495ed8e3927588689ba08ec6419a3
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p274.model.layers.30.self_attn.o_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:63b0c5c2b6941077ef7adb219fb39071a0b1c3cbda8b0bd048b20203bae16307
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p275.model.layers.30.mlp.gate_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2dfd519976e8bf7dab7a370ff6a0dc540925c288f42a0fd45501a4efe1f8a3f8
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p276.model.layers.30.mlp.up_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e13f17bf942a442be0bda22c84795a44f9c389802fb567ef91a5d887b4834561
|
| 3 |
+
size 234881910
|
pytorch_model.bin/p277.model.layers.30.mlp.down_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b4e20c8f2641ad123ae8cc3dbb825c77394d0d45c66a29158bea1f9fb498cb7b
|
| 3 |
+
size 234881916
|
pytorch_model.bin/p278.model.layers.30.input_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8155d6865465d5be229f60696f9ec08733c34b350d56a96fcd80448df6bf2eb0
|
| 3 |
+
size 17282
|
pytorch_model.bin/p279.model.layers.30.post_attention_layernorm.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:328e25796dc14d42ea5dbf2157b56cfe7c2b9d5bdda0d7dd470c19bbf5b577cd
|
| 3 |
+
size 17309
|
pytorch_model.bin/p28.model.layers.3.self_attn.q_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:600ea777bc41270d70b5f25a8a04568aa22bb09827c2421160816aaca987ead1
|
| 3 |
+
size 67109759
|
pytorch_model.bin/p280.model.layers.31.self_attn.q_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c7a4b0dd4cd4f49419d5e86d2ace601c3b7288061b09aba878e43252308ee17a
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p281.model.layers.31.self_attn.k_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f3c9949620adf162ac6896b88ae5c636c127d80b6af3d177f1b3aa3534ede6d1
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p282.model.layers.31.self_attn.v_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1aac130dd1b8627b51e6d0d00ed40889f183c2e9b0a627ccaba1185b46d47147
|
| 3 |
+
size 16778117
|
pytorch_model.bin/p283.model.layers.31.self_attn.o_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d0493002efa15644c26060b093a3eb10ba1c2b30d2a7a84746003f6fd35f9e89
|
| 3 |
+
size 67109765
|
pytorch_model.bin/p284.model.layers.31.mlp.gate_proj.weight
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7d3b12e72091170abd8bcbe2bb1a0408cbc5660d9e37238bdf6e43d9b0f1b24b
|
| 3 |
+
size 234881916
|