Instructions to use optimum/gpt2-neuronx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use optimum/gpt2-neuronx with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="optimum/gpt2-neuronx")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("optimum/gpt2-neuronx") model = AutoModelForCausalLM.from_pretrained("optimum/gpt2-neuronx", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use optimum/gpt2-neuronx with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "optimum/gpt2-neuronx" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/optimum/gpt2-neuronx
- SGLang
How to use optimum/gpt2-neuronx with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "optimum/gpt2-neuronx" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "optimum/gpt2-neuronx" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "optimum/gpt2-neuronx", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use optimum/gpt2-neuronx with Docker Model Runner:
docker model run hf.co/optimum/gpt2-neuronx
Commit History
-
Upload compiled/neuron-program.pkl with huggingface_hub
0c941ce -
Upload checkpoint/pytorch_model.bin/p118.transformer.h.9.mlp.c_fc.weight with huggingface_hub
6b0bc4d -
Upload checkpoint/pytorch_model.bin/p70.transformer.h.5.mlp.c_fc.weight with huggingface_hub
9267175 -
Upload checkpoint/pytorch_model.bin/p111.transformer.h.9.ln_1.bias with huggingface_hub
68157cf -
Upload checkpoint/pytorch_model.bin/p27.transformer.h.2.ln_1.bias with huggingface_hub
adfb24e -
Upload checkpoint/pytorch_model.bin/p115.transformer.h.9.attn.c_proj.bias with huggingface_hub
f7d639e -
Upload checkpoint/pytorch_model.bin/p69.transformer.h.5.ln_2.bias with huggingface_hub
8cc733f -
Upload checkpoint/pytorch_model.bin/p137.transformer.h.11.attn.c_attn.bias with huggingface_hub
08c2173 -
Upload checkpoint/pytorch_model.bin/p21.transformer.h.1.ln_2.bias with huggingface_hub
0165a5d -
Upload checkpoint/pytorch_model.bin/p32.transformer.h.2.ln_2.weight with huggingface_hub
6d882f2 -
Upload checkpoint/pytorch_model.bin/p83.transformer.h.6.mlp.c_fc.bias with huggingface_hub
aafad8f -
Upload checkpoint/pytorch_model.bin/p18.transformer.h.1.attn.c_proj.weight with huggingface_hub
b779a72 -
Upload checkpoint/pytorch_model.bin/p132.transformer.h.10.mlp.c_proj.weight with huggingface_hub
3f2525e -
Upload checkpoint/pytorch_model.bin/p117.transformer.h.9.ln_2.bias with huggingface_hub
168d352 -
Upload checkpoint/pytorch_model.bin/p75.transformer.h.6.ln_1.bias with huggingface_hub
a89d7c1 -
Upload checkpoint/pytorch_model.bin/p0.transformer.wte.weight with huggingface_hub
9198f4e -
Upload checkpoint/pytorch_model.bin/p82.transformer.h.6.mlp.c_fc.weight with huggingface_hub
d52f77e -
Upload checkpoint/pytorch_model.bin/p148.lm_head.weight with huggingface_hub
a2f38ab -
Upload checkpoint/pytorch_model.bin/p102.transformer.h.8.attn.c_proj.weight with huggingface_hub
97d9cc4 -
Upload checkpoint/pytorch_model.bin/p112.transformer.h.9.attn.c_attn.weight with huggingface_hub
243d880 -
Upload checkpoint/pytorch_model.bin/p56.transformer.h.4.ln_2.weight with huggingface_hub
b793e0a -
Upload checkpoint/pytorch_model.bin/p135.transformer.h.11.ln_1.bias with huggingface_hub
4cdc2b9 -
Upload checkpoint/pytorch_model.bin/p37.transformer.h.2.mlp.c_proj.bias with huggingface_hub
1abf5b8 -
Upload checkpoint/pytorch_model.bin/p121.transformer.h.9.mlp.c_proj.bias with huggingface_hub
756fe80 -
Upload checkpoint/pytorch_model.bin/p127.transformer.h.10.attn.c_proj.bias with huggingface_hub
d13fe5c -
Upload checkpoint/pytorch_model.bin/p138.transformer.h.11.attn.c_proj.weight with huggingface_hub
4cdaba6 -
Upload checkpoint/pytorch_model.bin/p129.transformer.h.10.ln_2.bias with huggingface_hub
528e215 -
Upload checkpoint/pytorch_model.bin/p53.transformer.h.4.attn.c_attn.bias with huggingface_hub
360e3e0 -
Upload checkpoint/pytorch_model.bin/p33.transformer.h.2.ln_2.bias with huggingface_hub
78144c5 -
Upload checkpoint/pytorch_model.bin/p63.transformer.h.5.ln_1.bias with huggingface_hub
18d3d4f -
Upload checkpoint/pytorch_model.bin/p38.transformer.h.3.ln_1.weight with huggingface_hub
e8d28a9 -
Upload checkpoint/pytorch_model.bin/p44.transformer.h.3.ln_2.weight with huggingface_hub
172f4a5 -
Upload checkpoint/pytorch_model.bin/p30.transformer.h.2.attn.c_proj.weight with huggingface_hub
980143a -
Upload checkpoint/pytorch_model.bin/p91.transformer.h.7.attn.c_proj.bias with huggingface_hub
7d788e6 -
Upload checkpoint/pytorch_model.bin/p13.transformer.h.0.mlp.c_proj.bias with huggingface_hub
c9608ba -
Upload checkpoint/pytorch_model.bin/p110.transformer.h.9.ln_1.weight with huggingface_hub
bf7e564 -
Upload checkpoint/pytorch_model.bin/p36.transformer.h.2.mlp.c_proj.weight with huggingface_hub
eae1759 -
Upload checkpoint/pytorch_model.bin/p88.transformer.h.7.attn.c_attn.weight with huggingface_hub
bc8e8b3 -
Upload checkpoint/pytorch_model.bin/p35.transformer.h.2.mlp.c_fc.bias with huggingface_hub
8c1f708 -
Upload checkpoint/pytorch_model.bin/p86.transformer.h.7.ln_1.weight with huggingface_hub
4178ce9 -
Upload checkpoint/pytorch_model.bin/p122.transformer.h.10.ln_1.weight with huggingface_hub
9185afa -
Upload checkpoint/pytorch_model.bin/p93.transformer.h.7.ln_2.bias with huggingface_hub
796a1f4 -
Upload checkpoint/pytorch_model.bin/p45.transformer.h.3.ln_2.bias with huggingface_hub
257dd3a -
Upload checkpoint/pytorch_model.bin/key_to_filename.json with huggingface_hub
be1a465 -
Upload checkpoint/pytorch_model.bin/p124.transformer.h.10.attn.c_attn.weight with huggingface_hub
dd1735a -
Upload checkpoint/pytorch_model.bin/p95.transformer.h.7.mlp.c_fc.bias with huggingface_hub
fda7edf -
Upload checkpoint/pytorch_model.bin/p26.transformer.h.2.ln_1.weight with huggingface_hub
0759f8c -
Upload checkpoint/pytorch_model.bin/p126.transformer.h.10.attn.c_proj.weight with huggingface_hub
387d454 -
Upload checkpoint/pytorch_model.bin/p76.transformer.h.6.attn.c_attn.weight with huggingface_hub
2e156aa -
Upload checkpoint/pytorch_model.bin/p113.transformer.h.9.attn.c_attn.bias with huggingface_hub
7e1da61