Instructions to use nvidia/Nemotron-H-47B-Base-8K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nvidia/Nemotron-H-47B-Base-8K with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="nvidia/Nemotron-H-47B-Base-8K")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("nvidia/Nemotron-H-47B-Base-8K", dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use nvidia/Nemotron-H-47B-Base-8K with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nvidia/Nemotron-H-47B-Base-8K" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Nemotron-H-47B-Base-8K", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/nvidia/Nemotron-H-47B-Base-8K
- SGLang
How to use nvidia/Nemotron-H-47B-Base-8K with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "nvidia/Nemotron-H-47B-Base-8K" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Nemotron-H-47B-Base-8K", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "nvidia/Nemotron-H-47B-Base-8K" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nvidia/Nemotron-H-47B-Base-8K", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use nvidia/Nemotron-H-47B-Base-8K with Docker Model Runner:
docker model run hf.co/nvidia/Nemotron-H-47B-Base-8K
Instruct Models Release Date?
The blogpost published on March 21, 2025 mentions that Instruct models are coming soon. When will they be released? I have used the Base models, and they do not perform very well on certain tasks. I would love to use Instruct models in my research since I expect them to perform better. Thanks.
I wish Nvidia would respond to these directly. But anyway to my knowledge their paper does not mention a 47B instruct model, but they do have one (a reasoning model) here: https://huggingface.co/nvidia/Nemotron-H-47B-Reasoning-128K
I wish Nvidia would respond to these directly. But anyway to my knowledge their paper does not mention a 47B instruct model, but they do have one (a reasoning model) here: https://huggingface.co/nvidia/Nemotron-H-47B-Reasoning-128K
Yep, thanks! Page 18 of their paper mentions "In this work, we chose to build Nemotron-H-8B-VLM and Nemotron-H-56B-VLM on Nemotron-H-8B-Instruct and Nemotron-H-56B-Base (since Nemotron-H-56B-Instruct was unavailable)." Based on this, I am assuming there does not exist a 56B or 47B Instruct model at all, but hopefully they will release the 8B-Instruct model soon. Also, not sure if you are familiar with their Hymba models -- these are awesome, but due to the low interest from the public they did not release more of these models. Hopefully showing interest here will encourage them to create and release a 56B or 47B Instruct model as well!
Thanks for pointing that out. I am most interested in new high-quality base models I can fine tune and open instruct datasets like Nvidia's HelpSteer family which are awesome. I am still using their Llama 3.1 Nemotron 70B model because I keep finding issues with newer models around that size and cannot find smaller models that can compete with it yet.