Instructions to use nitrai-research/Moderato-V1-Pro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nitrai-research/Moderato-V1-Pro with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nitrai-research/Moderato-V1-Pro")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("nitrai-research/Moderato-V1-Pro", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nitrai-research/Moderato-V1-Pro with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nitrai-research/Moderato-V1-Pro"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nitrai-research/Moderato-V1-Pro",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nitrai-research/Moderato-V1-Pro

SGLang

How to use nitrai-research/Moderato-V1-Pro with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nitrai-research/Moderato-V1-Pro" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nitrai-research/Moderato-V1-Pro",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nitrai-research/Moderato-V1-Pro" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nitrai-research/Moderato-V1-Pro",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nitrai-research/Moderato-V1-Pro with Docker Model Runner:
```
docker model run hf.co/nitrai-research/Moderato-V1-Pro
```

NitrAI commited on 6 days ago

Commit

4c1ac63

verified ·

1 Parent(s): c5ce3a6

Add files using upload-large-folder tool

Browse files

Files changed (20) hide show

model-00003-of-00047.safetensors +3 -0
model-00006-of-00047.safetensors +3 -0
model-00009-of-00047.safetensors +3 -0
model-00010-of-00047.safetensors +3 -0
model-00011-of-00047.safetensors +3 -0
model-00012-of-00047.safetensors +3 -0
model-00013-of-00047.safetensors +3 -0
model-00014-of-00047.safetensors +3 -0
model-00015-of-00047.safetensors +3 -0
model-00016-of-00047.safetensors +3 -0
model-00017-of-00047.safetensors +3 -0
model-00018-of-00047.safetensors +3 -0
model-00019-of-00047.safetensors +3 -0
model-00020-of-00047.safetensors +3 -0
model-00021-of-00047.safetensors +3 -0
model-00037-of-00047.safetensors +3 -0
model-00038-of-00047.safetensors +3 -0
model-00039-of-00047.safetensors +3 -0
model-00040-of-00047.safetensors +3 -0
model-00041-of-00047.safetensors +3 -0

model-00003-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b18977cb9aa2cbd940e731c1558e41f05037fd4c9dae70a3a7e7e5460650ee44
+size 4980661048

model-00006-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:75066f49f16849bc09e442d6f7d9069ab9a863f1172fada661357e9a3fb13d53
+size 3774971128

model-00009-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:144bc427fdfa655eb070603d2a2c07911aa0e4b2f046976a0ea5b27dabc6e373
+size 4866607120

model-00010-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5be062cdf73d64c0cbd39a924304b302be861c293611988cc505d16f33b956cb
+size 4866545536

model-00011-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a9984d070ae12f80898a6632498681c3a9a1f7245a7e54a3756a923cba75f47a
+size 4898086776

model-00012-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:131384d761b99f0e60c8fe3e441fa007d2b68beec113c83b69b3667f726bf22d
+size 4866545504

model-00013-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e5d977e52c94452781481e2fead32d5b14eab4bffcfca49959293cb091b9e4f0
+size 4920061520

model-00014-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ed8f899f51b8410c622f3cc353a0220567e9d81eeaf69c4d0ffbee73c3a92ea4
+size 4866607120

model-00015-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:596d35edf2411193d12ba99d5ddf8cbc4cd9248aae2c49187de0d0f39f70a526
+size 4898025208

model-00016-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a9702dfb411f9e6037f62ff9eae768ee883314bb4c5f32c4063e4473a0b6a01d
+size 4866607120

model-00017-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:808f124d6cc173dffe18ea8e3b2dcac8cdd253995bfffe1c29033e29cd112e58
+size 4866545536

model-00018-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b5879058834bdb061432ecc433596b2b971f75c4caeef8779a6b95a770b84a48
+size 4898025216

model-00019-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:63e24247a0542630ff075cf71fbd4f50cc92bf5692fb6dcd73346230b7ef9107
+size 4866607120

model-00020-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a04afb394365a267d2e359cd984d49412da3d077d322517d90dd2f3a7a38ea17
+size 4866566272

model-00021-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1fc912f9b43e162bdecdbb4edb84a04ab9b04cadba77e64392f5ca5a6ec9b147
+size 4898066056

model-00037-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dde6b09a9ffdd3c22b860622f956d40552164de7f8283204c560703803c081e7
+size 4938941648

model-00038-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5d3775f1c295ed7548cd89d4a99d38fff853e72aec8f8386703db0431db284db
+size 4898004472

model-00039-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1e87775645a2d095eb294b8ce2cbd98f30e0a1bb25bd9e1a216825d5e655cbc3
+size 4920061528

model-00040-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4fe4ddcb4fe0bcf7ebb392b4ab00cdb6273e5380f96048984d615097300060e0
+size 4991369944

model-00041-of-00047.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6ca39fd2961f6b4e79c6bb76911e6efa93e4ed92eb56945b5746bdef24df7f2b
+size 4951520448