Instructions to use RedHatAI/Inkling-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Inkling-FP8-dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/Inkling-FP8-dynamic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/Inkling-FP8-dynamic")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/Inkling-FP8-dynamic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Inkling-FP8-dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Inkling-FP8-dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Inkling-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/Inkling-FP8-dynamic

SGLang

How to use RedHatAI/Inkling-FP8-dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Inkling-FP8-dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Inkling-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Inkling-FP8-dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Inkling-FP8-dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/Inkling-FP8-dynamic with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Inkling-FP8-dynamic
```

krishnateja95 commited on 15 days ago

Commit

c18b7ca

verified ·

1 Parent(s): f671737

Add files using upload-large-folder tool

Browse files

Files changed (50) hide show

model-00036-of-00108.safetensors +3 -0
model-00037-of-00108.safetensors +3 -0
model-00038-of-00108.safetensors +3 -0
model-00039-of-00108.safetensors +3 -0
model-00040-of-00108.safetensors +3 -0
model-00041-of-00108.safetensors +3 -0
model-00042-of-00108.safetensors +3 -0
model-00043-of-00108.safetensors +3 -0
model-00044-of-00108.safetensors +3 -0
model-00046-of-00108.safetensors +3 -0
model-00047-of-00108.safetensors +3 -0
model-00048-of-00108.safetensors +3 -0
model-00049-of-00108.safetensors +3 -0
model-00050-of-00108.safetensors +3 -0
model-00051-of-00108.safetensors +3 -0
model-00052-of-00108.safetensors +3 -0
model-00055-of-00108.safetensors +3 -0
model-00057-of-00108.safetensors +3 -0
model-00058-of-00108.safetensors +3 -0
model-00059-of-00108.safetensors +3 -0
model-00060-of-00108.safetensors +3 -0
model-00061-of-00108.safetensors +3 -0
model-00062-of-00108.safetensors +3 -0
model-00063-of-00108.safetensors +3 -0
model-00064-of-00108.safetensors +3 -0
model-00065-of-00108.safetensors +3 -0
model-00066-of-00108.safetensors +3 -0
model-00067-of-00108.safetensors +3 -0
model-00068-of-00108.safetensors +3 -0
model-00069-of-00108.safetensors +3 -0
model-00070-of-00108.safetensors +3 -0
model-00071-of-00108.safetensors +3 -0
model-00072-of-00108.safetensors +3 -0
model-00073-of-00108.safetensors +3 -0
model-00074-of-00108.safetensors +3 -0
model-00075-of-00108.safetensors +3 -0
model-00076-of-00108.safetensors +3 -0
model-00077-of-00108.safetensors +3 -0
model-00078-of-00108.safetensors +3 -0
model-00079-of-00108.safetensors +3 -0
model-00080-of-00108.safetensors +3 -0
model-00081-of-00108.safetensors +3 -0
model-00082-of-00108.safetensors +3 -0
model-00083-of-00108.safetensors +3 -0
model-00084-of-00108.safetensors +3 -0
model-00085-of-00108.safetensors +3 -0
model-00086-of-00108.safetensors +3 -0
model-00087-of-00108.safetensors +3 -0
model-00089-of-00108.safetensors +3 -0
model-00090-of-00108.safetensors +3 -0

model-00036-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c3157b945aadb6199a7ca1138a20917ca46b1189de324b97d2e72473d63a1d9d
+size 10283633728

model-00037-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20c849c2d860b3246ad7d3631abf168b3a8d304ab07ce6c88ffe6f2218674990
+size 10269736984

model-00038-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d95d4d0a1e6c0e5bde5ba04bef735033ed4612a6012c9f6035b1c5b0a130fe4c
+size 9742321220

model-00039-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7b13531a36685aa7bf3ed48c58e1baf4de9f4e6a4d7f39c6c8e7c0822511f842
+size 9742543592

model-00040-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1723d9db68ef7ff209ebcc0130e17cc09751a4d6a8f217b2a93016ff35999108
+size 10069530096

model-00041-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b166b4fcc445859f2cef952fcd3ce700d855ffd946f4c3c72ae403736505490
+size 9884072816

model-00042-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2033f50dff6ead637077222d5c9dcaf691972cbd762f51987f28b26f39dd99fe
+size 10258493760

model-00043-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a952dd4b8a5f88b07d28db83fdb46851951bf1856146f16ec55f1f9a263db42f
+size 9909183912

model-00044-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cf2d0c73b6fd05da398bfd696d3b2464cb939154f2e2e742dc31abd260a601df
+size 8167367388

model-00046-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2f582fe9b783f463fae507302924dcf72dc07e4fd0a74609651e8975336d951e
+size 6379838304

model-00047-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b8a190cdbe7b96407a5b49086432e10d8b8e01608d68e1ead099020747d91bbb
+size 9669984864

model-00048-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ce258998148f77baa2aa7bac16cce45ffa99721c1d1b0e88d698b0c57a579c36
+size 9950182608

model-00049-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d4641b35b3839063c24d2ee50da0fba3cdf2da806fe00df2da9a007df770a81e
+size 10318289488

model-00050-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a25eaa38d80596b224765df50db16b67180eae2d93e13f5b40b94d7c009dff8f
+size 5263075896

model-00051-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f047579542e81411e6cb4a9352a33cfe29d97bb59ecbff75b0f556d7808bf5d1
+size 9767552680

model-00052-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ddf8e18707aea8068bc522e7977fc8a096c40156085c63087069dedf4f98c130
+size 5904920824

model-00055-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e8246e39b64b62b0c5d42441d06071b2c8f896013259f3f137668cb6aa56d160
+size 717243192

model-00057-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cfd91f1a80d57d079ddb501503bbf4e87092a5de3e5f7eb123080709b46f18ad
+size 9940769238

model-00058-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:76600b7604a4bd0b5f0cd971334df28879220d1de7a4816956dadcbfe619bd4a
+size 9981438432

model-00059-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:45ea71d24bd8006e0688fd369109aeb22df9a1763ea12c07c2740862cd04fed9
+size 10299132136

model-00060-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ee52d3b3c10587e772c3c692ecb9bb7d6df041513bcf60b1a71dc4408b97e6cb
+size 9795871424

model-00061-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:867ce327379f01e78644ed2e1f74fc101a94781b12ed5724f36422b743150aa0
+size 9912278528

model-00062-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:00a80f03844354edea567e7bdbe479948893082daab105d682b429bd01dda4c5
+size 9784820296

model-00063-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4048628239981cc3dcd8c634d007617cc81045b8fbe20c2805b8a135ab44cadb
+size 9666822400

model-00064-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:249679eec99ac7c12810240d55a0ead16cc82b7231ec0be6fff1654ba5dc456f
+size 9786401828

model-00065-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ce31552c8592de0aa366e3cca3ea21adbe05ec55529d906cb26e95fd2a195c59
+size 9767485808

model-00066-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:baf2e400bdb0d432620eb67c29dd8cba554bd0d78b3b7d57f036862d8d13790e
+size 9670006312

model-00067-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9d32303c3e14edf637a7cebf06c2b2e96c28965ee494495939a01eb9e1d4259e
+size 9994007384

model-00068-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5b6d99316346ca81909b9ed8c82ea2e6c56ee307d510f44e1f69f93f58ed28d2
+size 4834985304

model-00069-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4b38e86b9466172156f6957f8c402c4e5eb62944cf187356bf3f019bd0a8f8b1
+size 10245848940

model-00070-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b55ecf22139fe42087d3add604fbd9f2f59333425c0d77e3349ead33d8a6749b
+size 9994258668

model-00071-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c4dda308bb424ae9cfa5ccf2cf4e4d3bb29f309e5b5437b10549f6fd3c9ecc4f
+size 5174873048

model-00072-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:493fbdad7f4c3a5c88e5a15a9fbe10cfa5abd9ad0243e0b795f0dcbde229ab2d
+size 10315004420

model-00073-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:646a45206325a1db34bb6b537f653d9f5bab8c39eeb22d44def3847c4b5af13f
+size 9956248584

model-00074-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:624b2996d5fb8245207f8f1ed81f6224ac0550088597ee14005876a529516539
+size 9666822400

model-00075-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:13d46e5b2dbbeafa28e99b35673b53f8c8a3eb73f865e208c72586472af02d80
+size 4847593128

model-00076-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ac310c19e1cb6672187e8977121eca4b58f112b672d123a9fa65a99013f5bba6
+size 10261726532

model-00077-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f851e469949294ae201dafe2f467107ad47f1192da3e0286b3d16401960fde32
+size 4834996576

model-00078-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2d2132b8bf4dec9d50c884ee9db1dfc9887d3c77f2e24520a3565e16266d2ad
+size 9956354216

model-00079-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:df8d831664041cd8362fe17e15a2e3a9bb83a38cf9a84dad9c478a0867b0a9ec
+size 10009903808

model-00080-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:550df8cc1e8a174d4fa17582f5b5ce910d94dd842b3fbefdc04d6be08ff46e7a
+size 10173413556

model-00081-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4e576f6f6b22e5ea5f9502f5f35b47bdea2690b39f335719cc52a92ae59526fd
+size 4960846896

model-00082-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f55dfb51ccca64c1fab6ca7a4bd46bd34d2a45eb74ff07c852efa5e6348d5cd0
+size 10094778412

model-00083-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cc7574f7f2b0a0e4e435414902ef2574f7c169f8f3dae90df9cd5530e996118b
+size 9717236528

model-00084-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:08a1be8c558f0932c1d1a86edb73a462ec90e2c254e19416c7d0484e020c8684
+size 5014359756

model-00085-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:78690e1044bb62e6cc36f58026de753e67fc6d8062b37df71092b9a82646c4c7
+size 9770784980

model-00086-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7b950cb9eb7b61e9e685c9d940a7369a34f49392a0aaed2121ba035244c85e0e
+size 9670026160

model-00087-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e562b66fc9b6c8d460c053da190ea74b9561f3f28fdfbd0d0b90b9b0c0999884
+size 10123032444

model-00089-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a478e2ff08433c1f9a8d2d93fe73673bec96d1b864f48cb988ea3d05b8502b64
+size 7443658096

model-00090-of-00108.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:24038db4c8cfb18cbcd2d2497c1a781eff49df00b5565675f4fa5ba61ab1af64
+size 5980215736