Instructions to use whpthomas/Ornith-1.0-35B-int4-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="whpthomas/Ornith-1.0-35B-int4-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("whpthomas/Ornith-1.0-35B-int4-AutoRound")
model = AutoModelForMultimodalLM.from_pretrained("whpthomas/Ornith-1.0-35B-int4-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "whpthomas/Ornith-1.0-35B-int4-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/whpthomas/Ornith-1.0-35B-int4-AutoRound

SGLang

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "whpthomas/Ornith-1.0-35B-int4-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "whpthomas/Ornith-1.0-35B-int4-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with Docker Model Runner:
```
docker model run hf.co/whpthomas/Ornith-1.0-35B-int4-AutoRound
```

Ornith-1.0-35B-int4-AutoRound

File size: 5,142 Bytes

4ec72e5

block_name,cosine_sim,psnr_db,init_loss,best_loss,best_iter,total_iters,passed
model.language_model.layers.0,0.9994282214902341,62.61240528351975,1.086852171283681e-05,4.780737867804419e-07,690,691,True
model.language_model.layers.1,0.9993868209421635,63.23861862698409,3.2157749956240878e-06,7.419856160595373e-07,946,947,True
model.language_model.layers.2,0.9994161232607439,66.8303698468095,6.380719241860788e-06,1.2434938980732113e-06,992,993,True
model.language_model.layers.3,0.9989858630578965,61.96891905147396,8.361008440260775e-06,2.3179466097644763e-06,812,813,True
model.language_model.layers.4,0.9988396158441901,61.50016394318749,6.107314220571425e-06,2.8818117243645247e-06,833,834,True
model.language_model.layers.5,0.9986090282909572,60.76878510543246,8.179788892448414e-06,3.7402801353891846e-06,861,862,True
model.language_model.layers.6,0.9985814117826521,71.3428088709036,9.07492903934326e-06,4.349758455646224e-06,873,874,True
model.language_model.layers.7,0.9977093262132257,67.25255514594416,1.522897946415469e-05,6.76060972182313e-06,943,944,True
model.language_model.layers.8,0.997463951818645,65.78136686599743,1.3584285625256598e-05,7.830098184058443e-06,549,550,True
model.language_model.layers.9,0.997400562861003,64.942588097757,1.5543886547675356e-05,9.39762503548991e-06,813,814,True
model.language_model.layers.10,0.997822972945869,79.22538349460294,2.351433431613259e-05,1.1548830116225872e-05,822,823,True
model.language_model.layers.11,0.9970905280206352,78.09018765921036,3.045690209546592e-05,1.5231271390803158e-05,910,911,True
model.language_model.layers.12,0.9967787717469037,77.96767858259926,2.5730510969879106e-05,1.556791175971739e-05,945,946,True
model.language_model.layers.13,0.9963473707903177,76.97447921758224,2.816005508066155e-05,1.9503560906741768e-05,977,978,True
model.language_model.layers.14,0.9967280139680952,76.44420215441924,3.52841270796489e-05,2.228792800451629e-05,975,976,True
model.language_model.layers.15,0.994690747698769,74.84375816965878,5.106958997203037e-05,3.070014645345509e-05,673,674,True
model.language_model.layers.16,0.9944504785817116,74.08631232116076,5.747607065131888e-05,3.6027940950589254e-05,966,967,True
model.language_model.layers.17,0.993601277587004,73.56260022645554,6.0964084696024656e-05,3.887760976795107e-05,466,467,True
model.language_model.layers.18,0.9933218620717525,72.18386972331749,8.36757681099698e-05,5.309697735356167e-05,594,595,True
model.language_model.layers.19,0.9912880986230448,70.77998330576067,0.00011228519724681973,7.46259611332789e-05,840,841,True
model.language_model.layers.20,0.9913514874642715,70.78659454864224,0.00010154077608603984,7.487440598197281e-05,648,649,True
model.language_model.layers.21,0.9910565897589549,70.44826708704605,0.00011644807818811387,7.618666859343648e-05,823,824,True
model.language_model.layers.22,0.9899991841521114,68.82308325212145,0.00020121876150369644,0.00011534220539033413,513,514,False
model.language_model.layers.23,0.9875282642897218,68.1135760734223,0.00019173507462255657,0.00013455559383146465,387,388,False
model.language_model.layers.24,0.9870607437333092,68.09814939657885,0.00017393328016623855,0.000138601113576442,510,511,False
model.language_model.layers.25,0.9863792106043547,67.55776906098212,0.00021217566973064095,0.00015702028758823872,414,415,False
model.language_model.layers.26,0.988103867159225,67.31797240571517,0.00022298246040008962,0.00016582501120865345,815,816,False
model.language_model.layers.27,0.9862225498072803,66.59014750574974,0.0002658969606272876,0.00019332914962433279,911,912,False
model.language_model.layers.28,0.9876697306754068,66.35733801900192,0.0002887144801206887,0.00021515984553843737,869,870,False
model.language_model.layers.29,0.9878157926723361,66.4322792490862,0.0002884732966776937,0.00020298466552048922,827,828,False
model.language_model.layers.30,0.9886730170110241,65.69833769126265,0.00035062278038822114,0.0002443731646053493,925,926,False
model.language_model.layers.31,0.9869619427481666,48.684063575921186,0.00041166465962305665,0.0002981695579364896,949,950,False
model.language_model.layers.32,0.9870096608065069,47.55705495393835,0.0004311591910663992,0.00030763569520786405,782,783,False
model.language_model.layers.33,0.9884082111530006,48.34967712413036,0.0006033648969605565,0.0003948768717236817,230,231,False
model.language_model.layers.34,0.9883257938781753,58.17115699223362,0.0008100846316665411,0.0005357383051887155,919,920,False
model.language_model.layers.35,0.9882124409778044,48.344503007052744,0.0010183278936892748,0.0006849644705653191,915,916,False
model.language_model.layers.36,0.9877944573527202,47.42658036172788,0.0011374535970389843,0.0007850615656934679,912,913,False
model.language_model.layers.37,0.9890564372763038,47.65615274062573,0.0015317033976316452,0.0009988396195694804,769,770,False
model.language_model.layers.38,0.9895406146533787,56.18524897111788,0.0022372640669345856,0.0014756792224943638,885,886,False
model.language_model.layers.39,0.9895580977899954,54.98397156794091,0.0033467705361545086,0.0019760383293032646,552,553,False