Image-Text-to-Text
Safetensors
English
qwen3_5
qwen3.5
gptq
int8
quantized
coding
agentic
vlm
vision
conversational
8-bit precision
Instructions to use raydelossantos/OmniCoder-9B-GPTQ-Int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps Settings
- vLLM
How to use raydelossantos/OmniCoder-9B-GPTQ-Int8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "raydelossantos/OmniCoder-9B-GPTQ-Int8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/raydelossantos/OmniCoder-9B-GPTQ-Int8
- SGLang
How to use raydelossantos/OmniCoder-9B-GPTQ-Int8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "raydelossantos/OmniCoder-9B-GPTQ-Int8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "raydelossantos/OmniCoder-9B-GPTQ-Int8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int8", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use raydelossantos/OmniCoder-9B-GPTQ-Int8 with Docker Model Runner:
docker model run hf.co/raydelossantos/OmniCoder-9B-GPTQ-Int8
| layer,module,loss,samples,damp,time | |
| 0,mlp.up_proj,0.0000000180,0.01000,14.698 | |
| 0,mlp.gate_proj,0.0000000209,0.01000,14.703 | |
| 0,mlp.down_proj,0.0000000002,0.01000,19.578 | |
| 1,mlp.up_proj,0.0000000474,0.01000,15.614 | |
| 1,mlp.gate_proj,0.0000000532,0.01000,15.719 | |
| 1,mlp.down_proj,0.0000000005,0.01000,19.688 | |
| 2,mlp.gate_proj,0.0000000909,0.01000,13.516 | |
| 2,mlp.up_proj,0.0000000763,0.01000,13.525 | |
| 2,mlp.down_proj,0.0000000012,0.01000,16.068 | |
| 3,mlp.up_proj,0.0000001099,0.01000,12.868 | |
| 3,mlp.gate_proj,0.0000001215,0.01000,12.880 | |
| 3,mlp.down_proj,0.0000000016,0.01000,18.303 | |
| 4,mlp.gate_proj,0.0000001759,0.01000,13.142 | |
| 4,mlp.up_proj,0.0000001591,0.01000,13.220 | |
| 4,mlp.down_proj,0.0000000028,0.01000,18.932 | |
| 5,mlp.up_proj,0.0000002053,0.01000,13.543 | |
| 5,mlp.gate_proj,0.0000002436,0.01000,13.568 | |
| 5,mlp.down_proj,0.0000000051,0.01000,19.012 | |
| 6,mlp.gate_proj,0.0000003231,0.01000,13.359 | |
| 6,mlp.up_proj,0.0000002633,0.01000,13.382 | |
| 6,mlp.down_proj,0.0000000087,0.01000,18.271 | |
| 7,mlp.up_proj,0.0000002905,0.01000,13.692 | |
| 7,mlp.gate_proj,0.0000003638,0.01000,13.694 | |
| 7,mlp.down_proj,0.0000000082,0.01000,19.970 | |
| 8,mlp.gate_proj,0.0000003752,0.01000,14.667 | |
| 8,mlp.up_proj,0.0000003103,0.01000,14.686 | |
| 8,mlp.down_proj,0.0000000086,0.01000,19.069 | |
| 9,mlp.gate_proj,0.0000003788,0.01000,15.200 | |
| 9,mlp.up_proj,0.0000003346,0.01000,15.199 | |
| 9,mlp.down_proj,0.0000000093,0.01000,19.630 | |
| 10,mlp.gate_proj,0.0000003548,0.01000,15.047 | |
| 10,mlp.up_proj,0.0000003430,0.01000,15.056 | |
| 10,mlp.down_proj,0.0000000100,0.01000,19.467 | |
| 11,mlp.up_proj,0.0000003548,0.01000,14.072 | |
| 11,mlp.gate_proj,0.0000003465,0.01000,14.080 | |
| 11,mlp.down_proj,0.0000000102,0.01000,20.719 | |
| 12,mlp.gate_proj,0.0000003402,0.01000,14.767 | |
| 12,mlp.up_proj,0.0000003621,0.01000,14.772 | |
| 12,mlp.down_proj,0.0000000107,0.01000,19.933 | |
| 13,mlp.up_proj,0.0000003774,0.01000,14.899 | |
| 13,mlp.gate_proj,0.0000003481,0.01000,14.922 | |
| 13,mlp.down_proj,0.0000000113,0.01000,19.574 | |
| 14,mlp.up_proj,0.0000003916,0.01000,14.631 | |
| 14,mlp.gate_proj,0.0000003426,0.01000,14.635 | |
| 14,mlp.down_proj,0.0000000125,0.01000,18.898 | |
| 15,mlp.up_proj,0.0000004363,0.01000,14.067 | |
| 15,mlp.gate_proj,0.0000003765,0.01000,14.079 | |
| 15,mlp.down_proj,0.0000000152,0.01000,20.600 | |
| 16,mlp.up_proj,0.0000004807,0.01000,15.427 | |
| 16,mlp.gate_proj,0.0000004186,0.01000,15.434 | |
| 16,mlp.down_proj,0.0000000196,0.01000,20.297 | |
| 17,mlp.up_proj,0.0000005352,0.01000,15.860 | |
| 17,mlp.gate_proj,0.0000004607,0.01000,15.861 | |
| 17,mlp.down_proj,0.0000000268,0.01000,20.976 | |
| 18,mlp.gate_proj,0.0000005921,0.01000,14.978 | |
| 18,mlp.up_proj,0.0000006467,0.01000,14.996 | |
| 18,mlp.down_proj,0.0000000558,0.01000,20.681 | |
| 19,mlp.up_proj,0.0000007851,0.01000,13.961 | |
| 19,mlp.gate_proj,0.0000006913,0.01000,14.010 | |
| 19,mlp.down_proj,0.0000000765,0.01000,21.126 | |
| 20,mlp.gate_proj,0.0000009249,0.01000,15.675 | |
| 20,mlp.up_proj,0.0000008837,0.01000,15.676 | |
| 20,mlp.down_proj,0.0000000899,0.01000,21.957 | |
| 21,mlp.up_proj,0.0000009645,0.01000,15.818 | |
| 21,mlp.gate_proj,0.0000010762,0.01000,15.819 | |
| 21,mlp.down_proj,0.0000001164,0.01000,21.863 | |
| 22,mlp.up_proj,0.0000012398,0.01000,14.501 | |
| 22,mlp.gate_proj,0.0000015297,0.01000,14.519 | |
| 22,mlp.down_proj,0.0000001975,0.01000,20.517 | |
| 23,mlp.up_proj,0.0000012593,0.01000,13.889 | |
| 23,mlp.gate_proj,0.0000014170,0.01000,13.901 | |
| 23,mlp.down_proj,0.0000001780,0.01000,21.184 | |
| 24,mlp.gate_proj,0.0000015803,0.01000,15.295 | |
| 24,mlp.up_proj,0.0000013320,0.01000,15.305 | |
| 24,mlp.down_proj,0.0000001857,0.01000,22.126 | |
| 25,mlp.gate_proj,0.0000018567,0.01000,14.848 | |
| 25,mlp.up_proj,0.0000015032,0.01000,14.854 | |
| 25,mlp.down_proj,0.0000001956,0.01000,22.090 | |
| 26,mlp.gate_proj,0.0000021709,0.01000,14.203 | |
| 26,mlp.up_proj,0.0000017253,0.01000,14.226 | |
| 26,mlp.down_proj,0.0000002361,0.01000,20.607 | |
| 27,mlp.up_proj,0.0000017307,0.01000,12.948 | |
| 27,mlp.gate_proj,0.0000021289,0.01000,12.958 | |
| 27,mlp.down_proj,0.0000002777,0.01000,20.681 | |
| 28,mlp.up_proj,0.0000017948,0.01000,14.732 | |
| 28,mlp.gate_proj,0.0000021849,0.01000,14.745 | |
| 28,mlp.down_proj,0.0000003420,0.01000,21.724 | |
| 29,mlp.gate_proj,0.0000023903,0.01000,14.719 | |
| 29,mlp.up_proj,0.0000019471,0.01000,14.733 | |
| 29,mlp.down_proj,0.0000004111,0.01000,21.808 | |
| 30,mlp.gate_proj,0.0000022353,0.01000,14.004 | |
| 30,mlp.up_proj,0.0000018579,0.01000,14.013 | |
| 30,mlp.down_proj,0.0000005554,0.01000,19.780 | |
| 31,mlp.up_proj,0.0000016236,0.01000,9.651 | |
| 31,mlp.gate_proj,0.0000020070,0.01000,9.733 | |
| 31,mlp.down_proj,0.0000009926,0.01000,8.318 | |