Image-Text-to-Text
Safetensors
English
qwen3_5
qwen3.5
gptq
int4
quantized
coding
agentic
vlm
vision
conversational
4-bit precision
Instructions to use raydelossantos/OmniCoder-9B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Local Apps
- vLLM
How to use raydelossantos/OmniCoder-9B-GPTQ-Int4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "raydelossantos/OmniCoder-9B-GPTQ-Int4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/raydelossantos/OmniCoder-9B-GPTQ-Int4
- SGLang
How to use raydelossantos/OmniCoder-9B-GPTQ-Int4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "raydelossantos/OmniCoder-9B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "raydelossantos/OmniCoder-9B-GPTQ-Int4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "raydelossantos/OmniCoder-9B-GPTQ-Int4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use raydelossantos/OmniCoder-9B-GPTQ-Int4 with Docker Model Runner:
docker model run hf.co/raydelossantos/OmniCoder-9B-GPTQ-Int4
| layer,module,loss,samples,damp,time | |
| 0,mlp.up_proj,0.0000052520,0.01000,2.565 | |
| 0,mlp.gate_proj,0.0000060901,0.01000,2.583 | |
| 0,mlp.down_proj,0.0000000488,0.01000,3.894 | |
| 1,mlp.gate_proj,0.0000154864,0.01000,2.392 | |
| 1,mlp.up_proj,0.0000137887,0.01000,2.402 | |
| 1,mlp.down_proj,0.0000001388,0.01000,3.849 | |
| 2,mlp.gate_proj,0.0000264523,0.01000,2.366 | |
| 2,mlp.up_proj,0.0000222212,0.01000,2.384 | |
| 2,mlp.down_proj,0.0000003503,0.01000,3.846 | |
| 3,mlp.gate_proj,0.0000353122,0.01000,2.457 | |
| 3,mlp.up_proj,0.0000319611,0.01000,2.463 | |
| 3,mlp.down_proj,0.0000004708,0.01000,3.825 | |
| 4,mlp.gate_proj,0.0000511893,0.01000,2.379 | |
| 4,mlp.up_proj,0.0000462275,0.01000,2.392 | |
| 4,mlp.down_proj,0.0000008137,0.01000,3.839 | |
| 5,mlp.gate_proj,0.0000710654,0.01000,2.431 | |
| 5,mlp.up_proj,0.0000597791,0.01000,2.440 | |
| 5,mlp.down_proj,0.0000014788,0.01000,3.849 | |
| 6,mlp.up_proj,0.0000762642,0.01000,2.388 | |
| 6,mlp.gate_proj,0.0000941938,0.01000,2.397 | |
| 6,mlp.down_proj,0.0000025239,0.01000,3.862 | |
| 7,mlp.gate_proj,0.0001062528,0.01000,2.470 | |
| 7,mlp.up_proj,0.0000846265,0.01000,2.480 | |
| 7,mlp.down_proj,0.0000023836,0.01000,3.842 | |
| 8,mlp.gate_proj,0.0001096991,0.01000,2.383 | |
| 8,mlp.up_proj,0.0000904530,0.01000,2.394 | |
| 8,mlp.down_proj,0.0000025044,0.01000,3.867 | |
| 9,mlp.up_proj,0.0000976257,0.01000,2.288 | |
| 9,mlp.gate_proj,0.0001108505,0.01000,2.298 | |
| 9,mlp.down_proj,0.0000027384,0.01000,3.837 | |
| 10,mlp.gate_proj,0.0001038610,0.01000,2.406 | |
| 10,mlp.up_proj,0.0001001346,0.01000,2.415 | |
| 10,mlp.down_proj,0.0000029445,0.01000,3.839 | |
| 11,mlp.gate_proj,0.0001014121,0.01000,2.417 | |
| 11,mlp.up_proj,0.0001036301,0.01000,2.426 | |
| 11,mlp.down_proj,0.0000029805,0.01000,3.832 | |
| 12,mlp.gate_proj,0.0000995174,0.01000,2.331 | |
| 12,mlp.up_proj,0.0001058227,0.01000,2.343 | |
| 12,mlp.down_proj,0.0000031435,0.01000,3.826 | |
| 13,mlp.gate_proj,0.0001018733,0.01000,2.416 | |
| 13,mlp.up_proj,0.0001103018,0.01000,2.423 | |
| 13,mlp.down_proj,0.0000033060,0.01000,3.839 | |
| 14,mlp.gate_proj,0.0001002481,0.01000,2.423 | |
| 14,mlp.up_proj,0.0001143888,0.01000,2.431 | |
| 14,mlp.down_proj,0.0000036796,0.01000,3.846 | |
| 15,mlp.gate_proj,0.0001101084,0.01000,2.481 | |
| 15,mlp.up_proj,0.0001274935,0.01000,2.483 | |
| 15,mlp.down_proj,0.0000044571,0.01000,3.827 | |
| 16,mlp.gate_proj,0.0001224214,0.01000,2.282 | |
| 16,mlp.up_proj,0.0001404774,0.01000,2.293 | |
| 16,mlp.down_proj,0.0000057479,0.01000,3.839 | |
| 17,mlp.gate_proj,0.0001344692,0.01000,2.404 | |
| 17,mlp.up_proj,0.0001562540,0.01000,2.413 | |
| 17,mlp.down_proj,0.0000078513,0.01000,3.833 | |
| 18,mlp.gate_proj,0.0001728188,0.01000,2.305 | |
| 18,mlp.up_proj,0.0001883828,0.01000,2.317 | |
| 18,mlp.down_proj,0.0000162554,0.01000,3.836 | |
| 19,mlp.gate_proj,0.0002017946,0.01000,2.449 | |
| 19,mlp.up_proj,0.0002292997,0.01000,2.459 | |
| 19,mlp.down_proj,0.0000223045,0.01000,3.839 | |
| 20,mlp.gate_proj,0.0002697421,0.01000,2.286 | |
| 20,mlp.up_proj,0.0002577816,0.01000,2.294 | |
| 20,mlp.down_proj,0.0000260796,0.01000,3.842 | |
| 21,mlp.up_proj,0.0002807504,0.01000,2.412 | |
| 21,mlp.gate_proj,0.0003130450,0.01000,2.419 | |
| 21,mlp.down_proj,0.0000336712,0.01000,3.839 | |
| 22,mlp.gate_proj,0.0004448497,0.01000,2.305 | |
| 22,mlp.up_proj,0.0003608636,0.01000,2.313 | |
| 22,mlp.down_proj,0.0000570311,0.01000,3.837 | |
| 23,mlp.up_proj,0.0003658677,0.01000,2.345 | |
| 23,mlp.gate_proj,0.0004118672,0.01000,2.351 | |
| 23,mlp.down_proj,0.0000517022,0.01000,3.847 | |
| 24,mlp.gate_proj,0.0004587031,0.01000,2.284 | |
| 24,mlp.up_proj,0.0003868172,0.01000,2.285 | |
| 24,mlp.down_proj,0.0000539071,0.01000,3.853 | |
| 25,mlp.up_proj,0.0004362826,0.01000,2.287 | |
| 25,mlp.gate_proj,0.0005384646,0.01000,2.292 | |
| 25,mlp.down_proj,0.0000567441,0.01000,3.861 | |
| 26,mlp.up_proj,0.0005004350,0.01000,2.273 | |
| 26,mlp.gate_proj,0.0006299323,0.01000,2.283 | |
| 26,mlp.down_proj,0.0000682908,0.01000,3.862 | |
| 27,mlp.up_proj,0.0005009991,0.01000,2.337 | |
| 27,mlp.gate_proj,0.0006160481,0.01000,2.338 | |
| 27,mlp.down_proj,0.0000803794,0.01000,3.838 | |
| 28,mlp.gate_proj,0.0006322201,0.01000,2.244 | |
| 28,mlp.up_proj,0.0005199409,0.01000,2.253 | |
| 28,mlp.down_proj,0.0000990631,0.01000,3.850 | |
| 29,mlp.gate_proj,0.0006915109,0.01000,2.336 | |
| 29,mlp.up_proj,0.0005634309,0.01000,2.345 | |
| 29,mlp.down_proj,0.0001193226,0.01000,3.844 | |
| 30,mlp.gate_proj,0.0006471335,0.01000,2.418 | |
| 30,mlp.up_proj,0.0005385604,0.01000,2.428 | |
| 30,mlp.down_proj,0.0001617193,0.01000,3.849 | |
| 31,mlp.up_proj,0.0004708984,0.01000,2.331 | |
| 31,mlp.gate_proj,0.0005815547,0.01000,2.336 | |
| 31,mlp.down_proj,0.0002899637,0.01000,3.829 | |