Instructions to use Dexy2/Piko-9b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Dexy2/Piko-9b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Dexy2/Piko-9b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Dexy2/Piko-9b")
model = AutoModelForMultimodalLM.from_pretrained("Dexy2/Piko-9b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Dexy2/Piko-9b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Dexy2/Piko-9b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Dexy2/Piko-9b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Dexy2/Piko-9b

SGLang

How to use Dexy2/Piko-9b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Dexy2/Piko-9b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Dexy2/Piko-9b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Dexy2/Piko-9b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Dexy2/Piko-9b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Dexy2/Piko-9b with Docker Model Runner:
```
docker model run hf.co/Dexy2/Piko-9b
```

Piko-9b / docs /quantization.md

Dexy2

Rewrite model card around verified evidence; correct misattributed benchmarks and config path leak

0810902 verified 2 days ago

preview code

Raw

History Blame Contribute Delete

5.19 kB

Quantization

Summary

Format	Status	Weights	Practical VRAM	Notes
bfloat16 (native)	Shipped	19.8 GB	~22 GB	The published checkpoint
4-bit NF4 (bitsandbytes)	Validated	5.6 GB	8.1 GB measured	Every measured result here used this
8-bit (bitsandbytes)	Untested here	10.1 GB	~12 GB	Should work; not validated
GGUF	Not produced	—	—	See below
AWQ	Not produced	—	—	Script provided, not run
GPTQ	Not produced	—	—	Script provided, not run

No quantized artefact has been published for Piko-9b. Nothing in this section claims a quantization works unless it was run.

4-bit NF4 — the validated path

This is what every number in this repository was measured with.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, BitsAndBytesConfig

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForMultimodalLM.from_pretrained(
    "Dexy2/Piko-9b",
    quantization_config=quantization,
    device_map={"": 0},          # never "auto"
    dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained("Dexy2/Piko-9b")

Measured on an RTX 5070 Ti (17.1 GB), weights on NVMe:


Cold load	100–102 s
Resident VRAM	8.1 GB
Text generation	0.9–2.9 s per short answer
Image + text	~4 s per answer
14,429-token prompt	3.5 s, needle retrieved correctly
Custom suite	65/70

The vision path survives 4-bit. OCR scored 10/10 and document understanding 10/10 under NF4, so the projector and vision tower are not visibly damaged by weight quantization. That is a measurement on this suite, not a general guarantee.

The rule that matters more than the format

Whatever quantization you choose, the whole model must stay on one device.

Piko-9b keeps a recurrent float32 state across its 24 linear-attention layers. device_map="auto" on an undersized GPU offloads layers to CPU, corrupts that state, and the model emits a single repeated character with no error raised. Choosing a stronger quantization is the correct response to insufficient VRAM; enabling offload is not.

device_map={"": 0}     # correct
device_map="auto"      # silently breaks unless the model fits entirely

8-bit

quantization = BitsAndBytesConfig(load_in_8bit=True)

Expected ~10.1 GB of weights. Not validated here — no result in this repository was produced in 8-bit. If you use it, run evaluation/run_smoke_eval.py first; its first check exists to catch a degenerate model.

GGUF — why there isn't one

llama.cpp support depends on the architecture being implemented there. Piko-9b is qwen3_5: a hybrid stack of gated linear attention plus periodic full attention, with mRoPE and a Qwen3.5 vision tower. Converting it means the converter must handle:

linear-attention layers (in_proj_qkv, in_proj_a/b/z, A_log, dt_bias, conv1d)
the full_attention_interval layer pattern
mRoPE with mrope_section [11, 11, 10] and partial_rotary_factor 0.25
a separate multimodal projector for the vision tower

A GGUF of an earlier, text-only checkpoint in this project's history exists locally (Ornith-1.0-9B.BF16.gguf plus an mmproj file), which shows the family can be converted. That is not this checkpoint and says nothing about whether the published composition converts cleanly.

No GGUF is published, and none should be assumed to work until someone runs scripts/validate_quantized_model.py against it.

AWQ and GPTQ

Scripts are provided but were not run:

Both are activation-aware methods that need a calibration pass. Two specific risks for this architecture:

Linear-attention projections. A_log and dt_bias are not ordinary linear weights. Quantizing them like q_proj can destabilise the recurrent state. Both scripts exclude them by default.
The vision tower. Most AWQ/GPTQ tooling calibrates on text only. A text-calibrated quantizer applied to the vision tower can degrade image handling in a way no text benchmark will show. Both scripts leave the vision tower in bf16 by default.

Validating any quantized build

Never publish a quantized artefact without this:

python scripts/validate_quantized_model.py \
  --quantized ./piko-9b-awq \
  --reference Dexy2/Piko-9b \
  --output reports/quantization_validation.json

It checks loading, degenerate output, text quality against the reference, image input, memory, and throughput. A quantization that passes text checks but breaks the vision path is the most likely failure mode here, and the easiest to miss.

Choosing

You have	Use
≥ 22 GB VRAM	bfloat16, `device_map={"": 0}`
12–22 GB	4-bit NF4 (validated) or 8-bit (untested)
8–12 GB	4-bit NF4
< 8 GB	Not supported. Do not use offload