Instructions to use ayushadarsh7/kayak_v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ayushadarsh7/kayak_v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ayushadarsh7/kayak_v2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ayushadarsh7/kayak_v2")
model = AutoModelForMultimodalLM.from_pretrained("ayushadarsh7/kayak_v2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ayushadarsh7/kayak_v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ayushadarsh7/kayak_v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayushadarsh7/kayak_v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ayushadarsh7/kayak_v2

SGLang

How to use ayushadarsh7/kayak_v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ayushadarsh7/kayak_v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayushadarsh7/kayak_v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ayushadarsh7/kayak_v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ayushadarsh7/kayak_v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ayushadarsh7/kayak_v2 with Docker Model Runner:
```
docker model run hf.co/ayushadarsh7/kayak_v2
```

kayak_v2 / evaluation /protocol.json

ayushadarsh7

Add held-out evaluation metrics and scenario-by-scenario failed-task reports

8044386 verified 15 days ago

Raw History Blame Contribute Delete

2.27 kB

{
  "model": "ayushadarsh7/kayak_v2",
  "model_revision": "8f741f994f694670d47330488e90f5c835cec2fa",
  "model_dtype": "float32",
  "weight_hashes_verified": true,
  "weights_source": "local export identical by SHA-256 to public model shards",
  "test_sha256": "fab178212b171cc35e45ff1f9fa350f6a21138f869103c64d6ef8d3c43684a7b",
  "train_sha256": "86b71f7712c8c8194e67ead954d259de1649a45d5237ca52382fb95588bc46ab",
  "examples": 147,
  "episodes": 8,
  "seed": 1,
  "attention": "sdpa",
  "tf32": true,
  "deterministic_algorithms": true,
  "do_sample": false,
  "max_new_tokens": 128,
  "min_pixels": 262144,
  "max_pixels": 524288,
  "teacher_forced_loss": "Assistant-token-weighted negative log likelihood; includes assistant end-of-turn tokens",
  "success_rule": "Parsed action matches; valid xywh box; predicted box center lies inside target; exact fill value or press key when applicable. IoU has no threshold in this success rule.",
  "json_ok_rule": "Existing grounding parser can extract a JSON object from surrounding text; strict_json_ok additionally requires the entire response to be one JSON object.",
  "bbox_valid_rule": "Finite numeric normalized xywh; x/y>=0; w/h>0; components<=1000; right/bottom<=1001 (inherited scorer tolerance).",
  "seen_box_rule": "Exact normalized bbox tuple occurs in the training set; this is not semantic element familiarity.",
  "scenario_status_rule": "all_evaluated_steps_passed only if every held-out eligible click/fill example in that episode passes independently; this is not end-to-end browser success.",
  "limitations": [
    "One shared browser session.",
    "Nine cross-split similar-image pairs flagged by the split audit.",
    "Only eligible click/fill steps evaluated; scroll, wait, drag, press and excluded captures are outside this test.",
    "No model selection or tuning performed on this test set.",
    "Inference limit 128 matches published helper; legacy study evaluator default is 64, so settings are not identical to older runs."
  ],
  "versions": {
    "torch": "2.9.0+cu128",
    "transformers": "4.57.1",
    "accelerate": "1.11.0",
    "Pillow": "12.3.0"
  },
  "started_utc": "2026-10-24T19:24:13Z",
  "script_sha256": "c7b4a2443c0e1446df81e22452dc23b7ddc05664d970d21828c255081a30440f"
}

	{
	"model": "ayushadarsh7/kayak_v2",
	"model_revision": "8f741f994f694670d47330488e90f5c835cec2fa",
	"model_dtype": "float32",
	"weight_hashes_verified": true,
	"weights_source": "local export identical by SHA-256 to public model shards",
	"test_sha256": "fab178212b171cc35e45ff1f9fa350f6a21138f869103c64d6ef8d3c43684a7b",
	"train_sha256": "86b71f7712c8c8194e67ead954d259de1649a45d5237ca52382fb95588bc46ab",
	"examples": 147,
	"episodes": 8,
	"seed": 1,
	"attention": "sdpa",
	"tf32": true,
	"deterministic_algorithms": true,
	"do_sample": false,
	"max_new_tokens": 128,
	"min_pixels": 262144,
	"max_pixels": 524288,
	"teacher_forced_loss": "Assistant-token-weighted negative log likelihood; includes assistant end-of-turn tokens",
	"success_rule": "Parsed action matches; valid xywh box; predicted box center lies inside target; exact fill value or press key when applicable. IoU has no threshold in this success rule.",
	"json_ok_rule": "Existing grounding parser can extract a JSON object from surrounding text; strict_json_ok additionally requires the entire response to be one JSON object.",
	"bbox_valid_rule": "Finite numeric normalized xywh; x/y>=0; w/h>0; components<=1000; right/bottom<=1001 (inherited scorer tolerance).",
	"seen_box_rule": "Exact normalized bbox tuple occurs in the training set; this is not semantic element familiarity.",
	"scenario_status_rule": "all_evaluated_steps_passed only if every held-out eligible click/fill example in that episode passes independently; this is not end-to-end browser success.",
	"limitations": [
	"One shared browser session.",
	"Nine cross-split similar-image pairs flagged by the split audit.",
	"Only eligible click/fill steps evaluated; scroll, wait, drag, press and excluded captures are outside this test.",
	"No model selection or tuning performed on this test set.",
	"Inference limit 128 matches published helper; legacy study evaluator default is 64, so settings are not identical to older runs."
	],
	"versions": {
	"torch": "2.9.0+cu128",
	"transformers": "4.57.1",
	"accelerate": "1.11.0",
	"Pillow": "12.3.0"
	},
	"started_utc": "2026-10-24T19:24:13Z",
	"script_sha256": "c7b4a2443c0e1446df81e22452dc23b7ddc05664d970d21828c255081a30440f"
	}