kayak_v2

ayushadarsh7/kayak_v2 is a full-parameter fine-tune of Qwen3-VL-4B-Instruct for single-step visual grounding on Kayak screenshots.

Input: a clean screenshot and a specific action instruction.
Output: action JSON with a target box, and an exact text value for fill actions.
Scope: locating controls for click/fill actions. It does not plan a complete trip, execute browser actions, or verify a completed workflow.

The repository contains merged FP32 weights in standard Hugging Face safetensors format. No adapter merging, custom model code, or trust_remote_code=True is required. The weights were reconstructed from the pinned base plus the trained BF16-saved parameter delta; they are not bit-identical to the original FP32 training master weights because delta storage was lossy.

Quick start

A CUDA GPU is recommended. FP32 model weights alone occupy about 17.8 GB; allow additional memory for vision activations, KV cache, and generation. Actual inference memory requirements depend on the screenshot and generation length.

Download the helper and its dependencies:

pip install 'huggingface_hub>=0.34,<1.0'
hf download ayushadarsh7/kayak_v2 infer.py requirements.txt --local-dir kayak_v2_code
pip install -r kayak_v2_code/requirements.txt

Install a PyTorch build suitable for your GPU/CUDA version if the default package does not match your machine. The tested environment uses Python 3.12, PyTorch 2.9.0+cu128, Transformers 4.57.1, and Accelerate 1.11.0.

Click example:

python kayak_v2_code/infer.py \
  --image /path/to/screenshot.png \
  --instruction 'Click Search.'

Fill example:

python kayak_v2_code/infer.py \
  --image /path/to/screenshot.png \
  --instruction 'Enter New Delhi in the origin field.' \
  --value 'New Delhi'

To clear a field, pass --value '' with a clear-field instruction. Empty strings are intentional training targets.

The helper defaults to --model ayushadarsh7/kayak_v2 --device cuda:0 --dtype float32. It uses SDPA, greedy decoding, and up to 128 new tokens. It also applies the mathematically equivalent linear vision-patch projection used by the training project to avoid a slow Conv3d path on some Blackwell systems. It prints the prediction and does not click or type anything.

For lower memory use, --dtype bfloat16 is available, but rounding full weights can change small fine-tuning updates; it has not been separately benchmarked for this release. FP32 is the default for preserving the reconstructed weights. CPU execution is possible with --device cpu --dtype float32, but slower.

Python API

After downloading infer.py, from the kayak_v2_code directory:

from infer import Grounder

grounder = Grounder('ayushadarsh7/kayak_v2', device='cuda:0', dtype='float32')
prediction = grounder.predict(
    '/path/to/screenshot.png',
    'Enter New Delhi in the origin field.',
    value='New Delhi',
)
print(prediction)

Standard Transformers loading also works:

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

processor = AutoProcessor.from_pretrained(
    'ayushadarsh7/kayak_v2', min_pixels=262144, max_pixels=524288,
)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    'ayushadarsh7/kayak_v2', dtype=torch.float32, attn_implementation='sdpa',
).to('cuda:0').eval()

Use the helper for the complete prompt/image preprocessing and generation procedure. The processor and chat template are shipped with the model.

Prompt and output contract

The training prompt is:

Task: <instruction>
Text to enter: "<value>"
Respond with only JSON using action and the applicable bbox, value, or key fields. bbox is [x, y, width, height] on a 0-1000 scale relative to the image; x/y is the top-left corner.

The Text to enter line is present only for fill actions; the value is JSON-escaped. The user message contains the screenshot followed by this text. Do not provide red-box overlays or after-action screenshots as grounding inputs.

Illustrative output (not a measured prediction):

{"action":"click","bbox":[100,200,150,50]}

bbox is [x, y, width, height], normalized to 0–1000. Decimal coordinates are allowed. It is not [x1, y1, x2, y2] and is not in pixels. For an original image of width W and height H, the box center in pixels is:

x, y, w, h = prediction['bbox']
click_x = (x + w / 2) * W / 1000
click_y = (y + h / 2) * H / 1000

Parse and validate generated JSON and image bounds before using it in an application. The inherited prompt mentions key, but this fine-tune's 728 eligible examples contain only click and fill actions; keyboard, scroll, drag, planner and checker behavior are not trained or evaluated by this release.

Training data and split

The source collection is kayak-v2-batch-2026-10-23: 40 recorded episodes and 973 actions. Its filtered clicker export contains 728 grounding examples from 39 usable episodes: 626 clicks and 102 fills, including 44 field-clearing fills.

Product	Train episodes	Test episodes	Train examples	Test examples
Flights	8	2	125	43
Hotels / Stays	8	2	155	38
Car rental	8	2	147	37
Flight + Hotel packages	7	2	154	29
Total	31	8	581	147

Whole episodes stay in one split. The allocation balances functional element categories and action types. All 20 recurring categories appear in both splits; hotel-name search, map-location search and result comparison occur in only one episode each and stay in training. All examples are used once, with no episode or byte-identical screenshot overlap.

There are nine flagged cross-split similar-image pairs, primarily recurring search-page layouts with different form values or menu states. All episodes came from one shared browser session. This is a within-session split, not evidence of independent-session or unseen-site generalization. Source records remain pending independent acceptance; collection review and structural validation are not human-approval guarantees. Raw data and screenshots are not distributed in this model repository.

Training configuration

Setting	Value
Base	Qwen/Qwen3-VL-4B-Instruct
Pinned base revision	ebb281ec70b05090aa6165b016eac8ec08e71b17
Method	Full-parameter SFT; assistant-answer-only loss
Trainable parameters	Approximately 4.438 billion
Epochs / optimizer steps	4 / 148
Seed	1
Learning rate	5e-6
Optimizer	AdamW, betas (0.9, 0.999), epsilon 1e-8
Scheduler	3% warmup, then cosine decay
Effective batch	16 (micro-batch 4 × 4 accumulation passes)
Weight decay / gradient clipping	0 / 1.0
Precision	FP32 master weights, BF16 autocast compute
Attention / memory	SDPA; gradient checkpointing
Image pixels	262,144 minimum; 524,288 maximum
Maximum training sequence	2,048 tokens
GPU	NVIDIA RTX PRO 6000 Blackwell Server Edition
Training duration	567.8 seconds
Peak allocated training GPU memory	89.4 GB
Aggregate training loss	0.535825

The learning rate and four-epoch schedule were inherited from a previous study run, not selected using this V2 test set. Settings and checkpoint provenance are also in training_metadata.json.

Held-out evaluation results

Evaluated the frozen 147-example / 8-episode test split once, using the published FP32 model, deterministic SDPA with TF32 enabled, greedy decoding, and 128 generated tokens. The token limit matches the published inference helper; older study evaluations used 64. Grounding success requires the correct action, valid box, predicted center inside the target, and exact fill text when applicable. Scenario status covers independent recorded screenshots, not live end-to-end workflow success.

Metric	Held-out result
Grounding + action success	137/147 (93.20%)
Box-center hit rate	93.88%
Action accuracy	99.32%
Parsed / strict JSON	100.00% / 100.00%
Valid bounding boxes	100.00%
Exact fill value	25/25 (100.00%)
Mean / median IoU	0.796108 / 0.893316
IoU >= 0.50 / >= 0.75	88.44% / 74.15%
Assistant-token loss / perplexity	0.502382 / 1.652653
Episodes with every evaluated step passing	4/8
Generation limit reached	0
Press-key exact match	N/A: no press examples

Episode	Product	Passed / evaluated	Failed original steps
b08-packages-august	packages	19/19	None
b09-flights-airline	flights	16/19	18, 19, 21
b11-cars-search	cars	13/13	None
b12-packages-search	packages	10/10	None
b21-flights-multicity	flights	20/24	13, 19, 21, 22
b26-stays-named-gallery	stays	12/14	2, 4
b27-cars-van-passengers	cars	24/24	None
b34-stays-stay-total-budget	stays	23/24	19

The separate scenario_results.md report lists every evaluated task with its original step number and PASS/FAIL status. For each failure it includes the expected JSON, actual model output and failure reason. See scenario_results.json for structured results and failed_steps.jsonl for failures only.

Full metrics: evaluation_results.md, summary.json, and metrics_by_group.json. Every prediction is saved in predictions.jsonl; protocol.json records settings and hashes; verification.json records independent rescoring and coverage checks.

All 147 expected IDs were evaluated exactly once. Original labels, tasks and step numbers were checked against the frozen source, every prediction was rescored, and scenario totals reconciled with overall metrics. The shared-session and near-image limitations still apply; these numbers are not an unseen-site or live-workflow benchmark.

Evaluation and limitations

Held-out evaluation is now complete. The 147-example test set was not used during training or checkpoint selection. The results below are for this model; the training loss above and previous Kayak-model results are separate.

All converted examples passed the actual training collator's token-prefix, answer-mask and sequence-length checks. The exported model and inference helper passed a generation smoke check on training examples; that checks loading and output wiring, not generalization. See inference_validation.json for the recorded result.

The model is specialized to the captured Kayak UI and English instructions. Layout, locale, content, scaling and website changes may affect performance. It may produce invalid JSON, incorrect boxes or incorrect text. It has not been evaluated for end-to-end browser tasks, other websites, or safety-critical actions.

License and base model

Apache-2.0; see LICENSE. The base model is Qwen3-VL-4B-Instruct, whose upstream model card identifies the same license. This model is a community fine-tune and is not affiliated with or endorsed by Kayak or Qwen.