Instructions to use Kaelyn01/VisAlloc-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kaelyn01/VisAlloc-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="Kaelyn01/VisAlloc-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Kaelyn01/VisAlloc-9B") model = AutoModelForMultimodalLM.from_pretrained("Kaelyn01/VisAlloc-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kaelyn01/VisAlloc-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kaelyn01/VisAlloc-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kaelyn01/VisAlloc-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/Kaelyn01/VisAlloc-9B
- SGLang
How to use Kaelyn01/VisAlloc-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kaelyn01/VisAlloc-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kaelyn01/VisAlloc-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kaelyn01/VisAlloc-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kaelyn01/VisAlloc-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use Kaelyn01/VisAlloc-9B with Docker Model Runner:
docker model run hf.co/Kaelyn01/VisAlloc-9B
VisAlloc-9B
๐ค Training Data | ๐ฆ VisAlloc-4B | ๐ฆ VisAlloc-9B
Model Summary
VisAlloc-9B is a multimodal LLM built on the Qwen/Qwen3.5-9B model family and trained with VisAlloc, the visual self-distillation framework introduced in VisAlloc: Allocating Visual Supervision for On-Policy Self-Distillation.
VisAlloc allocates privileged visual supervision at two levels. A fixed paired-teacher filter first retains examples answered correctly from an evidence-centered positive crop and incorrectly from a spatially displaced negative crop, after annotation-quality checks. During training, both teacher views score the same full-image student's response under aligned prefixes. Their view sensitivity and distribution disagreement determine token weights for distillation from the positive teacher.
The resulting model accepts the full image at inference. Paired teacher crops and token-weight construction are used during training. See also VisAlloc-4B.
Key Features
- ๐ Full-image inference โ visual question answering through the standard multimodal model interface, without inference-time cropping or zooming tools.
- ๐ช Paired-view self-distillation โ positive and negative teacher passes share one detached EMA model and score the same student-generated response.
- ๐ฏ Selective token supervision โ a continuous sensitivity base is combined with additional weight admitted by sensitivity or teacher-distribution disagreement under shared context gates.
- โ๏ธ Frequency-aware allocation โ inverse-frequency redistribution preserves the total additional weight within each micro-batch before mean normalization.
- ๐ฆ Standard model release โ complete BF16 safetensors weights, model configuration, tokenizer, processor, and chat template.
How It Works
- Paired-view construction and filtering โ pair an evidence-centered positive crop with a geometry-matched, spatially displaced negative crop. Before optimization, a fixed teacher independently answers from both views; retain positive-correct, negative-incorrect examples with reliable annotations.
- Shared student trajectory โ the full-image student generates a response. The positive and negative teacher passes use the same detached EMA parameters and score every position under the same textual prefix.
- Token-level allocation โ absolute realized-token log-probability differences supply the continuous base. Sensitivity or teacherโteacher JSD admits additional weight under frozen-prior, template-bigram, and studentโteacher JSD gates. Inverse-frequency redistribution preserves its total, followed by mean normalization.
- Positive-teacher distillation โ minimize weighted token-level JSD between the student and the positive teacher on the student-selected top-100 vocabulary entries plus a residual tail bucket. Gradients pass through the student; the negative teacher determines weights.
- EMA regularization โ accumulate micro-batch gradients before each mini-batch optimizer step. Keep the teacher fixed throughout the actor update, then update it once if the student was updated.
Quickstart
Weights load with standard transformers:
pip install "transformers>=5.16.1" accelerate torch pillow
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "Kaelyn01/VisAlloc-9B"
model = AutoModelForImageTextToText.from_pretrained(
model_id, dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)
messages = [{
"role": "user",
"content": [
{"type": "image", "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
{"type": "text", "text": "Describe the main objects in this image."},
],
}]
inputs = processor.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
eos_token_id=processor.tokenizer.eos_token_id,
pad_token_id=processor.tokenizer.pad_token_id,
)
print(processor.decode(
outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))
Or serve with vLLM using its OpenAI-compatible API:
vllm serve Kaelyn01/VisAlloc-9B --port 8000
Training Details
| Base model family | Qwen/Qwen3.5-9B |
| Data | VisAlloc-6K, derived from Vision-OPD-6K; 5,795 retained examples from 6,241 source pairs |
| Student input | Full image and question |
| Teacher views | Evidence-centered positive crop and spatially displaced negative crop, sharing detached EMA parameters |
| Objective | Weighted token-level JSD, mixing coefficient 0.5; student-selected top-100 support plus a residual tail bucket |
| Token allocation | Continuous sensitivity base, gated additional weight, inverse-frequency redistribution, and mean normalization |
| Additional-weight strength | 2.0 |
| Teacher update | EMA rate 0.05, once after the actor update |
| Rollout length | Up to 1,024 tokens, 1 epoch |
| Released weights | BF16 safetensors |
The paired data and sample-level selection procedure are documented in VisAlloc-6K. The regional-to-global self-distillation setup builds on Vision-OPD.
Citation
Please cite the VisAlloc paper for the method. To reference this specific model release, cite the repository entry and record the revision you used.
@misc{xu2026visalloc,
title={{VisAlloc}: Allocating Visual Supervision for On-Policy Self-Distillation},
author={Xu, Haoran and Chen, Tian and Wu, Yuhe and Zhao, Yanbo and Liao, Junchi and Wang, Hongyu and Li, Jiaze and Wang, Wentao and Li, Yanfeng and Qian, Rui},
year={2026}
}
@misc{kaelyn01_2026_visalloc_9b,
author={{Kaelyn01}},
title={{VisAlloc-9B}},
year={2026},
howpublished={Hugging Face model repository},
url={https://huggingface.co/Kaelyn01/VisAlloc-9B}
}
License
Released under Apache-2.0, following the base model Qwen/Qwen3.5-9B. The training dataset's component-specific terms are documented separately in VisAlloc-6K.
- Downloads last month
- 30