VisAlloc-9B

๐Ÿค— Training Data  |  ๐Ÿ“ฆ VisAlloc-4B  |  ๐Ÿ“ฆ VisAlloc-9B

Model Summary

VisAlloc-9B is a multimodal LLM built on the Qwen/Qwen3.5-9B model family and trained with VisAlloc, the visual self-distillation framework introduced in VisAlloc: Allocating Visual Supervision for On-Policy Self-Distillation.

VisAlloc allocates privileged visual supervision at two levels. A fixed paired-teacher filter first retains examples answered correctly from an evidence-centered positive crop and incorrectly from a spatially displaced negative crop, after annotation-quality checks. During training, both teacher views score the same full-image student's response under aligned prefixes. Their view sensitivity and distribution disagreement determine token weights for distillation from the positive teacher.

The resulting model accepts the full image at inference. Paired teacher crops and token-weight construction are used during training. See also VisAlloc-4B.

Key Features

  • ๐Ÿ” Full-image inference โ€” visual question answering through the standard multimodal model interface, without inference-time cropping or zooming tools.
  • ๐Ÿชž Paired-view self-distillation โ€” positive and negative teacher passes share one detached EMA model and score the same student-generated response.
  • ๐ŸŽฏ Selective token supervision โ€” a continuous sensitivity base is combined with additional weight admitted by sensitivity or teacher-distribution disagreement under shared context gates.
  • โš–๏ธ Frequency-aware allocation โ€” inverse-frequency redistribution preserves the total additional weight within each micro-batch before mean normalization.
  • ๐Ÿ“ฆ Standard model release โ€” complete BF16 safetensors weights, model configuration, tokenizer, processor, and chat template.

How It Works

  1. Paired-view construction and filtering โ€” pair an evidence-centered positive crop with a geometry-matched, spatially displaced negative crop. Before optimization, a fixed teacher independently answers from both views; retain positive-correct, negative-incorrect examples with reliable annotations.
  2. Shared student trajectory โ€” the full-image student generates a response. The positive and negative teacher passes use the same detached EMA parameters and score every position under the same textual prefix.
  3. Token-level allocation โ€” absolute realized-token log-probability differences supply the continuous base. Sensitivity or teacherโ€“teacher JSD admits additional weight under frozen-prior, template-bigram, and studentโ€“teacher JSD gates. Inverse-frequency redistribution preserves its total, followed by mean normalization.
  4. Positive-teacher distillation โ€” minimize weighted token-level JSD between the student and the positive teacher on the student-selected top-100 vocabulary entries plus a residual tail bucket. Gradients pass through the student; the negative teacher determines weights.
  5. EMA regularization โ€” accumulate micro-batch gradients before each mini-batch optimizer step. Keep the teacher fixed throughout the actor update, then update it once if the student was updated.

Quickstart

Weights load with standard transformers:

pip install "transformers>=5.16.1" accelerate torch pillow
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "Kaelyn01/VisAlloc-9B"
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype="bfloat16", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
        {"type": "text", "text": "Describe the main objects in this image."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
    eos_token_id=processor.tokenizer.eos_token_id,
    pad_token_id=processor.tokenizer.pad_token_id,
)
print(processor.decode(
    outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))

Or serve with vLLM using its OpenAI-compatible API:

vllm serve Kaelyn01/VisAlloc-9B --port 8000

Training Details

Base model family Qwen/Qwen3.5-9B
Data VisAlloc-6K, derived from Vision-OPD-6K; 5,795 retained examples from 6,241 source pairs
Student input Full image and question
Teacher views Evidence-centered positive crop and spatially displaced negative crop, sharing detached EMA parameters
Objective Weighted token-level JSD, mixing coefficient 0.5; student-selected top-100 support plus a residual tail bucket
Token allocation Continuous sensitivity base, gated additional weight, inverse-frequency redistribution, and mean normalization
Additional-weight strength 2.0
Teacher update EMA rate 0.05, once after the actor update
Rollout length Up to 1,024 tokens, 1 epoch
Released weights BF16 safetensors

The paired data and sample-level selection procedure are documented in VisAlloc-6K. The regional-to-global self-distillation setup builds on Vision-OPD.

Citation

Please cite the VisAlloc paper for the method. To reference this specific model release, cite the repository entry and record the revision you used.

@misc{xu2026visalloc,
  title={{VisAlloc}: Allocating Visual Supervision for On-Policy Self-Distillation},
  author={Xu, Haoran and Chen, Tian and Wu, Yuhe and Zhao, Yanbo and Liao, Junchi and Wang, Hongyu and Li, Jiaze and Wang, Wentao and Li, Yanfeng and Qian, Rui},
  year={2026}
}

@misc{kaelyn01_2026_visalloc_9b,
  author={{Kaelyn01}},
  title={{VisAlloc-9B}},
  year={2026},
  howpublished={Hugging Face model repository},
  url={https://huggingface.co/Kaelyn01/VisAlloc-9B}
}

License

Released under Apache-2.0, following the base model Qwen/Qwen3.5-9B. The training dataset's component-specific terms are documented separately in VisAlloc-6K.

Downloads last month
30
Safetensors
Model size
9B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Kaelyn01/VisAlloc-9B

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1,443)
this model
Quantizations
2 models

Dataset used to train Kaelyn01/VisAlloc-9B