Qwen3-VL-8B AURA Focal

Final model from one epoch of AURA full fine-tuning of Qwen/Qwen3-VL-8B-Instruct with token-level Focal loss (gamma=2.0). This repository contains the final BF16 model weights, tokenizer, custom model and processor code, and training summary. External evaluation has not been performed.

Training

  • Base revision: 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b.
  • 173,553 cached training examples; 2,712 optimizer steps; one epoch.
  • 32 GPUs; per-device batch 1; gradient accumulation 2; global batch 64.
  • Learning rate 1e-5; cosine scheduler; warmup ratio 0.1; seed 42.
  • BF16, FlashAttention 2, DeepSpeed ZeRO-3; sequence limit 11,000 tokens.
  • AURA answer selection and silent-token weighting are preserved. Focal modulation applies to all retained supervised tokens, with the original weighted-token denominator.
  • Visual patch embedding, transformer blocks, deepstack mergers and projector are frozen. model.visual.pos_embed.weight remains trainable to match the CE baseline. The language model and LM head are trainable.

Mean training Focal loss: 0.493920. Final logged step loss: 0.3440. Focal loss values cannot be directly compared with CE values as model-quality scores.

Training loss

Loading

Tested with Transformers 4.57.1 and PyTorch 2.6.0.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "leobo09263013/focal"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)

The checkpoint retains the custom <|silent|> token (ID 151669). The saved Fast tokenizer removes it when skip_special_tokens=True; account for that when evaluating silence decisions.

For continued training, Focal is a runtime loss hook and is not restored by loading model weights alone. Reattach it from training/aura_loss.py using attach_aura_loss(model, processor.tokenizer, "focal", focal_gamma=2.0). The original CE model implementation is retained in the custom model file. This export does not contain the distributed optimizer state.

Downloads last month
32
Safetensors
Model size
770k params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for leobo09263013/focal

Finetuned
(754)
this model