LabOS-VLM-7B

LabOS-VLM-7B is a PEFT LoRA adapter for Qwen/Qwen2.5-VL-7B-Instruct, fine-tuned on wet-lab supervision prompts for protocol monitoring, step prediction, pipette mistake detection, spatial grounding, protocol generation, and protocol VQA over egocentric and third-person lab videos; it was trained on the prompt families included in this repository, so it should work best with the same JSON-style monitoring and benchmark prompts and should be expected to generalize less reliably to prompts outside that distribution.

Adapter Details

• Base model: Qwen/Qwen2.5-VL-7B-Instruct
• Adapter type: LoRA via PEFT / MS-SWIFT
• Rank: 32
• LoRA alpha: 64
• LoRA dropout: 0.05
• Target modules: Qwen language-model projection layers matching q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• Training checkpoint: checkpoint-750
• Training data: lab_supervision_25k_train.jsonl / lab_supervision_25k_val.jsonl

Training Parameters

• Trainer: MS-SWIFT SFT with PEFT LoRA
• Epochs: 2.0
• Devices: 8 GPUs (global_world_size=8)
• Per-device train batch size: 1
• Gradient accumulation steps: 4
• Effective global train batch size: 32 examples per optimizer step
• Per-device eval batch size: 1
• Learning rate: 1e-4
• Scheduler: cosine
• Warmup ratio: 0.03
• Optimizer: adamw_torch_fused
• Weight decay: 0.1
• Adam betas: (0.9, 0.95)
• Max gradient norm: 1.0
• Precision: bfloat16
• Max sequence length: 4096
• Gradient checkpointing: enabled
• DeepSpeed: ZeRO-2
• Vision tower: frozen (freeze_vit=true)
• Aligner: frozen (freeze_aligner=true)
• Evaluation interval: every 250 steps
• Checkpoint interval: every 250 steps

Run With MS-SWIFT

Install the Qwen video runtime dependencies and MS-SWIFT in your environment, then run:

swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters labos1/labos-vlm-7b \
  --infer_backend pt

For local staged weights before upload:

swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters /data/david/models/release/labos-vlm-7b \
  --infer_backend pt

Run With Transformers And PEFT

from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

base_model = "Qwen/Qwen2.5-VL-7B-Instruct"
adapter = "labos1/labos-vlm-7b"

processor = AutoProcessor.from_pretrained(base_model)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Example Monitoring Prompt

This adapter was fine-tuned with prompts similar to:

You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.

The current protocol state/history is provided below. Watch the current window and update the state.

Report protocol errors only when supported by the visible time window or state.

Ignore irrelevant unknown keys in the state JSON.

Compare the protocol order, prior history, and watched window.

Identify the main protocol step being performed in this watched video window.

STATE:
{"equipment":["P20 pipette","sterile tips","microcentrifuge tube rack"],"history":[{"step":"1","tas":20,"tds":20}],"objects":{},"on":"2","protocol":[{"desc":"Place the tubes in a clean rack.","order":1,"step":"1"},{"desc":"Attach a new sterile pipette tip.","order":2,"step":"2"},{"desc":"Aspirate reagent without touching the tube wall.","order":3,"step":"3"}],"protocol_summary":"Demonstration of aseptic pipetting technique.","reagents":["sample buffer"]}

Return strict JSON only.

## Response Format
{
  "explanation": "string",
  "observed_step_id": "string or null"
}

## Question
Which protocol step is being performed in this video window?

Run python generate_monitoring_prompts.py from this folder to print example monitoring prompts that match the training/evaluation prompt style.