README.md

---
base_model: Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
license: cc-by-nc-4.0
tags:
  - lora
  - peft
  - qwen2_5_vl
  - video
  - video-language-model
  - lab-supervision
  - wet-lab
  - protocol-monitoring
  - error-detection
  - spatial-grounding
---

# LabOS-VLM-7B

LabOS-VLM-7B is a PEFT LoRA adapter for `Qwen/Qwen2.5-VL-7B-Instruct`, fine-tuned on wet-lab supervision tasks from the LabOS JoVE and FineBio datasets and validated on the LSV benchmark. These tasks include protocol monitoring/step prediction, error detection, spatial grounding, protocol generation, and general VQA with first-person and third-person views.

The adapter is intended for research on laboratory video-language assistants. It works best with the JSON-style monitoring and benchmark prompts used by the LabOS datasets, for example the public LSV benchmark at [`cong-lab/lsv`](https://huggingface.co/datasets/cong-lab/lsv). Note, the expected performance may drop when evaluating on different modalities, or prompts, as is typically expected with SFT.

## Adapter Details

- Base model: `Qwen/Qwen2.5-VL-7B-Instruct`
- Adapter repo: `cong-lab/labos-vlm-7b`
- Adapter type: LoRA via PEFT / MS-SWIFT
- Rank / alpha / dropout: `32 / 64 / 0.05`
- Target modules: Qwen language-model projection layers matching `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, and `down_proj`
- Training data: FineBio + JoVE

## Install

Install a CUDA-enabled PyTorch stack plus the model runtime dependencies:

```bash
python -m pip install -r requirements.txt
```

For Qwen2.5-VL video inference, `flash-attn` is recommended.

## Run With MS-SWIFT

```bash
swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters cong-lab/labos-vlm-7b \
  --infer_backend pt
```

For a locally downloaded or modified adapter, replace the adapter ID with a local folder path:

```bash
swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters ./labos-vlm-7b \
  --infer_backend pt
```

## Run With Transformers And PEFT

```python
from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

base_model = "Qwen/Qwen2.5-VL-7B-Instruct"
adapter = "cong-lab/labos-vlm-7b"  # or a path to a local adapter folder

processor = AutoProcessor.from_pretrained(base_model)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)
```

## Evaluate On LSV

The public LSV benchmark (`cong-lab/lsv`) includes video manifests, prompt loaders, and report generation for step prediction, monitoring-state advancement, and error detection.

```bash
hf download cong-lab/lsv --repo-type dataset --local-dir ./lsv
cd lsv
python -m pip install -r requirements.txt
python inference.py \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapter cong-lab/labos-vlm-7b \
  --output runs/labos_vlm_7b \
  --gpus 0
python generate_report.py \
  --output runs/labos_vlm_7b \
  --report-dir runs/labos_vlm_7b_report
```

## Example Monitoring Prompt

This adapter was fine-tuned with prompts similar to:

```text
You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.

The current protocol state/history is provided below. Watch the current window and update the state.

Report protocol errors only when supported by the visible time window or state.

Compare the protocol order, prior history, and watched window.

Identify the main protocol step being performed in this watched video window.

STATE:
{"history":[{"step":"2","tas":0,"tds":0},{"step":"3","tas":30,"tds":30}],"on":"3","protocol":[{"desc":"Take HEK293T cells and culture them to ~70% confluency in a 10 cm dish.","order":1,"step":"1"},{"desc":"In a sterile 1.5 mL tube, mix lentiviral backbone, packaging plasmid, and envelope plasmid.","order":2,"step":"2"},{"desc":"Add transfection reagent and bring to volume with serum-free medium.","order":3,"step":"3"},{"desc":"Incubate the mixture at room temperature for 15 minutes.","order":4,"step":"4"}]}

Return the visible protocol step ID for the watched video window.
```

Run `python generate_monitoring_prompts.py` from this repository to print additional self-contained monitoring examples.

## Training Parameters

- Trainer: MS-SWIFT SFT with PEFT LoRA
- Epochs: `2.0` (best chkpt @1ep)
- Devices: `8` H100 GPUs
- Per-device train batch size: `1`
- Gradient accumulation steps: `4`
- Effective global train batch size: `32` examples per optimizer step
- Per-device eval batch size: `1`
- Learning rate: `1e-4`
- Scheduler: cosine
- Warmup ratio: `0.03`
- Optimizer: `adamw_torch_fused`
- Weight decay: `0.1`
- Adam betas: `(0.9, 0.95)`
- Max gradient norm: `1.0`
- Precision: `bfloat16`
- Max sequence length: `4096`
- Gradient checkpointing: enabled
- DeepSpeed: ZeRO-2
- Vision tower and aligner: frozen
- Evaluation/checkpoint interval: every `250` steps

## Limitations

This adapter is intended for research on wet-lab video supervision and should not be used as the sole source of truth for laboratory safety or procedural correctness. Human review remains required for wet-lab execution.

## License

This adapter is released for non-commercial research use under the Creative Commons Attribution-NonCommercial 4.0 license (`CC-BY-NC-4.0`), matching the public LSV benchmark license.

## Research Use Only

This model is provided for research purposes only and for non-commercial use. It is not intended for clinical decision-making or replacing trained human supervision in real wet-lab procedures.