Initial commit

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
 *.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,154 @@
+---
+base_model: Qwen/Qwen2.5-VL-7B-Instruct
+library_name: peft
+pipeline_tag: image-text-to-text
+license: cc-by-nc-4.0
+tags:
+  - lora
+  - peft
+  - qwen2_5_vl
+  - video
+  - video-language-model
+  - lab-supervision
+  - wet-lab
+  - protocol-monitoring
+  - error-detection
+  - spatial-grounding
+---
+
+# LabOS-VLM-7B
+
+LabOS-VLM-7B is a PEFT LoRA adapter for `Qwen/Qwen2.5-VL-7B-Instruct`, fine-tuned on wet-lab supervision tasks from the LabOS JoVE and FineBio datasets and validated on the LSV benchmark. These tasks include protocol monitoring/step prediction, error detection, spatial grounding, protocol generation, and general VQA with first-person and third-person views.
+
+The adapter is intended for research on laboratory video-language assistants. It works best with the JSON-style monitoring and benchmark prompts used by the LabOS datasets, for example the public LSV benchmark at [`cong-lab/lsv`](https://huggingface.co/datasets/cong-lab/lsv).
+
+## Adapter Details
+
+- Base model: `Qwen/Qwen2.5-VL-7B-Instruct`
+- Adapter repo: `cong-lab/labos-vlm-7b`
+- Adapter type: LoRA via PEFT / MS-SWIFT
+- Rank / alpha / dropout: `32 / 64 / 0.05`
+- Target modules: Qwen language-model projection layers matching `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, and `down_proj`
+- Training data: `lab_supervision_25k_train.jsonl` / `lab_supervision_25k_val.jsonl`
+
+## Install
+
+Install a CUDA-enabled PyTorch stack plus the model runtime dependencies:
+
+```bash
+python -m pip install -r requirements.txt
+```
+
+For Qwen2.5-VL video inference, `flash-attn` is recommended.
+
+## Run With MS-SWIFT
+
+```bash
+swift infer \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --adapters cong-lab/labos-vlm-7b \
+  --infer_backend pt
+```
+
+For a locally downloaded or modified adapter, replace the adapter ID with a local folder path:
+
+```bash
+swift infer \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --adapters ./labos-vlm-7b \
+  --infer_backend pt
+```
+
+## Run With Transformers And PEFT
+
+```python
+from peft import PeftModel
+from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
+
+base_model = "Qwen/Qwen2.5-VL-7B-Instruct"
+adapter = "cong-lab/labos-vlm-7b"  # or a path to a local adapter folder
+
+processor = AutoProcessor.from_pretrained(base_model)
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    base_model,
+    torch_dtype="auto",
+    device_map="auto",
+)
+model = PeftModel.from_pretrained(model, adapter)
+```
+
+## Evaluate On LSV
+
+The public LSV benchmark (`cong-lab/lsv`) includes video manifests, prompt loaders, and report generation for step prediction, monitoring-state advancement, and error detection.
+
+```bash
+hf download cong-lab/lsv --repo-type dataset --local-dir ./lsv
+cd lsv
+python -m pip install -r requirements.txt
+python inference.py \
+  --model Qwen/Qwen2.5-VL-7B-Instruct \
+  --adapter cong-lab/labos-vlm-7b \
+  --output runs/labos_vlm_7b \
+  --gpus 0
+python generate_report.py \
+  --output runs/labos_vlm_7b \
+  --report-dir runs/labos_vlm_7b_report
+```
+
+## Example Monitoring Prompt
+
+This adapter was fine-tuned with prompts similar to:
+
+```text
+You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.
+
+The current protocol state/history is provided below. Watch the current window and update the state.
+
+Report protocol errors only when supported by the visible time window or state.
+
+Compare the protocol order, prior history, and watched window.
+
+Identify the main protocol step being performed in this watched video window.
+
+STATE:
+{"history":[{"step":"2","tas":0,"tds":0},{"step":"3","tas":30,"tds":30}],"on":"3","protocol":[{"desc":"Take HEK293T cells and culture them to ~70% confluency in a 10 cm dish.","order":1,"step":"1"},{"desc":"In a sterile 1.5 mL tube, mix lentiviral backbone, packaging plasmid, and envelope plasmid.","order":2,"step":"2"},{"desc":"Add transfection reagent and bring to volume with serum-free medium.","order":3,"step":"3"},{"desc":"Incubate the mixture at room temperature for 15 minutes.","order":4,"step":"4"}]}
+
+Return the visible protocol step ID for the watched video window.
+```
+
+Run `python generate_monitoring_prompts.py` from this repository to print additional self-contained monitoring examples.
+
+## Training Parameters
+
+- Trainer: MS-SWIFT SFT with PEFT LoRA
+- Epochs: `2.0` (best chkpt @1ep)
+- Devices: `8` H100 GPUs
+- Per-device train batch size: `1`
+- Gradient accumulation steps: `4`
+- Effective global train batch size: `32` examples per optimizer step
+- Per-device eval batch size: `1`
+- Learning rate: `1e-4`
+- Scheduler: cosine
+- Warmup ratio: `0.03`
+- Optimizer: `adamw_torch_fused`
+- Weight decay: `0.1`
+- Adam betas: `(0.9, 0.95)`
+- Max gradient norm: `1.0`
+- Precision: `bfloat16`
+- Max sequence length: `4096`
+- Gradient checkpointing: enabled
+- DeepSpeed: ZeRO-2
+- Vision tower and aligner: frozen
+- Evaluation/checkpoint interval: every `250` steps
+
+## Limitations
+
+This adapter is intended for research on wet-lab video supervision and should not be used as the sole source of truth for laboratory safety or procedural correctness. Human review remains required for wet-lab execution.
+
+## License
+
+This adapter is released for non-commercial research use under the Creative Commons Attribution-NonCommercial 4.0 license (`CC-BY-NC-4.0`), matching the public LSV benchmark license.
+
+## Research Use Only
+
+This model is provided for research purposes only and for non-commercial use. It is not intended for clinical decision-making or replacing trained human supervision in real wet-lab procedures.

adapter_config.json

@@ -0,0 +1,40 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-VL-7B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [],
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": "^(model\\.language_model(?=\\.).*\\.(up_proj|k_proj|o_proj|gate_proj|down_proj|q_proj|v_proj))$",
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:992d28b53a5dbacee5f60be8ef1bfb28b16439750e548e0084cfe6e64b5f6b12
+size 161539464

generate_monitoring_prompts.py

@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""Print example LabOS-VLM monitoring prompts.
+
+These examples are self-contained and mirror the JSON-style monitoring prompts
+used when fine-tuning the LabOS-VLM-7B LoRA adapter.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+
+def compact_json(value: Any) -> str:
+    return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+
+
+def response_format_block(schema: dict[str, Any]) -> str:
+    return "\n".join([
+        "Return strict JSON only.",
+        f"## Response Format\n{json.dumps(schema, ensure_ascii=False, indent=2, sort_keys=True)}",
+    ])
+
+
+def example_state() -> dict[str, Any]:
+    return {
+        "protocol_summary": "Demonstration of aseptic pipetting technique.",
+        "reagents": ["sample buffer", "cell culture media"],
+        "equipment": ["P20 pipette", "sterile tips", "microcentrifuge tube rack"],
+        "protocol": [
+            {"step": "1", "order": 1, "desc": "Place the tubes in a clean rack."},
+            {"step": "2", "order": 2, "desc": "Attach a new sterile pipette tip."},
+            {"step": "3", "order": 3, "desc": "Aspirate reagent without touching the tube wall."},
+            {"step": "4", "order": 4, "desc": "Dispense reagent into the target tube."},
+        ],
+        "history": [{"step": "1", "tds": 20, "tas": 20}],
+        "on": "2",
+        "objects": {},
+    }
+
+
+def render_step_identification_prompt(state: dict[str, Any]) -> str:
+    schema = {
+        "observed_step_id": "string or null",
+        "explanation": "string",
+    }
+    return "\n\n".join([
+        "You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.",
+        "The current protocol state/history is provided below. Watch the current window and update the state.",
+        "Report protocol errors only when supported by the visible time window or state.",
+        "Ignore irrelevant unknown keys in the state JSON.",
+        "Compare the protocol order, prior history, and watched window.",
+        "Identify the main protocol step being performed in this watched video window.",
+        f"STATE:\n{compact_json(state)}",
+        response_format_block(schema),
+        "## Question\nWhich protocol step is being performed in this video window?",
+    ])
+
+
+def render_monitoring_delta_prompt(state: dict[str, Any]) -> str:
+    schema = {
+        "history": [
+            {
+                "step": "string",
+                "tds": "integer seconds spent on this step in the watched window",
+                "tas": "integer absolute seconds from the start of the video",
+            }
+        ],
+        "on": "string or null",
+        "errors": [
+            {
+                "type": "skipped_step | wrong_order | wrong_reagent | contamination | other",
+                "step": "string or null",
+                "description": "string",
+            }
+        ],
+    }
+    return "\n\n".join([
+        "You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.",
+        "The current protocol state/history is provided below. Watch the current window and update the state.",
+        "Report protocol errors only when supported by the visible time window or state.",
+        "Ignore irrelevant unknown keys in the state JSON.",
+        "Append only the newly observed history entries and update the current step.",
+        f"STATE:\n{compact_json(state)}",
+        response_format_block(schema),
+        "Return an append-style delta update for only the watched video window.",
+    ])
+
+
+def main() -> None:
+    state = example_state()
+    prompts = [
+        ("STEP IDENTIFICATION", render_step_identification_prompt(state)),
+        ("MONITORING DELTA", render_monitoring_delta_prompt(state)),
+    ]
+    for title, prompt in prompts:
+        print(f"===== {title} =====")
+        print(prompt)
+        print()
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,6 @@
+torch
+transformers
+peft
+qwen-vl-utils
+decord
+flash-attn