LabOS-VLM-32B is a PEFT LoRA adapter for Qwen/Qwen2.5-VL-32B-Instruct, fine-tuned on wet-lab supervision tasks from the LabOS JoVE and FineBio datasets and validated on the LSV benchmark. These tasks include protocol monitoring/step prediction, error detection, spatial grounding, protocol generation, and general VQA with first-person and third-person views.

The adapter is intended for research on laboratory video-language assistants. It works best with the JSON-style monitoring and benchmark prompts used by the LabOS datasets, such as the public LSV benchmark at cong-lab/lsv. Note, the expected performance may drop when evaluating on different modalities, or prompts, as is typically expected with SFT.

Adapter Details

Base model: Qwen/Qwen2.5-VL-32B-Instruct
Adapter repo: cong-lab/labos-vlm-32b
Adapter type: LoRA via PEFT / MS-SWIFT
LoRA rank / alpha / dropout: 32 / 64 / 0.05
LoRA dtype: bfloat16
Target modules: Qwen language-model projection layers matching q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
Vision tower and aligner: frozen
Training data: FineBio + JoVE

Install

Install a CUDA-enabled PyTorch stack plus the model runtime dependencies:

python -m pip install -r requirements.txt

For Qwen2.5-VL video inference, flash-attn is recommended.

Run With MS-SWIFT

bash

swift infer \
  --model Qwen/Qwen2.5-VL-32B-Instruct \
  --adapters cong-lab/labos-vlm-32b \
  --infer_backend pt

For a locally downloaded or modified adapter, replace the adapter ID with a local folder path:

bash

swift infer \
  --model Qwen/Qwen2.5-VL-32B-Instruct \
  --adapters ./labos-vlm-32b \
  --infer_backend pt

Run With Transformers And PEFT

python

from peft import PeftModel
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

base_model = "Qwen/Qwen2.5-VL-32B-Instruct"
adapter = "cong-lab/labos-vlm-32b"  # or a path to a local adapter folder

processor = AutoProcessor.from_pretrained(base_model)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Evaluate On LSV

The public LSV benchmark (cong-lab/lsv) includes video manifests, prompt loaders, and report generation for step prediction, monitoring-state advancement, and error detection.

bash

hf download cong-lab/lsv --repo-type dataset --local-dir ./lsv
cd lsv
python -m pip install -r requirements.txt
python inference.py \
  --model Qwen/Qwen2.5-VL-32B-Instruct \
  --adapter cong-lab/labos-vlm-32b \
  --output runs/labos_vlm_32b \
  --gpus 0
python generate_report.py \
  --output runs/labos_vlm_32b \
  --report-dir runs/labos_vlm_32b_report

Example Monitoring Prompt

text

You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.

The current protocol state/history is provided below. Watch the current window and update the state.

Report protocol errors only when supported by the visible time window or state.

Compare the protocol order, prior history, and watched window.

Identify the main protocol step being performed in this watched video window.

STATE:
{"history":[{"step":"2","tas":0,"tds":0},{"step":"3","tas":30,"tds":30}],"on":"3","protocol":[{"desc":"Take HEK293T cells and culture them to ~70% confluency in a 10 cm dish.","order":1,"step":"1"},{"desc":"In a sterile 1.5 mL tube, mix lentiviral backbone, packaging plasmid, and envelope plasmid.","order":2,"step":"2"},{"desc":"Add transfection reagent and bring to volume with serum-free medium.","order":3,"step":"3"},{"desc":"Incubate the mixture at room temperature for 15 minutes.","order":4,"step":"4"}]}

Return the visible protocol step ID for the watched video window.

Training Parameters

Trainer: MS-SWIFT SFT
Epochs: 2.0 (best chkpt @1ep)
Devices: 8 H100 GPUs
Per-device train batch size: 1
Gradient accumulation steps: 4
Effective global batch size: 32
Per-device eval batch size: 1
Learning rate: 5e-5
Scheduler: cosine
Warmup ratio: 0.03
Weight decay: 0.1
Adam betas: (0.9, 0.95)
Max gradient norm: 1.0
Precision: bfloat16
Max sequence length: 4096
Vision tower and aligner: frozen
Evaluation/checkpoint interval: 250 steps
Dataloader workers: 8

Limitations

This adapter is intended for research on wet-lab video supervision and should not be used as the sole source of truth for laboratory safety or procedural correctness. Human review remains required for wet-lab execution.

License

This adapter is released for non-commercial research use under the Creative Commons Attribution-NonCommercial 4.0 license (CC-BY-NC-4.0), matching the public LSV benchmark license.

Research Use Only

This model is provided for research purposes only and for non-commercial use. It is not intended for clinical decision-making or replacing trained human supervision in real wet-lab procedures.