LabOS-VLM-7B is a PEFT LoRA adapter for Qwen/Qwen2.5-VL-7B-Instruct, fine-tuned on wet-lab supervision prompts for protocol monitoring, step prediction, pipette mistake detection, spatial grounding, protocol generation, and protocol VQA over egocentric and third-person lab videos; it was trained on the prompt families included in this repository, so it should work best with the same JSON-style monitoring and benchmark prompts and should be expected to generalize less reliably to prompts outside that distribution.

Adapter Details

Base model: Qwen/Qwen2.5-VL-7B-Instruct
Adapter type: LoRA via PEFT / MS-SWIFT
Rank: 32
LoRA alpha: 64
LoRA dropout: 0.05
Target modules: Qwen language-model projection layers matching q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
Training checkpoint: checkpoint-750
Training data: lab_supervision_25k_train.jsonl / lab_supervision_25k_val.jsonl

Training Parameters

Trainer: MS-SWIFT SFT with PEFT LoRA
Epochs: 2.0
Devices: 8 GPUs (global_world_size=8)
Per-device train batch size: 1
Gradient accumulation steps: 4
Effective global train batch size: 32 examples per optimizer step
Per-device eval batch size: 1
Learning rate: 1e-4
Scheduler: cosine
Warmup ratio: 0.03
Optimizer: adamw_torch_fused
Weight decay: 0.1
Adam betas: (0.9, 0.95)
Max gradient norm: 1.0
Precision: bfloat16
Max sequence length: 4096
Gradient checkpointing: enabled
DeepSpeed: ZeRO-2
Vision tower: frozen (freeze_vit=true)
Aligner: frozen (freeze_aligner=true)
Evaluation interval: every 250 steps
Checkpoint interval: every 250 steps

Run With MS-SWIFT

Install the Qwen video runtime dependencies and MS-SWIFT in your environment, then run:

bash

swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters labos1/labos-vlm-7b \
  --infer_backend pt

For local staged weights before upload:

bash

swift infer \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --adapters /data/david/models/release/labos-vlm-7b \
  --infer_backend pt

Run With Transformers And PEFT

python

from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

base_model = "Qwen/Qwen2.5-VL-7B-Instruct"
adapter = "labos1/labos-vlm-7b"

processor = AutoProcessor.from_pretrained(base_model)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    base_model,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

Example Monitoring Prompt

This adapter was fine-tuned with prompts similar to:

text

You are a real-time lab assistant monitoring a scientist's wet-lab procedure from short video windows.

The current protocol state/history is provided below. Watch the current window and update the state.

Report protocol errors only when supported by the visible time window or state.

Ignore irrelevant unknown keys in the state JSON.

Compare the protocol order, prior history, and watched window.

Identify the main protocol step being performed in this watched video window.

STATE:
{"equipment":["P20 pipette","sterile tips","microcentrifuge tube rack"],"history":[{"step":"1","tas":20,"tds":20}],"objects":{},"on":"2","protocol":[{"desc":"Place the tubes in a clean rack.","order":1,"step":"1"},{"desc":"Attach a new sterile pipette tip.","order":2,"step":"2"},{"desc":"Aspirate reagent without touching the tube wall.","order":3,"step":"3"}],"protocol_summary":"Demonstration of aseptic pipetting technique.","reagents":["sample buffer"]}

Return strict JSON only.

## Response Format
{
  "explanation": "string",
  "observed_step_id": "string or null"
}

## Question
Which protocol step is being performed in this video window?

Run python generate_monitoring_prompts.py from this folder to print example monitoring prompts that match the training/evaluation prompt style.