KAIROS-MM-Qwen2.5-VL-7B-SFT

KAIROS-MM-Qwen2.5-VL-7B-SFT is a multimodal reasoning model based on Qwen2.5-VL, designed to enable robots and vision AI agents to reason about the real world using physics understanding, prior knowledge, and common sense. The model understands space, time, and fundamental physical principles, and is well suited for planning, decision making, and long horizon video reasoning in embodied and agentic systems. It serves as a planning and reasoning backbone for embodied agents, allowing them to infer what actions to take next based on visual observations, temporal context, and physical constraints.

Key Capabilities

Multimodal Reasoning Across Space and Time Understands spatial relationships, temporal dynamics, and causal interactions from images and long horizon videos.
Physics and Common Sense Understanding Reasons about motion, forces, object permanence, collisions, and everyday physical interactions in real world environments.
Long Horizon Video Reasoning Maintains contextual understanding across extended video sequences for activity understanding, forecasting, and planning.
Embodied Agent Planning Acts as a high level planning model to reason about the next steps an embodied agent or robot should take in a given environment.
Vision Language Grounding Aligns visual perception with language instructions and goals for reliable reasoning and action guidance.
Task Level Decision Making Supports complex multi step reasoning for navigation, manipulation, and interaction tasks in dynamic environments.
Cross Domain Generalization Adapts to robotics, physical AI, simulation environments, and real world visual reasoning scenarios.

Quick Start with Transformers

python

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "prithivMLmods/KAIROS-MM-Qwen2.5-VL-7B-SFT",
    torch_dtype="auto",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "prithivMLmods/KAIROS-MM-Qwen2.5-VL-7B-SFT"
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "<LONG_HORIZON_VIDEO>"},
            {"type": "text", "text": "What should the robot do next to safely pick up the object?"},
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output_text)

Training Datasets

The model is fine tuned using large scale multimodal reasoning and planning datasets focused on physical understanding and long horizon reasoning:

SAGE-MM-SFT-417K A large scale multimodal supervised fine tuning dataset focused on grounded reasoning, perception, and decision making. https://huggingface.co/datasets/allenai/SAGE-MM-SFT-417K
Cosmos-Reason1-SFT-Dataset A dataset designed for physical AI and robotics reasoning, emphasizing common sense, physics understanding, and temporal reasoning. https://huggingface.co/datasets/nvidia/Cosmos-Reason1-SFT-Dataset

Intended Use

Embodied AI and robotics reasoning
Long horizon video understanding and planning
Physical AI and simulation environments
Vision based decision making and forecasting
Robot task planning and action sequencing
Multimodal reasoning research and development

Limitations

Performance may degrade on extremely long or noisy video inputs
Highly ambiguous physical scenarios may require external planners or simulators
Domain specific robotics tasks may benefit from additional fine tuning
Fine grained low level control is outside the scope of this model

References

Qwen2.5 VL Technical Report https://huggingface.co/papers/2502.13923
YaRN: Efficient Context Window Extension https://arxiv.org/pdf/2309.00071
Qwen2 VL: High Resolution Perception https://arxiv.org/pdf/2409.12191
Model Soups: Averaging Weights of Multiple Fine Tuned Models Improves Accuracy Without Increasing Inference Time https://arxiv.org/abs/2203.05482
Model Stock: All We Need Is Just a Few Fine Tuned Models https://arxiv.org/abs/2403.19522