This is the latest model trained with Qwen3-VL-8B, with the same technique and data used in QoQ-Med. The original description is attached below.

This repository contains the model weights for QoQ-Med3-VL-8B (Qwen Omni-Reasoning on Medical Questions), a multimodal clinical foundation model with reasoning capabilities.

📚 Paper | 💻 Code

Clinical decision-making routinely demands reasoning over heterogeneous data, yet existing multimodal language models (MLLMs) remain largely vision-centric and fail to generalize across clinical specialties. To bridge this gap, we introduce QoQ-Med-7B/32B, the first open generalist clinical foundation model that jointly reasons across medical images, time-series signals, and text reports. QoQ-Med is trained with Domain-aware Relative Policy Optimization (DRPO), a novel reinforcement-learning objective that hierarchically scales normalized rewards according to domain rarity and modality difficulty, mitigating performance imbalance caused by skewed clinical data distributions. Trained on 2.61 million instruction tuning pairs spanning 9 clinical domains, we show that DRPO training boosts diagnostic performance by 43% in macro-F1 on average across all visual domains as compared to other critic-free training methods like GRPO. Furthermore, with QoQ-Med trained on intensive segmentation data, it is able to highlight salient regions related to the diagnosis, with an IoU 10x higher than open models while reaching the performance of OpenAI o4-mini. To foster reproducibility and downstream research, we release (i) the full model weights, (ii) the modular training pipeline, and (iii) all intermediate reasoning traces at this https URL .

QoQ-Med is the first open generalist clinical foundation model that jointly reasons across:

The model is trained with our novel Domain-aware Relative Policy Optimization (DRPO), a reinforcement learning objective that hierarchically scales normalized rewards according to domain rarity and modality difficulty, addressing performance imbalance in heterogeneous clinical data.

QoQ-Med spans multiple clinical specialties:

Prefer a point-and-click experience? Community-maintained GGUF builds are already on the Hub. They load instantly in desktop chat front-ends such as LM Studio, Ollama, and other llama.cpp-compatible apps—just search for “QoQ-Med-VL-7B/32B,” click Download, and start chatting. No Python environment, GPU, or command-line setup required.

First, ensure you have the necessary dependencies:

You may load the QoQ-Med model and processors via transformers package:

For better performance with flash attention:

You can adjust the visual token range to balance performance and computational cost:

Create a message with both image and text content:

Prepare the input for model inference:

Run inference and decode the output:

If you find the project useful, please cite the following papers:

@article{dai2025climb,
  title={Climb: Data foundations for large scale multimodal clinical foundation models},
  author={Dai, Wei and Chen, Peilin and Lu, Malinda and Li, Daniel and Wei, Haowen and Cui, Hejie and Liang, Paul Pu},
  journal={International Conference on Machine Learning},
  year={2025}
}
@article{dai2025qoq,
  title={QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training},
  author={Dai, Wei and Chen, Peilin and Ekbote, Chanakya and Liang, Paul Pu},
  journal={arXiv preprint arXiv:2506.00711},
  year={2025}
}

This model is intended for research purposes only. Before extensive real-world testing (like human trials), it is not suitable for clinical deployment. This is a research preview, not a product approved by federal agencies.