You can deploy this model and start earning money today!

yfwang22/
RESA-mix-7B

Name: RESA-mix-7B
Availability: InStock

safetensors

llava_llama

multimodal large language model

safety

apache-2.0

Views

No views yet

RESA-mix-7B

RESA-mix-7B is a multimodal large language model (MLLM) based on the RESA-7B architecture, with additional training data from LLaVA-Next to enhance multimodal reasoning. This model was developed as part of the paper:
📄 Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation.

🧠 Model Overview

Base architecture: LLaVA
Visual encoder: openai/clip-vit-large-patch14-336
- To replicate the RESA-R-mix-7B variant, replace the visual encoder with chs20/fare4-clip.
Language model: Vicuna-7B
Fine-tuning method: Supervised Fine-Tuning (SFT)
Enhancement: Chain-of-Thought (CoT) prefix generation over a combined dataset
New Data: Mixed with 10k additional samples from LLaVA-Next, with CoT generated by GPT-4o
Total SFT samples: VLGuard-CoT + LLaVA-Next-CoT (~12k)

🧪 Training Details

Datasets:
- VLGuard-CoT (CoT generated with GPT-4o)
- LLaVA-Next 10k (also CoT-augmented with GPT-4o)
Training Objective: Multimodal instruction tuning with CoT-augmented supervision

🔍 Intended Use

RESA-mix-7B is designed for:

Evaluating MLLM alignment and safety with diverse multimodal instruction datasets
Enhancing multimodal reasoning via GPT-4o-guided CoT
Studying the trade-off between general-purpose and safety-aligned data on model generalization and trustworthiness.

🚀 Example Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("yfwang22/RESA-mix-7B")
model = AutoModelForCausalLM.from_pretrained("yfwang22/RESA-mix-7B")

inputs = tokenizer("Describe the ethical risks shown in this image.", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))