You can deploy this model and start earning money today!

sakp7/
Qwen2.5-VL-7B-Instruct-INT8

Name: Qwen2.5-VL-7B-Instruct-INT8
Availability: InStock

Views

No views yet

Qwen2.5-VL-7B-Instruct-INT8

This is an 8-bit quantized version of Qwen2.5-VL-7B-Instruct for more efficient deployment.

Model Details

Base Model: Qwen/Qwen2.5-VL-7B-Instruct
Quantization: 8-bit quantization using bitsandbytes
vLLM Compatible: Yes
Use Case: Vision-language tasks with reduced memory requirements

Usage with vLLM

python

from vllm import LLM, SamplingParams
from PIL import Image

# Initialize the model
llm = LLM(model="sakp7/Qwen2.5-VL-7B-Instruct-INT8", quantization="8bit")

# Prepare image and prompt
image = Image.open("example.jpg")
prompt = "Describe this image in detail."

# Generate response
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate_for_multimodal_prompts([[prompt, image]], sampling_params)
print(outputs[0].outputs[0].text)