You can deploy this model and start earning money today!

teohyc/
QwigLip-VLM

Name: QwigLip-VLM
Availability: InStock

Qwen/Qwen2-0.5B-Instruct

finetune

mit

endpoints_compatible

Views

No views yet

Qwiglip VLM (Qwen2 + SigLIP)

Custom Vision-Language Model built from scratch. Inspired by LLaVA VLM architecture, but with a custom MLP projector and LoRA fine-tuning for efficient training. Training data from https://huggingface.co/datasets/phiyodr/coco2017 Full repository at https://github.com/teohyc/qwiglip_vlm

Components

Base LLM: Qwen/Qwen2-0.5B-Instruct
Vision Encoder: SigLIP
LoRA fine-tuning
Custom MLP projector

Usage

***** CHECK OUT inference.py FOR DETAILED INFERENCE EXAMPLE *****

python

import torch
from PIL import Image
from transformers import AutoTokenizer, AutoProcessor, AutoModel, Qwen2ForCausalLM
from peft import PeftModel

from vlm_model import MLPProjector, SiglipQwenVLM

#configurations
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

LLM_NAME = "Qwen/Qwen2-0.5B-Instruct"
VISION_NAME = "google/siglip-base-patch16-224"

LORA_PATH = "lora_adapter"
PROJECTOR_PATH = "projector.pt"

NUM_IMAGE_TOKENS = 196

#refer to inference.py for full code

teohyc/QwigLip-VLM

Qwiglip VLM (Qwen2 + SigLIP)

Components

Usage

teohyc/
QwigLip-VLM