This repo contains trained projector weights + LoRA adapters from a controlled experiment comparing three vision encoders stitched into Qwen2.5-0.5B-Instruct. Trained on COCO-QA (46k Q&A pairs) for 2,000 steps.

See also: 1.5B version (VQAv2 training)

Links
Training Code	github.com/REDDITARUN/CLIP-ViT-IJEPA-VLM
1.5B Version	Teen-Different/CLIP-ViT-IJEPA-VLMs-1.5B
Blog Post	teendifferent.substack.com

The Experiment

[Frozen Vision Encoder] ──► [Trainable Projector] ──► [Qwen-0.5B + LoRA]
         │                          │                         │
    CLIP / ViT / I-JEPA      LayerNorm+MLP           rank=16, q_proj+v_proj

Vision Encoders Compared

Encoder	Model	Hidden Dim	Patches	Params	Pre-training
ViT-L/16	`google/vit-large-patch16-224`	1024	196	304M	Supervised (ImageNet-21k)
CLIP ViT-L/14	`openai/clip-vit-large-patch14`	1024	256	304M	Contrastive (400M image-text pairs)
I-JEPA ViT-H/14	`facebook/ijepa_vith14_1k`	1280	256	632M	Self-supervised (predict masked patches)

Training Setup

Component	Details
LLM	Qwen/Qwen2.5-0.5B-Instruct (frozen + LoRA rank=16)
Projector	LayerNorm → Linear → GELU → Linear
Dataset	HuggingFaceM4/the_cauldron (cocoqa subset, 46k Q&A pairs)
Optimizer	AdamW, lr=1e-4, linear warmup 100 steps
Steps	2,000 optimizer steps, effective batch size 16
Hardware	Single A100 GPU (~18 min per encoder)

Results

Training Loss

Encoder	Final Loss	Min Loss	Trainable Params	Total Params
CLIP	0.6815	0.1747	2,805,504 (0.35%)	800,018,048
ViT	0.5221	0.2141	2,805,504 (0.35%)	801,189,504
I-JEPA	0.4947	0.2282	3,035,392 (0.27%)	1,127,830,400

Evaluation Accuracy (exact match, 1K samples per benchmark)

Encoder	CLEVR	COCO-QA	TallyQA	Average
CLIP	5.0%	55.8%	0.2%	20.3%
I-JEPA	10.4%	51.2%	0.0%	20.5%
ViT	10.6%	51.0%	0.0%	20.5%

Sample Inference (Golden Retriever Image)

Question	CLIP	ViT	I-JEPA
"Describe the image?"	Dog	Dog	Dog
"Is the dog's tongue out?"	Yes	Yes	Yes
"What color is the fur?"	Dog	Yellow	Brown
"What color is the dog?"	Fur	Brown	Brown
"What animal is this?"	Dog	Dog	Dog

Key Findings (0.5B)

CLIP excels on COCO-QA (55.8%) — its text-aligned embeddings match the Q&A format well
I-JEPA and ViT lead on CLEVR (~10%) — better at compositional/spatial reasoning
All three struggle on TallyQA (counting) with only 0.5B LLM and short training
Answers are short (1 word) because COCO-QA uses short Q&A format

Repo Structure

├── clip/
│   ├── projector.pt           # Trained MLP bridge weights
│   ├── lora_adapter/          # LoRA adapter (loadable via peft)
│   ├── loss_history.json      # Per-step training loss
│   └── config.json            # Full experiment config
├── vit/
│   └── ... (same structure)
└── ijepa/
    └── ... (same structure)

Quick Start

Install

pip install torch transformers peft Pillow huggingface-hub

Inference (standalone — no repo clone needed)

python

import json, torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
from huggingface_hub import hf_hub_download, snapshot_download

REPO = "Teen-Different/CLIP-ViT-IJEPA-VLMs-0.5B"
ENCODER = "clip"  # or "vit" or "ijepa"

# Download weights
cfg = json.load(open(hf_hub_download(REPO, f"{ENCODER}/config.json")))
proj_path = hf_hub_download(REPO, f"{ENCODER}/projector.pt")
lora_dir = snapshot_download(REPO, allow_patterns=f"{ENCODER}/lora_adapter/*")
lora_path = f"{lora_dir}/{ENCODER}/lora_adapter"

# Load encoder (example for CLIP)
from transformers import CLIPVisionModel, CLIPImageProcessor
encoder = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14",
                                           torch_dtype=torch.bfloat16).cuda().eval()
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-large-patch14")

# Load projector
import torch.nn as nn
class Projector(nn.Module):
    def __init__(self, vd, ld):
        super().__init__()
        self.norm = nn.LayerNorm(vd)
        self.fc1 = nn.Linear(vd, ld)
        self.act = nn.GELU()
        self.fc2 = nn.Linear(ld, ld)
    def forward(self, x):
        return self.fc2(self.act(self.fc1(self.norm(x))))

llm_dim = AutoConfig.from_pretrained(cfg["llm_model_id"]).hidden_size
projector = Projector(1024, llm_dim).to(torch.bfloat16).cuda()
projector.load_state_dict(torch.load(proj_path, map_location="cuda", weights_only=True))
projector.eval()

# Load LLM + LoRA
tokenizer = AutoTokenizer.from_pretrained(cfg["llm_model_id"])
llm = AutoModelForCausalLM.from_pretrained(cfg["llm_model_id"],
                                            torch_dtype=torch.bfloat16).cuda()
llm = PeftModel.from_pretrained(llm, lora_path).cuda().eval()

# Run inference
image = Image.open("photo.jpg").convert("RGB")
pixels = processor(images=[image], return_tensors="pt")["pixel_values"].cuda()

with torch.no_grad():
    vis = encoder(pixel_values=pixels).last_hidden_state[:, 1:, :]  # drop CLS
    vis = projector(vis)
    tok = tokenizer("<image>\nWhat is in this image?", return_tensors="pt").to("cuda")
    txt = llm.model.model.embed_tokens(tok["input_ids"])
    embeds = torch.cat([vis, txt], dim=1)
    mask = torch.ones(embeds.shape[:2], dtype=torch.long, device="cuda")
    out = llm.generate(inputs_embeds=embeds, attention_mask=mask,
                       max_new_tokens=50, min_new_tokens=3, do_sample=False)
    print(tokenizer.decode(out[0], skip_special_tokens=True))

Using the CLI (from source repo)

bash

git clone https://github.com/REDDITARUN/CLIP-ViT-IJEPA-VLM.git && cd CLIP-ViT-IJEPA-VLM
pip install -r requirements.txt

# Inference
python inference.py --encoder clip --image photo.jpg --from-hub Teen-Different/CLIP-ViT-IJEPA-VLMs-0.5B

# Evaluate
python eval.py --from-hub Teen-Different/CLIP-ViT-IJEPA-VLMs-0.5B

# Interactive
python inference.py --encoder clip --image photo.jpg --from-hub Teen-Different/CLIP-ViT-IJEPA-VLMs-0.5B --interactive

What Makes Each Encoder Different?

Encoder	What the embeddings encode	Trained on	Best for
CLIP	Language-aligned concepts ("dog", "red", "beach")	400M image-text pairs	Fast convergence, concept questions
I-JEPA	Spatial structure, part-whole relations, layout	ImageNet (self-supervised)	Spatial reasoning, compositional tasks
ViT	ImageNet class-discriminative features	ImageNet-21k (supervised)	Classification-style tasks

Limitations

0.5B LLM — small language model limits answer quality and length
Short answers — trained on COCO-QA which has 1-3 word answers
2,000 steps — relatively short training; longer training would improve all encoders
Single dataset — trained only on COCO-QA; multi-dataset training would improve generalization

Citation

bibtex

@misc{clip-vit-ijepa-vlm-2026,
  title={CLIP vs ViT vs I-JEPA: Vision Encoder Stitching Comparison},
  author={Tarun Reddi},
  year={2026},
  url={https://github.com/REDDITARUN/CLIP-ViT-IJEPA-VLM}
}