SciCLIP is a Parameter-Efficient Fine-Tuned (PEFT) version of OpenAI's CLIP (ViT-Base-Patch32) optimized specifically for the domain of scientific figure-caption retrieval. By freezing the base CLIP backbone and adapting only target projection layers using LoRA (Low-Rank Adaptation), the model learns complex domain-specific alignments between highly technical captions and dense academic plots, charts, and diagrams.

Model Details

Base Backbone: openai/clip-vit-base-patch32 (151M parameters)
Adaptation Method: LoRA (Low-Rank Adaptation) via PEFT
LoRA Configurations:
- Rank ($r$): 8
- Alpha ($\alpha$): 16
- Target Modules: q_proj, k_proj, v_proj (applied to both visual and textual encoders)
- Trainable Parameters: Only ~0.1% of the original model size

Dataset

Source: CrowdAILab/scicap (captions extracted from Computer Science arXiv papers)
Preprocessing:
- Standardized captions by removing the redundant "Figure X:" prefix (using the caption_no_index field) to filter noise.
- Trimmed length limit to 20-300 characters to filter uninformative figures.
- Subsampled 20,000 clean pairs for training.

Evaluation Results

Evaluation performed on the test/validation splits of the SciCap dataset (cross-modal Text-to-Image retrieval over 2,000 candidate figures).

Every metrics represents a massive performance boost over the zero-shot baseline:

Model	R@1 (%)	R@5 (%)	R@10 (%)	MRR	Median Rank
Vanilla CLIP (Zero-Shot)	10.90	19.35	23.80	0.1564	110.0
SciCLIP (LoRA r=8)	16.55	29.60	35.45	0.2324	34.0
Relative Improvement	+51.8%	+52.9%	+48.9%	+48.6%	76 ranks saved

How to Use

You can easily load and run SciCLIP using the Hugging Face transformers and peft libraries:

python

import torch
from transformers import CLIPModel, CLIPProcessor
from peft import PeftModel

# 1. Load vanilla CLIP backbone
base_model = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(base_model)
processor = CLIPProcessor.from_pretrained(base_model)

# 2. Attach SciCLIP LoRA Adapter
adapter_id = "jkhyjkhy/sciclip-lora-r8" # Replace with your repo ID
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

print("✓ SciCLIP loaded successfully.")

@article{sciclip2026,
  title={SciCLIP: Parameter-Efficient Fine-Tuning of CLIP for Academic Figure Retrieval},
  author={Jeong, Younghee},
  journal={arXiv preprint},
  year={2026}
}

Citation

If you use this model or code in your research, please cite:

bibtex

@article{sciclip2026,
  title={SciCLIP: Parameter-Efficient Fine-Tuning of CLIP for Academic Figure Retrieval},
  author={Jeong, Younghee},
  journal={arXiv preprint},
  year={2026}
}