SciCLIP is a Parameter-Efficient Fine-Tuned (PEFT) version of OpenAI's CLIP (ViT-Base-Patch32) optimized specifically for the domain of scientific figure-caption retrieval. By freezing the base CLIP backbone and adapting only target projection layers using LoRA (Low-Rank Adaptation), the model learns complex domain-specific alignments between highly technical captions and dense academic plots, charts, and diagrams.

Model Details

Base Backbone: openai/clip-vit-base-patch32 (151M parameters)
Adaptation Method: LoRA (Low-Rank Adaptation) via PEFT
LoRA Configurations:
- Rank ($r$): 8
- Alpha ($\alpha$): 16
- Target Modules: q_proj, k_proj, v_proj (applied to both visual and textual encoders)
- Trainable Parameters: Only ~0.1% of the original model size

Dataset

Source: CrowdAILab/scicap (captions extracted from Computer Science arXiv papers)
Preprocessing:
- Standardized captions by removing the redundant "Figure X:" prefix (using the caption_no_index field) to filter noise.
- Trimmed length limit to 20-300 characters to filter uninformative figures.
- Subsampled 20,000 clean pairs for training.

Evaluation Results

Model	R@1 (%)	R@5 (%)	R@10 (%)	MRR	Median Rank
Vanilla CLIP (Zero-Shot)	10.90	19.35	23.80	0.1564	110.0
SciCLIP (LoRA r=16)	16.40	29.30	35.85	0.2312	34.0
Relative Improvement	+50.5%	+51.4%	+50.6%	+47.8%	76 ranks saved

How to Use

You can easily load and run SciCLIP using the Hugging Face transformers and peft libraries:

python

import torch
from transformers import CLIPModel, CLIPProcessor
from peft import PeftModel

# 1. Load vanilla CLIP backbone
base_model = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(base_model)
processor = CLIPProcessor.from_pretrained(base_model)

# 2. Attach SciCLIP LoRA Adapter
adapter_id = "jkhyjkhy/sciclip-lora-r8" # Replace with your repo ID
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

print("✓ SciCLIP loaded successfully.")

@article{sciclip2026,
  title={SciCLIP: Parameter-Efficient Fine-Tuning of CLIP for Academic Figure Retrieval},
  author={Jeong, Younghee},
  journal={arXiv preprint},
  year={2026}
}

Citation

If you use this model or code in your research, please cite:

bibtex

@article{sciclip2026,
  title={SciCLIP: Parameter-Efficient Fine-Tuning of CLIP for Academic Figure Retrieval},
  author={Jeong, Younghee},
  journal={arXiv preprint},
  year={2026}
}

jkhyjkhy/sciclip-lora-r16

🔬 SciCLIP: LoRA-Adapted CLIP for Scientific Figure Retrieval

Model Details

Dataset

Evaluation Results

How to Use

Citation

jkhyjkhy/
sciclip-lora-r16