You can deploy this model and start earning money today!

alessiopierdominici/
osmclip_vit32base

Name: osmclip_vit32base
Availability: InStock

Views

No views yet

🌍 OSM-CLIP / OpenStreetCLIP (ViT-B/32)

Official weights for OSM-CLIP (OpenStreetCLIP), a vision–language framework that leverages OpenStreetMap (OSM) annotations for patch-level geographic supervision in remote sensing.

🚀 Overview

Standard CLIP models operate on global image–text alignment, collapsing an entire satellite image into a single feature vector. OSM-CLIP instead injects patch-level geographic semantics by aligning ViT patches with OSM-derived textual tags (e.g. building residential, highway primary), enabling spatially grounded representations that better capture fine-grained geospatial structure.

This repository provides all three checkpoints used in the ablation study of the associated paper.

📦 Available Checkpoints

File Name	Paper name	Pre-training	Description
`osm-clip-pretrained-finetuned.pt`	OSM-PT+FT	OSM patch-level contrastive pre-training + FT	Main OSM-CLIP model. Pre-trained with OSM patch-level supervision and then fine-tuned on RS captioning datasets.
`clip-rs-pretrained-finetuned.pt`	RS-PT+FT	Generic RS image pre-training + FT	Ablation baseline. CLIP pre-trained on the same 265k RS images using a generic caption (“a satellite image”), then fine-tuned. Used to isolate the effect of more RS data without OSM semantics.
`clip-standard-finetuned.pt`	FT	No pre-training (CLIP only fine-tuned)	Baseline. Original OpenAI CLIP ViT-B/32 weights fine-tuned on RS captioning datasets only (no RS or OSM pre-training).

All three models share the same architecture and fine-tuning setup; only the pre-training strategy changes.

🛠 Technical Specifications

Backbone: ViT-B/32 (CLIP image encoder)
Text encoder: CLIP text transformer
Input size: 224 × 224 (patch grid 7 × 7 for ViT-B/32)
Framework: PyTorch
Pre-training (OSM-PT):
- Patch-level contrastive loss between aggregated patch embeddings and OSM tag embeddings
- ~265k images over the contiguous US with OSM-derived annotations
Fine-tuning (FT):
- Standard CLIP contrastive loss on RS captioning datasets (NWPU-RESISC45, RSICD, UCM, SYDNEY)
- Optimizer: Adafactor
- Single run per configuration (no multi-run ensembling)

🧪 Recommended Usage

Use osm-clip-pretrained-finetuned.pt when you want spatially grounded, OSM-aware representations for:
- zero-shot or linear-probe RS scene classification,
- image–text retrieval,
- geospatial grounding / visualization of patch activations.
Use clip-standard-finetuned.pt as a strong CLIP-based baseline without any RS- or OSM-specific pre-training.
Use clip-rs-pretrained-finetuned.pt if you want to replicate the ablation from the paper and test the effect of “more RS data without OSM semantics”.

📜 Licenses & Attribution

Code & Weights: Licensed under GPL-3.0.
OSM Data: This model was trained using data from OpenStreetMap.
OpenStreetMap data is licensed under the Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF).
© OpenStreetMap contributors.

📖 Citation

If you use these models, please cite the associated paper (currently under review):

bibtex

@article{openstreetclip2026,
  title   = {OSM-CLIP: Enhancing Remote Sensing Image-Text Representation Learning with OpenStreetMap Data},
  author  = {Pierdominici, Alessio and Ricci, Riccardo and Alruqimi, Mohammed and Melgani, Farid},
  journal = {Applied Sciences},
  year    = {2026},
  note    = {submitted}
}

(Update with volume/issue/DOI once published.)

🔗 Links

Code (training & evaluation): https://github.com/alessiopierdominici/OpenStreetCLIP
Dataset (OSM-CLIP pre-training): https://huggingface.co/datasets/alessiopierdominici/osm-clip-dataset

Developed at the University of Trento.

alessiopierdominici/osmclip_vit32base