The original Google release separates the vision-language backbone and DPT dense prediction heads into two repositories. This model combines both components into a unified checkpoint for easier deployment and archival.

Base Models

Backbone: google/tipsv2-so400m14
DPT Heads: google/tipsv2-so400m14-dpt

Original work by Google Research.

Model Overview

TIPSv2 (Text-Image Pre-training with Spatial Awareness) is a family of contrastive vision-language models that produce spatially rich image representations aligned with text embeddings.

The SO400m/14 variant contains:

Component	Parameters
Vision Encoder	412M
Text Encoder	448M
Embedding Dimension	1152

The DPT extension adds dense prediction heads for:

Depth Estimation
Surface Normal Prediction
Semantic Segmentation (ADE20K, 150 classes)

Architecture

Backbone:

ViT Vision Encoder
27 Vision Transformer Layers
14×14 Patch Size
1152-dimensional embeddings

Dense Prediction Heads:

DPT Depth Head
DPT Surface Normal Head
DPT Semantic Segmentation Head

Training datasets used by the original Google models:

NYU Depth V2 (depth estimation)
NYU Depth V2 (surface normals)
ADE20K (semantic segmentation)

Input Requirements

Images should:

Be converted to tensors in the [0, 1] range
Not use ImageNet normalization
Prefer resolutions divisible by 14

Example preprocessing:

python

from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((448, 448)),
    transforms.ToTensor(),
])

Usage

python

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Rikunarita-ORG/tipsv2-so400m14-dpt-merged",
    trust_remote_code=True
)

Depth Estimation

depth = model.predict_depth(pixel_values)

Surface Normals

normals = model.predict_normals(pixel_values)

Semantic Segmentation

segmentation = model.predict_segmentation(pixel_values)

Run All Heads

python

outputs = model(pixel_values)

depth = outputs.depth
normals = outputs.normals
segmentation = outputs.segmentation

Merged Checkpoint Notes

This repository is not an official Google release.

The checkpoint was created by merging:

TIPSv2 SO400m/14 backbone weights
TIPSv2 SO400m/14 DPT head weights

into a single model.safetensors file for convenience.

No additional training, finetuning, quantization, pruning, or weight modification was performed.

License

This repository inherits the Apache 2.0 license from the original Google TIPSv2 releases.

Citation

If you use this model, please cite the original TIPSv2 paper:

bibtex

@inproceedings{cao2026tipsv2,
  title     = {TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment},
  author    = {Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Rene and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {CVPR},
  year      = {2026}
}