TIPSv2-L/14 — ONNX Export
Large TIPS variant (303M vision / 184M text, 24-layer ViT). 448×448 images → 1024-dim embeddings.
Original:
google-deepmind/tips.
Exported with
tips-onnx — see the repo
for custom exports (other precisions, fixed sizes) and TensorRT engine builds.
Available files
| File | Precision | Size | Backend |
|---|
vision_encoder_fp32.onnx | FP32 | 1.13 GB | CPU / CUDA / TRT |
text_encoder_fp32.onnx | FP32 | 702 MB | CPU / CUDA / TRT |
vision_encoder_fp16.onnx | FP16 | 580 MB | CPU / CUDA / TRT |
text_encoder_fp16.onnx | FP16 | 351 MB | CPU / CUDA / TRT |
vision_encoder_int8_dynamic.onnx | INT8 (CPU) | 291 MB | CPU |
text_encoder_int8_dynamic.onnx | INT8 (CPU) | 176 MB | CPU |
Fixed-size 448×448 exports for TensorRT:
| vision_encoder_448_fp32.onnx | FP32 | 1.13 GB | TRT |
| vision_encoder_448_fp16.onnx | FP16 | 581 MB | TRT |
The _448 exports skip the dynamic position-encoding interpolation path,
producing a fixed-shape graph that trtexec can parse.
Calibration
INT8 Q/DQ quantization (
tools/export.py --precision int8_qdq) needs
calibration data in the vision/text form of the encoder inputs, from any
source — at least 64 samples per encoder (the minimum suggested by NVIDIA
ModelOpt; this project's calibration used 500). The development data was
built from
lmms-lab/COCO-Caption
(500 images + captions).
Rebuild calibration data with tools/make_calibration.py (synthetic,
structural testing only) or from your own dataset in the same format:
checkpoints/calib_vision.npy ((N, 3, 448, 448) float32 images) and
checkpoints/calib_text.npz (token_ids / padding_mask, (N, 64) int64).
Input specification
Vision: image (B, 3, H, W) float32 [0,1]. H,W must be multiples of 14 (patch size).
Text: token_ids (B, 64) int64, padding_mask (B, 64) int64 (0=valid, 1=pad).
Usage
1from huggingface_hub import hf_hub_download
2import onnxruntime as ort, numpy as np
3from PIL import Image
4
5path = hf_hub_download("Armaggheddon/tips-v2-l14-onnx", "vision_encoder_fp16.onnx")
6sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
7img = np.array(Image.open("photo.jpg").convert("RGB").resize((448,448)), dtype=np.float32) / 255.0
8cls1, cls2, patches = sess.run(None, {"image": img.transpose(2,0,1)[None]})
FP32 models: the .onnx file references an external .onnx.data
companion — download both files from the repo. See the repo's
example_inference.py for a download helper that handles this automatically.
Evaluation
Numerical accuracy vs PyTorch FP32 baseline (ONNX Runtime CPU, batch=1):
| Precision | Vision cosine | Text cosine | Cross-modal Δ |
|---|
| FP32 | 1.000000 | 1.000000 | 1.5×10⁻⁸ |
| FP16 | 1.000000 | 1.000000 | 2.5×10⁻⁵ |
| INT8 dyn | 0.995445 | 0.996339 | 1.9×10⁻³ |
Performance
GPU latency at batch=1, 448×448 vision, RTX 3070 Ti (lower is better):
| Encoder | Precision | PT CUDA | ORT CUDA | TRT |
|---|
| Vision | FP32 | 83.3 ms | 63.5 ms | 51.9 ms |
| Vision | FP16 | 83.9 ms | 34.2 ms | 13.0 ms |
| Text | FP32 | 13.6 ms | 5.6 ms | — |
| Text | FP16 | 13.3 ms | 5.5 ms | 1.01 ms |
TensorRT deployment
Fixed spatial dimensions required for vision. Use the _448 exports:
1# Build vision engine (FP16)
2trtexec --onnx=onnx/L/vision_encoder_448_fp16.onnx \
3 --minShapes=image:1x3x448x448 --optShapes=image:1x3x448x448 --maxShapes=image:1x3x448x448 \
4 --saveEngine=onnx/L/vision_encoder_fp16.engine
5
6# Build text engine (FP16)
7trtexec --onnx=onnx/L/text_encoder_fp16.onnx \
8 --minShapes=token_ids:1x64,padding_mask:1x64 \
9 --optShapes=token_ids:1x64,padding_mask:1x64 \
10 --maxShapes=token_ids:1x64,padding_mask:1x64 \
11 --saveEngine=onnx/L/text_encoder_fp16.engine
Citation
1@InProceedings{tips_v2_paper,
2 Title={{TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment}},
3 Author={Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Ren\'e and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andr\'e},
4 Booktitle={CVPR},
5 year={2026},
6}
