You can deploy this model and start earning money today!

Armaggheddon/
tips-v2-so400m-onnx

Name: tips-v2-so400m-onnx
Availability: InStock

google/tipsv2-so400m14

quantized

apache-2.0

Views

No views yet

TIPSv2-So400m/14 — ONNX Export

SoViT-400m variant (412M vision / 448M text, 27-layer symmetric architecture). 448×448 images → 1152-dim embeddings.

Note: This variant has 27 text encoder layers (matching the vision tower depth). TensorRT engine build is not supported due to internal dynamic shapes.

Original: google-deepmind/tips.
Exported with tips-onnx — see the repo for custom exports (other precisions, fixed sizes) and TensorRT engine builds.

Available files

File	Precision	Size	Backend
`vision_encoder_fp32.onnx`	FP32	1.54 GB	CPU / CUDA
`text_encoder_fp32.onnx`	FP32	1.67 GB	CPU / CUDA
`vision_encoder_fp16.onnx`	FP16	789 MB	CPU / CUDA
`text_encoder_fp16.onnx`	FP16	855 MB	CPU / CUDA ⚠️
`vision_encoder_int8_dynamic.onnx`	INT8 (CPU)	396 MB	CPU
`text_encoder_int8_dynamic.onnx`	INT8 (CPU)	429 MB	CPU

⚠️ FP16 text encoder needs ORT optimisations disabled at batch=1 — see Known issue.

Calibration

INT8 Q/DQ quantization (tools/export.py --precision int8_qdq) needs calibration data in the vision/text form of the encoder inputs, from any source — at least 64 samples per encoder (the minimum suggested by NVIDIA ModelOpt; this project's calibration used 500). The development data was built from lmms-lab/COCO-Caption (500 images + captions).

Rebuild calibration data with tools/make_calibration.py (synthetic, structural testing only) or from your own dataset in the same format: checkpoints/calib_vision.npy ((N, 3, 448, 448) float32 images) and checkpoints/calib_text.npz (token_ids / padding_mask, (N, 64) int64).

Input specification

Vision: image (B, 3, H, W) float32 [0,1]. H,W must be multiples of 14 (patch size). Text: token_ids (B, 64) int64, padding_mask (B, 64) int64.

Usage

python

from huggingface_hub import hf_hub_download
import onnxruntime as ort, numpy as np
from PIL import Image

path = hf_hub_download("Armaggheddon/tips-v2-so400m-onnx", "vision_encoder_fp32.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
img = np.array(Image.open("photo.jpg").convert("RGB").resize((448,448)), dtype=np.float32) / 255.0
cls1, cls2, patches = sess.run(None, {"image": img.transpose(2,0,1)[None]})

FP32 models: the .onnx file references an external .onnx.data companion — download both files from the repo. See the repo's example_inference.py for a download helper that handles this automatically.

Evaluation

Numerical accuracy vs PyTorch FP32 baseline (ONNX Runtime CPU, batch=1):

Precision	Vision cosine	Text cosine	Cross-modal Δ
FP32	1.000000	1.000000	3.7×10⁻⁸
FP16	1.000000	1.000000	4.2×10⁻⁵
INT8 dyn	0.995052	0.649074	5.8×10⁻³

Performance

GPU latency at batch=1, 448×448 vision, RTX 3070 Ti. TRT not available.

Encoder	Precision	PT CUDA	ORT CUDA
Vision	FP32	106.3 ms	85.2 ms
Vision	FP16	107.4 ms	46.8 ms
Text	FP32	27.0 ms	14.0 ms
Text	FP16	30.0 ms	⚠️ —*

*FP16 text encoder triggers the FusedMatMul buffer-reuse bug at batch=1. See Known issue.

Known issue: ORT `FusedMatMul` on FP16 text encoder

The FP16 text encoder may fail on ONNX Runtime ≥ 1.15 with default graph optimisations when batch=1:

Shape mismatch attempting to re-use buffer. {1,16,64,64} != {1,16,64,72}

Root cause: ORT's FusedMatMul optimisation fuses the two matmuls in each attention head into a shared kernel. For So400m/14 the per-head dimension (1152 ÷ 16 = 72) differs from the sequence length (64). The ORT allocation planner allocates a buffer sized for the larger dimension and then incorrectly reuses it for the smaller one (upstream bug microsoft/onnxruntime#23739).

This does not affect the ONNX model itself — it validates correctly with onnx.checker and runs correctly at batch > 1 or with optimisations disabled.

Workaround — disable ORT graph optimisations for this specific model: