BioCLIP-2 Quantized is a quantized version of BioCLIP-2, a foundation model for biological organismal images read more about the model here. The quantized model is designed to reduce memory usage and improve inference with cost of losing a little accuracy.
Model Description
BioCLIP-2 Quantized is dynamically quantized from the original BioCLIP-2 model. The Pytorch onnx.export function is used to convert the original model to ONNX format, and then the onnxruntime.quantization.quantize_dynamic function is used to perform dynamic quantization on the ONNX model with weight_type set to QuantType.QInt8.
To use the quantized model for inference, you can use the onnxruntime library. Here is an example code snippet:
python
12import onnxruntime as ort
3import torch
4import torch.nn.functional as F
5import numpy as np
6from huggingface_hub import hf_hub_download
789# Load the quantized model10ort_session = ort.InferenceSession("path/to/bioclip-2-quantized.onnx", providers=['CPUExecutionProvider'])1112# only return one label13k =11415# Preprocess image16img_tensor = preprocess_img(img).unsqueeze(0)17img_np = img_tensor.numpy()1819# Run ONNX inference20input_name = session.get_inputs()[0].name
21output_name = session.get_outputs()[0].name
2223img_features_np = session.run([output_name],{input_name: img_np})[0]2425# Convert back to torch for compatibility with existing code26img_features = torch.from_numpy(img_features_np)27img_features = F.normalize(img_features, dim=-1)2829# optional: for the open ended classification that you need a text embedding:30txt_emb = torch.from_numpy(31 np.load(32 hf_hub_download(33 repo_id="imageomics/TreeOfLife-200M",34 filename="embeddings/txt_emb_species.npy",35 repo_type="dataset",36)37)38)39# or you can skip this and use zero-shot classification with your own text inputs4041# Use the same text embeddings and logit scale from the original model (logit_scale of the main model: 100.00000762939453)42logits =(model.logit_scale.exp()* img_features @ txt_emb).squeeze()43probs = F.softmax(logits, dim=0)4445topk = probs.topk(k)46prediction_dict ={47 format_name(*txt_names[i]): prob
48for i, prob inzip(topk.indices, topk.values)49}5051print(prediction_dict)
Tradeoff
The Model got tested on Animals and Plants dataset by Nguyen Le Truong Thien for open-ended species classification and the highest probability class is selected as the predicted class and compared with the main BioCLIP-2 model. The results are as follows:
alt text
Acknowledgements
The main model is developed by the Imageomics Institute team. And the current model is just a quantized version of the main model to reduce memory usage and improve inference speed and make the model more accessible.
