You can deploy this model and start earning money today!

keras/
blip2_flan_t5_xl

Name: blip2_flan_t5_xl
Availability: InStock

keras-hub

2301.12597

Views

No views yet

Model Overview

Model Summary

BLIP-2 (Bootstrapping Language-Image Pre-training) is a generic and efficient pre-training strategy that bridges the modality gap between frozen image encoders and frozen large language models (LLMs). It was introduced by Salesforce in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models".

The architecture consists of three main components: a frozen image encoder (EVA-CLIP ViT-g/14), a lightweight querying transformer (Q-Former) that acts as an information bottleneck to extract the most relevant visual features, and a frozen LLM (like OPT or Flan-T5) that handles the text generation. Because the heavy vision and language models are kept frozen during pre-training, BLIP-2 achieves state-of-the-art performance on various vision-language tasks with significantly fewer trainable parameters than existing methods.

Key Features:

State-of-the-art vision-language pre-training method
Combines frozen EVA-CLIP vision encoder with frozen LLMs (OPT, Flan-T5)
Highly efficient pre-training via the lightweight Q-Former
Capable of visual question answering, image captioning, and conversational image understanding

Model Details

Model Family: BLIP-2
Architecture: Vision-Language Model (EVA-CLIP ViT + Q-Former + LLM)
Developer: Salesforce
Paper: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Source: Salesforce BLIP-2 Collection
Task Type: Image-to-Text / Vision-Language Generation
Max Sequence Length: Varies by LLM (typically 512+ tokens)

Architecture Details (BLIP-2 General)

Vision Encoder: EVA-CLIP ViT-g/14 (Frozen)
Vision Encoder Parameters: ~1 Billion
Q-Former: Querying Transformer (Trainable)
Q-Former Parameters: ~188 Million
Language Model: Varies (OPT-2.7B, OPT-6.7B, Flan-T5-XL, Flan-T5-XXL) (Frozen)
Total Parameters: ~3B to ~12B (depending on the LLM variant)

[BLIP2 Quickstart Notebook](coming soon..!)
[BLIP2 API Documentation](coming soon..!)
BLIP2 Model Card
BLIP2 Technical Paper
KerasHub Beginner Guide
KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Preset Table

Preset	Architecture	Vision Encoder	Language Model	Description
`blip2_opt_2.7b`	BLIP-2	EVA-CLIP ViT-g/14	OPT-2.7B	BLIP-2 model using OPT-2.7B as the frozen language model.
`blip2_opt_6.7b`	BLIP-2	EVA-CLIP ViT-g/14	OPT-6.7B	BLIP-2 model using OPT-6.7B as the frozen language model.
`blip2_flan_t5_xl`	BLIP-2	EVA-CLIP ViT-g/14	Flan-T5-XL	BLIP-2 model using Flan-T5-XL (~3B) as the frozen language model.
`blip2_flan_t5_xxl`	BLIP-2	EVA-CLIP ViT-g/14	Flan-T5-XXL	BLIP-2 model using Flan-T5-XXL (~11B) as the frozen language model.

Example Usage

BLIP-2 can be used for various vision-language tasks such as image captioning and visual question answering (VQA). Depending on the preset you choose, you will use either BLIP2CausalLM (for OPT models) or BLIP2Seq2SeqLM (for Flan-T5 models).

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

python

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("blip2_flan_t5_xl")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))

Example Usage with Hugging Face URI

The BLIP2Seq2SeqLM class supports BLIP-2 variants that use Flan-T5 as their base language model. This architecture requires passing your text prompt to the encoder using the "encoder_text" key.

python

import keras
import keras_hub
import numpy as np
from PIL import Image
import requests

model = keras_hub.models.BLIP2Seq2SeqLM.from_preset("hf://keras/blip2_flan_t5_xl")

image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
image_array = np.array(image)

vqa_input = {
    "images": image_array,
    "encoder_text": ["Question: what is in the picture? Answer:"]
}
print(model.generate(vqa_input))

caption_input = {
    "images": image_array,
    "encoder_text": ["A picture of"]
}
print(model.generate(caption_input))