You can deploy this model and start earning money today!

AhmedSSabir/
BERT-CNN-Visual-Semantic

2301.08784

2201.12086

Views

No views yet

Visual Semantic with BERT-CNN

This model can be used to assign an object-to-caption semantic relatedness score, which is valuable for (1) caption diverse re-ranking (this work), and (2) (as an application) generating soft labels for filtering out the related/non-related image-to-post when scraping images from the internet (e.g. Instagram).

To take advantage of the overlapping between the visual context and the caption, and to extract global information from each visual (i.e., object, scene, etc) we use BERT as an embedding layer followed by a shallow CNN (tri-gram kernel) (Kim, 2014).

Please refer to Github for more information.

arXiv

For datasets that are less than 100K please have look at our shallow model

The model is trained with a strict filter of 0.4 similarity distance thresholds between the object and its related caption.

For a quick start please have a look at this demo

For the dataset

# Result with SoTA pre-trained image Captioning BLIP

Comparison result with BLIP (125M pre-trained images) Table 7 COCO Caption Karpathy testset. For the VilBERT model (3.5M pre-trained images) please refer to the paper.

Accuarcy

Model	B-1	B-2	B-3	B-4	M	R	C	S	BERTscore
BLIP Beam Search b=3	.797	.649	.514	.403	.311	.606	1.365	.243	.9484
+ BERT-CNN $th=0$	.798	.646	.506	.392	.305	.598	1.339	.238	.9473
+ BERT-CNN $th\geq0.2$	.798	.647	.507	.393	.306	.600	1.342	.238	.9473
+ BERT-CNN $th\geq0.3$	.802	.651	.511	.397	.307	.601	1.349	.238	.9479
+ BERT-CNN $th\geq0.4$	.806	.654	.513	.397	.303	.599	1.343	.235	.9476

Diversity

Model	Uniq	Voc	mBLeu-1↓	Div-1	Div-2	SBERT-sts
BLIP Beam Search b=3	8.60	1406	.461	.68	.80	.8058
+ BERT-CNN $th=0$	8.49	1532	.457	.68	.80	.8046
+ BERT-CNN $th\geq0.2$	8.48	1486	.458	.68	.80	.8052
+ BERT-CNN $th\geq0.3$	8.41	1448	.458	.68	.80	.8060
+ BERT-CNN $th\geq0.4$	8.30	1448	.455	.68	.80	.8053
human	9.14	3425	.375	.74	.84	NA

conda create -n BERT_visual python=3.6 anaconda
conda activate BERT_visual
pip install tensorflow==1.15.0
pip install --upgrade tensorflow_hub==0.7.0

git clone https://github.com/gaphex/bert_experimental/

python

import tensorflow as tf
import numpy as np
import pandas as pd
import sys
from sklearn.model_selection import train_test_split

sys.path.insert(0, "bert_experimental")

from bert_experimental.finetuning.text_preprocessing import build_preprocessor
from bert_experimental.finetuning.graph_ops import load_graph

df = pd.read_csv("test.tsv", sep='\t')

texts = []
delimiter = " ||| "

for vis, cap  in zip(df.visual.tolist(), df.caption.tolist()):
  texts.append(delimiter.join((str(vis), str(cap))))

texts = np.array(texts)

trX, tsX = train_test_split(texts, shuffle=False, test_size=0.01)

restored_graph = load_graph("frozen_graph.pb")

graph_ops = restored_graph.get_operations()
input_op, output_op = graph_ops[0].name, graph_ops[-1].name
print(input_op, output_op)

x = restored_graph.get_tensor_by_name(input_op + ':0')
y = restored_graph.get_tensor_by_name(output_op + ':0')

preprocessor = build_preprocessor("vocab.txt", 64)
py_func = tf.numpy_function(preprocessor, [x], [tf.int32, tf.int32, tf.int32], name='preprocessor')

##predictions
sess = tf.Session(graph=restored_graph)

print(trX[:4])

y = tf.print(y, summarize=-1)
y_out = sess.run(y, feed_dict={
        x: trX[:4].reshape((-1,1))
 	
    })


print(y_out)

For training and inference

python BERT_CNN.py --train train_0.4.tsv --epochs 5