Views
No views yet
sentence-transformers and uses mean of the token embeddings as the aggregation function. I also converted it to TensorFlow with the aggregation function rewritten in TF to use it in my ai-aas repo on GitHub for production-grade deployment, but a simple usage example is as follows:1import time
2import tensorflow as tf
3from transformers import TFAutoModel, AutoTokenizer
4
5texts = ["Galatasaray, bu akşamki maçın ardından şampiyonluğunu ilan etmeye hazırlanıyor."]
6labels = ["spor", "siyaset", "kültür"]
7model_name = 'mys/bert-base-turkish-cased-nli-mean'
8
9tokenizer = AutoTokenizer.from_pretrained(model_name)
10model = TFAutoModel.from_pretrained(model_name)
11
12
13def label_text(model, tokenizer, texts, labels):
14 texts_length = len(texts)
15 tokens = tokenizer(texts + labels, padding=True, return_tensors='tf')
16 embs = model(**tokens)[0]
17
18 attention_masks = tf.cast(tokens['attention_mask'], tf.float32)
19 sample_length = tf.reduce_sum(attention_masks, axis=-1, keepdims=True)
20 masked_embs = embs * tf.expand_dims(attention_masks, axis=-1)
21 masked_embs = tf.reduce_sum(masked_embs, axis=1) / tf.cast(sample_length, tf.float32)
22
23 dists = tf.experimental.numpy.inner(masked_embs[:texts_length], masked_embs[texts_length:])
24 scores = tf.nn.softmax(dists)
25 results = list(zip(labels, scores.numpy().squeeze().tolist()))
26 sorted_results = sorted(results, key=lambda x: x[1], reverse=True)
27 sorted_results = [{"label": label, "score": f"{score:.4f}"} for label, score in sorted_results]
28 return sorted_results
29
30
31start = time.time()
32sorted_results = label_text(model, tokenizer, texts, labels)
33alapsed = time.time() - start
34
35print(sorted_results)
36print(f"Processed in {alapsed:.2f} secs")1[{'label': 'spor', 'score': '1.0000'}, {'label': 'siyaset', 'score': '0.0000'}, {'label': 'kültür', 'score': '0.0000'}]
2Processed in 0.22 secslabel_text() function runs the BERT model with a concatenation of texts and labels as the input, and it agregates per-token hidden states outputted by the BERT model to produce a single vector per sequence. Then, the inner product of text embeddings and label embeddings is calculated as the similarity metric, and softmax is applied to convert these distance values to probabilities.Emrah Budur, Rıza Özçelik, Tunga Güngör and Christopher Potts. 2020. Data and Representation for Turkish Natural Language Inference. To appear in Proceedings of EMNLP. [pdf] [bib]
@inproceedings{budur-etal-2020-data,
title = "Data and Representation for Turkish Natural Language Inference",
author = "Budur, Emrah and
\"{O}z\c{c}elik, R{\i}za and
G\"{u}ng\"{o}r, Tunga",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics"
}