Views
No views yet
Dynamic-TinyBERT is a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop).
| Model Detail | Description |
|---|---|
| Model Authors - Company | Intel |
| Model Card Authors | Intel in collaboration with Hugging Face |
| Date | November 22, 2021 |
| Version | 1 |
| Type | NLP - Question Answering |
| Architecture | "For our Dynamic-TinyBERT model we use the architecture of TinyBERT6L: a small BERT model with 6 layers, a hidden size of 768, a feed forward size of 3072 and 12 heads." Guskin et al. (2021) |
| Paper or Other Resources | Paper; Poster; GitHub Repo |
| License | Apache 2.0 |
| Questions or Comments | Community Tab and Intel Developers Discord |
| Intended Use | Description |
|---|---|
| Primary intended uses | You can use the model for the NLP task of question answering: given a corpus of text, you can ask it a question about that text, and it will find the answer in the text. |
| Primary intended users | Anyone doing question answering |
| Out-of-scope uses | The model should not be used to intentionally create hostile or alienating environments for people. |
1import torch
2from transformers import AutoTokenizer, AutoModelForQuestionAnswering
3
4tokenizer = AutoTokenizer.from_pretrained("Intel/dynamic_tinybert")
5model = AutoModelForQuestionAnswering.from_pretrained("Intel/dynamic_tinybert")
6
7context = "remember the number 123456, I'll ask you later."
8question = "What is the number I told you?"
9
10# Tokenize the context and question
11tokens = tokenizer.encode_plus(question, context, return_tensors="pt", truncation=True)
12
13# Get the input IDs and attention mask
14input_ids = tokens["input_ids"]
15attention_mask = tokens["attention_mask"]
16
17# Perform question answering
18outputs = model(input_ids, attention_mask=attention_mask)
19start_scores = outputs.start_logits
20end_scores = outputs.end_logits
21
22# Find the start and end positions of the answer
23answer_start = torch.argmax(start_scores)
24answer_end = torch.argmax(end_scores) + 1
25answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[0][answer_start:answer_end]))
26
27# Print the answer
28print("Answer:", answer)| Factors | Description |
|---|---|
| Groups | Many Wikipedia articles with question and answer labels are contained in the training data |
| Instrumentation | - |
| Environment | Training was completed on a Titan GPU. |
| Card Prompts | Model deployment on alternate hardware and software will change model performance |
| Metrics | Description |
|---|---|
| Model performance measures | F1 |
| Decision thresholds | - |
| Approaches to uncertainty and variability | - |
| Training and Evaluation Data | Description |
|---|---|
| Datasets | SQuAD1.1: "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable." (https://huggingface.co/datasets/squad) |
| Motivation | To build an efficient and accurate model for the question answering task. |
| Preprocessing | "We start with a pre-trained general-TinyBERT student, which was trained to learn the general knowledge of BERT using the general-distillation method presented by TinyBERT. We perform transformer distillation from a fine- tuned BERT teacher to the student, following the same training steps used in the original TinyBERT: (1) intermediate-layer distillation (ID) — learning the knowledge residing in the hidden states and attentions matrices, and (2) prediction-layer distillation (PD) — fitting the predictions of the teacher." (Guskin et al., 2021) |
| Model | Max F1 (full model) | Best Speedup within BERT-1% |
|---|---|---|
| Dynamic-TinyBERT | 88.71 | 3.3x |
| Ethical Considerations | Description |
|---|---|
| Data | The training data come from Wikipedia articles |
| Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of labelled Wikipedia articles. |
| Mitigations | No additional risk mitigation strategies were considered during model development. |
| Risks and harms | Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al., 2021, and Bender et al., 2021). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. Beyond this, the extent of the risks involved by using the model remain unknown. |
| Use cases | - |
| Caveats and Recommendations |
|---|
| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model. |
1@misc{https://doi.org/10.48550/arxiv.2111.09645,
2 doi = {10.48550/ARXIV.2111.09645},
3
4 url = {https://arxiv.org/abs/2111.09645},
5
6 author = {Guskin, Shira and Wasserblat, Moshe and Ding, Ke and Kim, Gyuwan},
7
8 keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
9
10 title = {Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length},
11
12 publisher = {arXiv},
13
14 year = {2021},