# Load model directly
from transformers import AutoTokenizer, TFAutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("Tien-THM/bert-mini-fine-tuning-squad")
model = TFAutoModelForQuestionAnswering.from_pretrained("Tien-THM/bert-mini-fine-tuning-squad")

import numpy as np

def Inference(context, question):
  encoding = tokenizer(context, question, return_tensors='tf')
  start_pos = model(encoding).start_logits
  end_pos = model(encoding).end_logits
  s = np.argmax(start_pos[0])
  e = np.argmax(end_pos[0])
  print(tokenizer.decode(encoding['input_ids'][0][s:e+1]))

question = 'How many layes does BERT-large have'
context = 'BERT-large is really big... it has 24-layers and an embedding size 
of 1,024, for a total of 340M parameters! Altogether it is 1.34GB, so 
expect it to take a couple minutes to download to your Colab instance'

Inference(context, question)
# Answer: 24 - layers and an em ##bed ##ding size of 1 , 02 ##4

Training Details

Training Data

Using 2 datasets:

SQUAD

Training Procedure

Optimization:

Adam

Loss function

Cross entropy

Training Hyperparameters

Learning rate: 2e-5
Batch size: 8
Epoch: 4

Training Loss

Epoch	Train loss	Validation loss	Exact Match
#1	4.7110	3.6251	0.38
#2	3.2650	3.3062	0.42
#3	2.7899	3.2184	0.44
#4	2.4633	3.1946	0.45

Evaluation

Testing Data, Factors & Metrics

Metrics

Exact Match: 0.45