DistilBERT on Finer-139
The goal of the project is to tune the DistilBERT model for a token
classification problem on the nlpaueb/finer-139 dataset.
Model
The model uses a fined tuned of
DistilBERT, a smaller and faster version of
Google's BERT model. For this project, the model was used for a token
classification problem, this means that the model shall return the same number
of labels as the input.
Dataset
The dataset used is the
Finer-139 dataset which is based on annual and
quarterly reports of different companies.
As a starting concept, the model is only fined tuned on the labels with the
smaller counts. These labels are listed below:
- CashAndCashEquivalentsFairValueDisclosure
- RevenueFromContractWithCustomerIncludingAssessedTax
- InterestExpense
- EmployeeServiceShareBasedCompensationTaxBenefitFromCompensationExpense
Performance
The model is a bit weak to getting the
RevenueFromContractWithCustomerIncludingAssessedTax compared to the other
selected labels. On the other hand, it has very strong performance on the
remaining labels, which improved the overall score.
Below is the summary of the performance on all the target labels.
| Label | Precision | Recall | F1 |
|---|
| CashAndCashEquivalentsFairValueDisclosure | 1.000 | 0.997 | 0.998 |
| EmployeeServiceShareBasedCompensationTaxBenefitFromCompensationExpense | 0.938 | 1.000 | 0.968 |
| InterestExpense | 0.949 | 0.945 | 0.947 |
| RevenueFromContractWithCustomerIncludingAssessedTax | 0.906 | 0.958 | 0.931 |
| Overall | 0.959 | 0.979 | 0.969 |
Usage
Example codes to fine-tune, and use both the Hugging Face and ONNX versions
of the model can be found by checking the
Source. Samples sentences can also
be fed directly on the Hugging Face
Model page.
Hugging Face
To use the model from the Hugging Face repository, simply load the model by
using the
Model repository.
1import torch
2from transformers import AutoTokenizer, AutoModelForTokenClassification
3
4tokenizer = AutoTokenizer.from_pretrained(HUGGING_FACE_REPOSITORY)
5model = AutoModelForTokenClassification.from_pretrained(HUGGING_FACE_REPOSITORY)
6model.eval()
7
8with torch.no_grad():
9 output = model(**dict(tokenizer("I paid $10 in interest", return_tensors="pt")))
ONNX
An exmple to export the ONNX model from the Hugging Face model can be seen
from
Source. For more details, feel free to check the Hugging Face
documentation on
Export to ONNX.
Below is an example python code to run the ONNX version of the model.
1from transformers import AutoTokenizer
2from onnxruntime import InferenceSession
3
4tokenizer = AutoTokenizer.from_pretrained(ONNX_OUTPUT_PATH)
5ort_session = InferenceSession(f"{ONNX_OUTPUT_PATH}/model.onnx")
6
7ort_output = ort_session.run(
8 output_names=["logits"],
9 input_feed=dict(tokenizer(["I paid $10 in interest"])),
10)
