Transformer Efficiency and Knowledge Distillation
This model is a fine-tuned version of
distilbert-base-uncased on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 0.7872
- Accuracy: 0.9206
Model description
This setup involves benchmarking the performance of a fine-tuned BERT model (transformersbook/bert-base-uncased-finetuned-clinc) and applying knowledge distillation to train a smaller DistilBERT model. The BERT model is used for text classification tasks, and its efficiency is evaluated in terms of accuracy, model size, and latency. The DistilBERT model is trained to mimic the BERT model's performance while being more efficient.
Intended uses & limitations
Intended uses:
Evaluating the performance efficiency of transformer models.
Applying knowledge distillation to create smaller and faster models for text classification.
Limitations:
The benchmark results are specific to the dataset used (CLINC150) and may not generalize to other datasets.
Knowledge distillation relies on the quality and performance of the teacher model.
Training and evaluation data
The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.
Training procedure
Training and evaluation data
The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits.
Performance Benchmark
The performance of the BERT model is evaluated using the PerformanceBenchmark class, which measures accuracy, model size, and latency.
Accuracy
The model's accuracy is computed on the test set of the CLINC150 dataset.
accuracy_score = load_metric("accuracy")
Model Size
The size of the model is computed by saving its state dictionary to disk and measuring the file size in megabytes.
def compute_size(self):
state_dict = self.pipeline.model.state_dict()
tmp_path = Path("model.pt")
torch.save(state_dict, tmp_path)
size_mb = Path(tmp_path).stat().st_size / (1024 * 1024)
tmp_path.unlink()
return {"size_mb": size_mb}
Latency
The average latency per query is measured over a sample of 100 queries.
def time_pipeline(self):
latencies = []
for example in self.dataset[:100]:
start_time = perf_counter()
_ = self.pipeline(example)
latency = perf_counter() - start_time
latencies.append(latency)
time_avg_ms = 1000 * np.mean(latencies)
time_std_ms = 1000 * np.std(latencies)
return {"time_avg_ms": time_avg_ms, "time_std_ms": time_std_ms}
Knowledge Distillation
Knowledge distillation is used to train a smaller DistilBERT model using the predictions of the fine-tuned BERT model as soft labels.
Distillation Process
Teacher Model: transformersbook/bert-base-uncased-finetuned-clinc
Student Model: distilbert-base-uncased
The distillation process involves computing a weighted average of the cross-entropy loss with the ground truth labels and the Kullback-Leibler divergence between the teacher and student model predictions.
class DistillationTrainer(Trainer):
def compute_loss(self, model, inputs, return_outputs=False):
outputs_stu = model(**inputs)
loss_ce = outputs_stu.loss
logits_stu = outputs_stu.logits
with torch.no_grad():
outputs_tea = self.teacher(**inputs)
logits_tea = outputs_tea.logits
loss_fct = nn.KLDivLoss(reduction="batchmean")
loss_kd = self.args.temperature ** 2 * loss_fct(
F.log_softmax(logits_stu / self.args.temperature, dim=-1),
F.softmax(logits_tea / self.args.temperature, dim=-1)
)
loss = self.args.alpha * loss_ce + (1. - self.args.alpha) * loss_kd
return (loss, outputs_stu) if return_outputs else loss
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 48
- eval_batch_size: 48
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5
Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|
| No log | 1.0 | 318 | 3.2931 | 0.7255 |
| 3.8009 | 2.0 | 636 | 1.8849 | 0.8526 |
| 3.8009 | 3.0 | 954 | 1.1702 | 0.8897 |
| 1.7128 | 4.0 | 1272 | 0.8717 | 0.9145 |
| 0.9206 | 5.0 | 1590 | 0.7872 | 0.9206 |
Framework versions
- Transformers 4.41.1
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1
