DistilBERT-base-uncased LoRA Text Classification Model
Model Description
This model is a fine-tuned version of distilbert-base-uncased on an unspecified dataset. It achieves the following results on the evaluation set:
- Loss: 0.4649
- Accuracy: 84.16%
Intended Uses & Limitations
This is a text-classification based model.
Training and Evaluation Data
Look below for more details about the performances.
Steps to follow
- Installing the Libraries
- Loading the Dataset from HuggingFace
- Train_test Split the Dataset
- Model
- Preprocess Data
- Evaluation
- Apply untrained base model("distilbert-base-uncased") to text
- Train Model using LoRA
- Generate Prediction
- Save the Model and the Tokenizer
- Load the Model and the Tokenizer to test
- Push Model to HuggingFaceHub
Training Hyperparameters
The following hyperparameters were used during training:
- Learning Rate: 0.001
- Train Batch Size: 4
- Eval Batch Size: 4
- Seed: 42
- Optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- LR Scheduler Type: Linear
- Number of Epochs: 10
Training Results
| Epoch | Training Loss | Validation Loss | Validation Accuracy |
|---|
| 1.0 | 0.5924 | 0.5523 | 78.45% |
| 2.0 | 0.5983 | 0.5236 | 80.29% |
| 3.0 | 0.5703 | 0.4498 | 79.56% |
| 4.0 | 0.5526 | 0.4976 | 80.66% |
| 5.0 | 0.5326 | 0.4317 | 80.85% |
| 6.0 | 0.5851 | 0.4562 | 82.87% |
| 7.0 | 0.5466 | 0.4713 | 81.95% |
| 8.0 | 0.5494 | 0.5072 | 82.50% |
| 9.0 | 0.5748 | 0.4802 | 82.87% |
| 10.0 | 0.5001 | 0.4649 | 84.16% |
Framework Versions
- PEFT: 0.12.0
- Transformers: 4.42.4
- PyTorch: 2.4.0+cu121
- Datasets: 2.21.0
- Tokenizers: 0.19.1
Dataset Viewer
You can view the dataset using the following link:
Simply click the link to open the dataset viewer in your browser.
Model Viewer
You can view the model using the following link:
Simply click the link to open the model file in your browser.
Check out the "Fine-tune LLM.pptx" file for the theory behind this code.
Github Repository
You can view the github using the following link:
Simply click the link to open the github repo in your browser.
Check out the "Fine-tune LLM.pptx" file in the GitHub repo for the theory behind this code.
