bert-20news-classification
This model is a fine-tuned version of
distilbert-base-uncased on an unknown dataset.
It achieves the following results on the evaluation set:
- Train Loss: 0.0479
- Train Accuracy: 0.9922
- Validation Loss: 0.2769
- Validation Accuracy: 0.9284
- Epoch: 9
Model description
This model is a fine-tuned version of the DistilBERT model for sequence classification tasks. It was trained using Hugging Face's transformers and TensorFlow. The model expects input sequences to be tokenized according to the DistilBERT's tokenizer.
The model was trained specifically for classifying text into 20 different categories derived from the 20 Newsgroups dataset. These categories include various topics such as 'alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc'.
Intended uses & limitations
This model is intended for classifying text into the above mentioned 20 categories. It can be used for categorizing text data from similar domains or topics.
Training and evaluation data
the model was trained on 90% of the data from the 20 Newsgroups dataset, with the remaining 10% used for validation.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': {'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 2e-05, 'decay_steps': 2120, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
- training_precision: float32
Training results
| Train Loss | Train Accuracy | Validation Loss | Validation Accuracy | Epoch |
|---|
| 1.8498 | 0.5829 | 0.9285 | 0.8012 | 0 |
| 0.6611 | 0.8406 | 0.4800 | 0.8807 | 1 |
| 0.3563 | 0.9128 | 0.3829 | 0.9002 | 2 |
| 0.2276 | 0.9475 | 0.3593 | 0.9072 | 3 |
| 0.1544 | 0.9659 | 0.3205 | 0.9214 | 4 |
| 0.1094 | 0.9779 | 0.3007 | 0.9214 | 5 |
| 0.0825 | 0.9846 | 0.2821 | 0.9258 | 6 |
| 0.0634 | 0.9895 | 0.2754 | 0.9337 | 7 |
| 0.0533 | 0.9916 | 0.2707 | 0.9337 | 8 |
| 0.0479 | 0.9922 | 0.2769 | 0.9284 | 9 |
Framework versions
- Transformers 4.28.0
- TensorFlow 2.12.0
- Datasets 2.12.0
- Tokenizers 0.13.3
