BERT Feature-based Classifier for MNLI
This model is a feature-based transfer learning classifier built on top of a frozen BERT-base-uncased encoder, fine-tuned on the MNLI (Multi-Genre Natural Language Inference) dataset. Instead of fine-tuning the entire BERT model, the encoder was frozen and a custom classification head was trained on top using pooled representations of sentence pairs.
Model Description
- Base model: bert-base-uncased
- Approach: Feature-based transfer learning
- Task: Natural Language Inference (NLI) on MNLI dataset
- Head architecture:
- Concatenation of
[CLS] embedding and mean-pooled hidden states
- Fully connected feed-forward classifier with LayerNorm, ReLU, Dropout layers
Training Details
- Dataset: GLUE: MultiNLI
- Training split size: ~392k sentence pairs
- Validation splits:
validation_matched, validation_mismatched
- Frozen parameters: All BERT parameters (only classification head trained)
- Optimizer: AdamW (lr=3e-5)
- Scheduler: StepLR (decay every 10 epochs, γ=0.5)
- Loss function: CrossEntropyLoss
- Batch size: 64
- Epochs: 50
- Device: CUDA (if available)
Results
After 50 epochs, the model achieved the following performance:
Final Training Loss: 0.1935
Validation (Matched split)
- Accuracy: 0.7878
- F1 Score: 0.7879
- Precision: 0.7882
- Recall: 0.7878
Validation (Mismatched split)
- Accuracy: 0.7941
- F1 Score: 0.7942
- Precision: 0.7944
- Recall: 0.7941
