Author: Anurag Singh Framework: Hugging Face Transformers + PyTorch Base Model:distilbert-base-uncased Task: Binary classification for multiple-choice question answering
Model Description
This model is a fine-tuned DistilBERT model developed for the Smart MCQ Solver Challenge.
The original task contains a question with five answer options (A-E). The problem was converted into a binary classification task by creating one training example for each (question, option) pair.
Label 1: The option is correct.
Label 0: The option is incorrect.
During inference, the model predicts a probability for each option, and the option with the highest probability is selected as the final answer.
Problem Formulation
Original MCQ
Question: What is the capital of France?
A. Berlin
B. Madrid
C. Paris
D. Rome
E. Lisbon
Converted Binary Samples
Input
Label
Question: What is the capital of France? Option: Berlin
0
Question: What is the capital of France? Option: Madrid
0
Question: What is the capital of France? Option: Paris
1
Question: What is the capital of France? Option: Rome
0
Question: What is the capital of France? Option: Lisbon
0
Model Architecture
This model is based on DistilBERT, a compressed version of BERT that retains most of BERT's language understanding capability while being significantly smaller and faster.
Architecture
text Question + Option Text
↓
DistilBERT Tokenizer
↓
Input IDs
↓
DistilBERT Encoder (6 Transformer Layers)
↓
Contextual Representation
↓
Classification Head (Linear Layer)
↓
Binary Correctness Score
Model Details
Component
Value
Base Model
distilbert-base-uncased
Transformer Layers
6
Hidden Size
768
Attention Heads
12
Vocabulary Size
30,522
Max Position Embeddings
512
Classification Head
Linear layer (768 → 2)
Output Labels
2 (incorrect, correct)
Input Representation
Each training sample is constructed as:
Question: <question> Option: <option>
The text is tokenized using the DistilBERT WordPiece tokenizer and converted into:
input_ids
attention_mask
Classification
The contextual representation produced by DistilBERT is passed to a linear classification head that outputs logits for two classes:
Class 0: Incorrect option
Class 1: Correct option
During inference, the probability of class 1 is used as the correctness score for each option. The option with the highest score among A-E is selected as the final answer.
distilbert_output
This model is a fine-tuned version of bert-base-uncased on the private dataset.
It achieves the following results on the evaluation set:
Loss: 0.0985
Accuracy: 0.986
Precision: 0.9769
Recall: 0.9525
F1: 0.9646
Intended uses & limitations
Intended Uses
Multiple-choice question answering
Educational NLP applications
Research on transformer-based text classification
Binary text classification
Demonstration of parameter-efficient NLP models
Limitations
The model is trained only on the project dataset and may not generalize well to unrelated domains.
Performance depends on the quality and wording of the question and options.
The model evaluates each question-option pair independently and does not explicitly model relationships between all options simultaneously.
It is intended for educational and research purposes and should not be used in high-stakes decision-making systems without further validation.
Training and evaluation data
The model was trained on a multiple-choice question dataset where each question contained five candidate answers (A-E).
The original dataset was converted into a binary classification dataset by generating one training sample for every (question, option) pair.
Training labels:
0 → Incorrect option
1 → Correct option
Each input was tokenized using the DistilBERT tokenizer with appropriate padding and truncation before training.
Model performance was evaluated using:
Accuracy
Precision
Recall
F1-score
Validation Loss
Training procedure
Preprocessing
Combined each question with one candidate option to create the input text.
Tokenized the input using the DistilBERT WordPiece tokenizer.
Applied padding and truncation to a fixed maximum sequence length.
Converted labels into binary classes.
Fine-tuning
The model was fine-tuned using Hugging Face Transformers with the following configuration:
Parameter
Value
Base Model
distilbert-base-uncased
Task
Binary Sequence Classification
Loss Function
CrossEntropyLoss
Optimizer
AdamW
Learning Rate
2e-5
Number of Labels
2
Framework
PyTorch + Hugging Face Transformers
The trained model predicts the probability that a candidate option is correct. For each MCQ, all options are scored independently, and the option with the highest probability is selected as the final answer.
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 2e-05
train_batch_size: 32
eval_batch_size: 32
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5
Training results
The DistilBERT model achieved the best overall performance among the evaluated models in this project, outperforming both the custom BiLSTM and BERT models in terms of validation performance and Kaggle leaderboard score.
The model was selected as the final submission because it provided the best balance between prediction accuracy, computational efficiency, and inference speed.
