This model is a fine-tuned version of
distilbert-base-uncased on the
misdirection.csv dataset.
The data is cleaned by selecting relevant columns and filtering rows based on whether they are labeled as 'accepted' or 'rejected'. It then groups the data by a unique identifier, concatenates text entries within each group into paragraphs, and prepares these paragraphs as predictors (X). Target labels (y) are derived from the final submission grade, mapping 'accepted' to 'violation' and 'rejected' to 'non-violation'. Finally, the data is split into training and testing sets using stratified sampling with a 20% test size and a random state of 1 for reproducibility.
The code begins by loading a DistilBERT model and tokenizer configured for sequence classification with two possible labels. It then preprocesses the data: training and testing text sequences are tokenized using BERT, ensuring uniform length with padding and truncation to 256 tokens.
A CustomDataset class is defined to organize the tokenized data into a format suitable for PyTorch training, converting labels ('non-violation' and 'violation') into numeric values. Evaluation metrics such as accuracy, precision, recall, and F1 score are set up to assess model performance.
The main task is hyperparameter optimization using Optuna. An objective function is defined to optimize dropout rate, learning rate, batch size, epochs, and weight decay. For each trial, the data is tokenized again, a new model is initialized with the chosen dropout rate, and a Trainer object manages training and evaluation using these parameters. The goal is to maximize the F1 score across 15 trials.
