This model is a fine-tuned version of Qwen/Qwen2.5-3B-Instruct on the Books dataset for Essay Grading.

The Books dataset is a synthetic collection of essay-style data points generated using public domain literature and large language model prompting. The dataset comprises a total of 300 entries and is built from six classic books. Four of these: The Life of James Watt, The Life of Julius Caesar, The Moonstone, and North and South; were used during the training phase, while the remaining two: The Life of Napoleon and Sense and Sensibility; were held out for benchmarking purposes. Each book contributed exactly 50 entries, leading to a structured split of 200 training samples and 100 benchmark samples.

All entries were generated using Le Chat Mistral, a model developed by Mistral AI. A carefully crafted prompt was used to ensure each generated entry included a question, a reference answer written by an expert, a student answer meant to simulate a real-world response, a mark scheme outlining the grading criteria, a score between 1 and 4, and a rationale explaining why the score was assigned. The prompt enforced strict quality control: no duplicate questions or answers were allowed, all required fields had to be present, and the scoring range was strictly limited to valid values. The final output was formatted as CSV files to maintain consistency and ensure compatibility with downstream processing.

For more details, the metadata can be accessed at: metadata.

The modeling approach for this study was carefully designed to evaluate the performance of different large language models (LLMs) on the automated essay grading task. We selected the Qwen2.5 architecture to represent a range of model sizes: 0.5B, 1.5B, and 3B. Each model was instruction-tuned on the Books dataset in varying sizes, with hyperparameters optimized to balance computational efficiency and performance. The experiments were conducted on GPU-accelerated hardware, leveraging techniques such as gradient checkpointing, flash attention, and mixed-precision training to maximize resource utilization.

The evaluation methodology employed both quantitative metrics and qualitative analysis. For quantitative assessment, we computed accuracy, precision, recall, F1 score, root mean squared error (RMSE), and Cohen's kappa score (CKS) for the scoring task, while using BERT-Score precision, recall, and F1 for rationale evaluation. On a held-out test set of 100 samples. Qualitative examination of models' outputs revealed cases where most of the models correctly identified key aspects of student answers but sometimes failed to properly align its scoring with the rubric criteria.

Below is an example of how to use the model with the Hugging Face Transformers library: