DistilGPT2 Fine-Tuned on PubMedQA Artificial Subset
This model is a fine-tuned version of the DistilGPT2 for question-answering tasks in the biomedical domain. The model was trained on a subset of 50,000 artificial samples from the PubMedQA dataset.
Model Details
Model architecture: DistilGPT2
Training dataset: 50,000 samples from PubMedQA artificial subset
Training epochs: 3
Tokenizer maximum length: 512
Fine-Tuning Details: Model finetuning was done for three epochs using a standard model training pipeline provided by the Huggingface library. During training, the tokenizer was configured with a maximum token length of 512.
Example Usage
You can use this model for medical question-answering tasks by simply loading it with the Huggingface transformers library and providing a prompt.
from transformers import GPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("ritakurban/DistilGPT_PubMedQA")
model = GPT2LMHeadModel.from_pretrained("ritakurban/DistilGPT_PubMedQA")
prompt = "question: What is the primary function of the liver? context: The liver is a vital organ that performs several essential functions, including detoxification, protein synthesis, and the production of biochemicals necessary for digestion."
input_ids = tokenizer.encode(prompt, return_tensors="pt")
output = model.generate(input_ids, max_length=100, num_return_sequences=1, no_repeat_ngram_size=2)
response = tokenizer.decode(output[0], skip_special_tokens=True)
print(response)
Evaluation Metrics
Model performance was evaluated using Semantic Textual Similarity, Word Mover's Distance, and Grammar Errors. Detailed evaluation results can be found in the accompanying paper.
Limitations
While this model has been fine-tuned on a specific biomedical dataset, it may not perform equally well on other medical or general domain questions. Additionally, the model may generate plausible-sounding but incorrect answers. Always verify the generated answers with reliable sources before using them for critical decision-making.
Acknowledgements
We want to thank Huggingface for their excellent transformers library and the creators of the original DistilGPT2 model. We also thank the authors of the PubMedQA dataset for providing a valuable resource for training and evaluating biomedical question-answering models.
