Model Card for Model ID
This project focuses on building a medical domain-specific chatbot using the DeepSeek R1 model. The chatbot is fine-tuned on the Medical O1 Reasoning SFT dataset to provide accurate, step-by-step reasoning for medical queries. It is designed to assist healthcare professionals, students, and patients by offering quick, reliable answers to medical questions while ensuring domain specificity.
Model Details
Model Description
Purpose of the Chatbot
The purpose of this project is to develop a medical domain-specific chatbot capable of
understanding and responding to medical queries with logical, step-by-step reasoning. The
chatbot is designed to assist healthcare professionals, students, and patients by providing
accurate and concise medical information.
Domain Alignment
The chatbot is aligned with the medical domain, focusing on clinical reasoning, diagnostics,
and treatment planning. It is trained on the Medical O1 Reasoning SFT dataset, which
contains medical questions, chain-of-thought reasoning, and responses. This alignment
ensures that the chatbot can handle complex medical queries while maintaining domain
specificity.
Relevance and Necessity
- Improved Access to Medical Knowledge: The chatbot provides quick, reliable
answers to medical questions, reducing the need for extensive manual research.
- Assistance for Healthcare Professionals: It helps healthcare professionals by
handling routine inquiries, allowing them to focus on more critical tasks.
- Educational Tool: The chatbot serves as a learning aid for medical students, offering
detailed explanations for medical concepts.
- Developed by: Oche Ankeli
- Funded by : Self-funded
Hardware Type
Hardware Type: GPU (A100 or T4) is sufficient
Preprocessing
Preprocessing Steps
- Tokenization: The dataset uses the DeepSeek R1 tokenizer, which employs
WordPiece for subword tokenization.
- Normalization:
○ Removed noise (e.g., special characters, irrelevant text).
○ Handled missing values by dropping rows with incomplete data.
○ Ensured consistent formatting across the dataset.
- Formatting:
○ Structured the dataset to include a system prompt, question, chain-of-thought
reasoning, and response.
○ Added a domain-specific restriction for non-medical questions.
Evaluation
The fine-tuned model was evaluated using the following metrics:
Quantitative Metrics
BLEU Score: Measures the similarity between the model's responses and reference answers.
Initial Model: 0.70
Fine-Tuned Model: 0.90
Improvement: 28.57%
Perplexity: Measures how well the model predicts the next token.
Initial Model: 3.50
Fine-Tuned Model: 2.68
Improvement: 23.43%
BERTScore: Evaluates the precision, recall, and F1 score of the generated text.
Precision: 0.80 (14.29% improvement)
Recall: 0.80 (23.08% improvement)
F1: 0.80 (19.40% improvement)
Summary
The chatbot effectively handles medical queries while maintaining domain
specificity, making it a valuable tool for healthcare professionals, students, and patients.
