MedMistral-SFT-7B is a French medical language model based on Mistral-7B-v0.1, adapted for medical domain applications through direct Supervised Fine-Tuning (SFT).

Model Details

Model Type: Causal Language Model
Base Model: Mistral-7B-v0.1
Language: French
Domain: Medical/Healthcare
License: Apache 2.0
Paper: Adaptation des connaissances médicales pour les grands modèles de langue : Stratégies et analyse comparative

Training Details

Supervised Fine-Tuning (SFT)

Dataset: 30K French medical question-answer pairs
- 10K native French medical questions
- 10K translated medical questions from English resources
- 10K generated questions from French medical texts
Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
Training Duration: 10 epochs
Rank: 16
Alpha: 16
LoRA Dropout: 0.05
Learning Rate: 2e-5
Batch Size: 4
Gradient Accumulation: 2
Optimizer: AdamW
Scheduler: Cosine
Target Modules: QKVOGUD

Computational Impact

Training Time: 53 hours
Carbon Emissions: 2.62 kgCO2e

Ethical Considerations

Medical Accuracy: This model is for research and educational purposes only. All outputs should be verified by qualified medical professionals
Bias: Training data may contain biases present in medical literature and online medical resources

Citation

If you use this model, please cite:

Contact

For questions about this model, please contact: ikram.belmadani@lis-lab.fr

ik-ram28/MedMistral-SFT-7B

Model Description