MedMistralInstruct-CPT-SFT-7B is a French medical language model based on Mistral-7B-Instruct-v0.1, adapted through Continual Pre-Training followed by Supervised Fine-Tuning.
Model Details
Model Type: Causal Language Model
Base Model: Mistral-7B-Instruct-v0.1
Language: French
Domain: Medical/Healthcare
Parameters: 7 billion
License: Apache 2.0
Training Details
Continual Pre-Training (CPT)
Dataset: NACHOS corpus (7.4 GB French medical texts)
Training Duration: 2.8 epochs
Hardware: 32 NVIDIA A100 80GB GPUs
Training Time: ~40 hours
Supervised Fine-Tuning (SFT)
Dataset: 30K French medical question-answer pairs
Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
Training Duration: 10 epochs
Hardware: 1 NVIDIA H100 80GB GPU
Training Time: ~42 hours
Computational Requirements
Carbon Emissions: 33.96 kgCO2e (CPT+SFT)
Training Time: 82 hours total (CPT+SFT)
Ethical Considerations
Medical Accuracy: For research and educational purposes only
Professional Oversight: Requires verification by qualified medical professionals
Bias Awareness: May contain biases from training data
