Foundation Model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
Dataset: https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max

The model has been fine-tuned using CUDA-enabled GPU hardware with optimized training through Unsloth.

Model Details

The model is based upon the foundation model: Mistral-7B-Instruct-v0.1.
It has been tuned with Supervised Fine-tuning Trainer using the Unsloth optimization framework for faster and more efficient training.

Libraries

unsloth
transformers
torch
trl
peft
einops
bitsandbytes
datasets

Training Configuration

Model Parameters

max_sequence_length = 2048
load_in_4bit = True
LoRA rank (r) = 32
lora_alpha = 16
lora_dropout = 0

Target Modules for LoRA

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Training Hyperparameters

per_device_train_batch_size = 2
gradient_accumulation_steps = 16
warmup_steps = 5
warmup_ratio = 0.03
max_steps = 1600
learning_rate = 1e-4
weight_decay = 0.01
lr_scheduler_type = "linear"
optimizer = "paged_adamw_32bit"

Training Statistics

Hardware Utilization

Training duration: 10,561.28 seconds (approximately 176.02 minutes)
Peak reserved memory: 5.416 GB
Peak reserved memory for training: 0.748 GB
Peak reserved memory % of max memory: 13.689%
Peak reserved memory for training % of max memory: 1.891%

Dataset

The model was trained on the MedQuad dataset, which contains medical questions and answers. The training data was processed using a chat template format for instruction-tuning.

Bias, Risks, and Limitations

Users (both direct and downstream) should be aware of the following:

This model is intended for medical question-answering but should not be used as a substitute for professional medical advice.

The model's responses should be verified by healthcare professionals before making any medical decisions.

Generation of plausible yet incorrect medical information remains a possibility.

The model's knowledge is limited to its training data and may not cover all medical conditions or recent medical developments.

Usage

The model can be loaded and used with the Unsloth library:

python

from unsloth import FastLanguageModel
max_seq_length = 2048  # Choose any! We auto support RoPE Scaling internally!
dtype = (
    None  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
)
model, tokenizer = FastLanguageModel.from_pretrained(
    "bouthros/Mistral-7B-Instruct-v0.1-Unsloth-MedicalQA",
    max_seq_length=2048,
    load_in_4bit=True,
)

Example usage:

python

messages = [
    {"from": "human", "value": "What are the types of liver cancer?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

Model Access

The model is available on Hugging Face Hub at: bouthros/Mistral-7B-Instruct-v0.1-Unsloth-MedicalQA

Citation

If you use this model, please cite the original Mistral-7B-Instruct-v0.1 model and the MedQuad dataset.