This model is a fine-tuned version of microsoft/BioGPT-Large on a formatted version of the MedQuad-MedicalQnADataset dataset. It achieves the following results on the evaluation set:

Loss: 1.1114

Model description

The base model used is Microsoft's BioGPT, it was fine-tuned with a custom prompt for a conversational chatbot between a patient and a doctor. The prompt used is as follows:

"""You are a Doctor. Below is a question from a patient. Write a response to the patient that answers their question\n\n"

### Patient: {question}"

### Doctor: {answer}
"""

Inference

The fine-tuned model has a saved generation config, to use it:

model_config = GenerationConfig.from_pretrained(
    DoctorGPT
)

This config is a diverse beam search strategy:

diversebeamConfig = GenerationConfig(
    min_length=20,
    max_length=256,
    do_sample=False,
    num_beams=4,
    num_beam_groups=4,
    diversity_penalty=1.0,
    repetition_penalty=3.0,
    eos_token_id=model.config.eos_token_id,
    pad_token_id=model.config.pad_token_id,
    bos_token_id=model.config.bos_token_id,
)

For best results, please use this as your generator function:

def generate(query):
  sys = "You are a Doctor. Below is a question from a patient. Write a response to the patient that answers their question\n\n"
  patient = f"### Patient:\n{query}\n\n"
  doctor = f"### Doctor:\n "

  prompt = sys+patient+doctor

  inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
  generated_ids = model.generate(
                          **inputs,
                          generation_config=generation_config,
                          )
  outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
  answer = '.'.join(answer.split('.')[:-1])
  torch.cuda.empty_cache()
  return answer + "."

Intended uses & limitations

This is a private project for fine-tuning a medical language model, it is not intended to be used as a source of medical advice.

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 4
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 64
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.03
num_epochs: 3
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.25	51	1.2418
1.3267	0.5	102	1.1900
1.3267	0.75	153	1.1348
1.1237	0.99	204	1.0887
1.1237	1.24	255	1.1018
0.7527	1.49	306	1.0770
0.7527	1.74	357	1.0464
0.7281	1.99	408	1.0233
0.7281	2.24	459	1.1212
0.4262	2.49	510	1.1177
0.4262	2.73	561	1.1125
0.4124	2.98	612	1.1114

Framework versions

Transformers 4.36.0.dev0
Pytorch 2.1.0+cu118
Datasets 2.15.0
Tokenizers 0.15.0

RobCzikkel/DoctorGPT

DoctorGPT