The objective is to train an LLM to answer insurance questions.
This LLM will be able to generate a text that answers general questions about insurance when queried with relevant questions.
The motivation is that many people have insurance related questions and rely on their agents for answers. Some of the answers are not of high quality because the agent does not have enough knowledge or has a vested interest. The development of a good LLM to answer insurance related questions can be of good use to consumers, agents, customer service reps and other players in the insurance industry. Currently there isn’t an LLM that does well at answering insurance
Training Data
The model was trained on a 750 question and answer set from the Insurance QA corpus (“https://github.com/shuzi/insuranceQA”).
This corpus contains questions and answers collected from the website Insurance Library (“https://www.insurancelibrary.com/”).
This consists of questions from real world users, the answers with high quality were composed by professionals with deep domain knowledge.
A training to validation split of 80% (600) to 20% (150) split was used.
Training Method
The training method I implemented is Low Rank Adaptation (LoRA).
LoRA has faster training times than fine tuning and many other approaches.
It doesn't use too many memory resources. LoRA enables tuning without making substantial changes to the pretrained model.
In some research reviewed, LoRA achieved better results than fine-tuning, and required much fewer parameters to train.
I compared LoRA versus various versions of prompt tuning. The output from LoRA gave better results than prompt tuning.
GPT2, Llama-3.2-1B and Gemma-2-2b-it were considered as starting base models. Based on some evaluation, Llama-3.2-1B was chosen.
Data Sample
image/jpeg
The hyperparameter values used:
lora_r = 64
lora_alpha = 64
lora_dropout = .05
bias="none"
task_type="CAUSAL_LM"
learning_rate=0.00001
epochs=2
eval_steps=50
Different epochs and eval_steps were tested. The values shown above gave the best values.
Evaluation
I had four benchmarks: gsm8k, mmlu, race and percent correct response of a small sample.
Base
Prompt Tuning
LORA
gsm8k_cot
exact_match,strict-match
0.0400
0.0000
0.0333
gsm8k_cot
exact_match_stderr,strict-match
0.0280
0.0000
0.0333
gsm8k_cot
exact_match,flexible-extract
0.0400
0.0200
0.0667
gsm8k_cot
exact_match_stderr,flexible-extract
0.0280
0.0200
0.0463
mmlu
acc,none
0.387018
0.247368
0.387719
mmlu
acc_stderr,none
0.008993
0.008047
0.011674
race
acc,none
0.380000
0.340000
0.366667
race
acc_stderr,none
0.069341
0.067673
0.089486
percent correct
accurate
0.1500
0.2500
0.5000
Testing using the insurance set shows that the model after LoRA provides better answers
Before LoRA
Q: When Can I Switch From Medicare Advantage To Medigap?
A: The decision to switch from Medicare Advantage to Medigap is not an easy one. It involves a series of steps and decisions that will have an impact on your financial situation. This guide will help you understand your options and provide you with some guidance on the decision process. The first step is to assess your financial situation. This includes assessing your income, expenses, and assets. It is important to understand
After LoRA
Q: When Can I Switch From Medicare Advantage To Medigap?
A: You can switch from Medicare Advantage to Medigap at any time. You can also switch from Medicare Advantage to Medicare Part D at any time. If you switch from Medicare Advantage to Medicare Part D, you will have to switch back to Medicare Advantage to get your Medicare Part D coverage again.
Post training the model gives better answers to both the training and validation sample questions tested.
The answers before training sometimes are just like rambling without answering the question.
The model doesn't really change on the standard benchmarks
Usage and Intended Uses
The model can be used to get better responses to basic insurance related questions.
It can also be used for answer selection.
To use the model from the huggingface transformers library:
