A small domain-specific language model prototype created by fine-tuning Qwen/Qwen2.5-0.5B-Instruct with LoRA (Low-Rank Adaptation) on a curated pharmaceutical drug-interaction dataset.
Version: v1.0
Base Model:Qwen/Qwen2.5-0.5B-InstructFine-Tuning: LoRA
Model Size: 0.5B parameters
Purpose: Educational / Research
Status: Learning Prototype
This project explores how a small Large Language Model (LLM) can be adapted toward the pharmaceutical drug-interaction domain using parameter-efficient fine-tuning.
Instead of training an LLM from scratch, this project takes an existing instruction-tuned model:
Qwen/Qwen2.5-0.5B-Instruct
and trains a small set of additional LoRA parameters using pharmaceutical drug-interaction examples.
This makes the training process significantly lighter than full-model fine-tuning.
📊 Dataset
The training data was created from pharmaceutical/drug-interaction datasets.
The preparation pipeline included:
text
1Raw Pharmaceutical Data
2 │
3 ▼
4Drug Name Normalization
5 │
6 ▼
7Drug Matching
8 │
9 ▼
10Interaction Records
11 │
12 ▼
13Conversational Training Examples
14 │
15 ▼
16train.jsonl
For this V1 prototype, the pharmaceutical drug selection was intentionally limited to a smaller subset because of available compute resources.
V1 scope
Top 100 drugs
13,534 matched interaction records
Multiple conversational question variations were generated for the interaction records
Final training file: train.jsonl
The smaller scope was intentional so the project could be trained and tested on available hardware.
🧪 Training Format
The training examples use a conversational format compatible with instruction-tuned models.
Example:
json
1{2"messages":[3{4"role":"user",5"content":"Does Zolpidem Tartrate interact with Itraconazole?"6},7{8"role":"assistant",9"content":"Interaction: increase drug exposure.\n\nItraconazole increased the exposure of zolpidem..."10}11]12}
Multiple question formulations were generated for interaction records to expose the model to different ways a user might ask about the same drug interaction.
⚙️ Training Configuration
Parameter
Value
Base model
Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning
LoRA
LoRA rank (r)
16
LoRA alpha
32
LoRA dropout
0.05
Bias
None
Task
Causal Language Modeling
Target modules
q_proj, k_proj, v_proj, o_proj
Epochs
1
Batch size
1
Gradient accumulation
4
Learning rate
2e-4
Precision
FP32
🔬 Before vs After Fine-Tuning
One of the main objectives of this project was to compare the base model with the fine-tuned model.
Before Fine-Tuning
The original Qwen model could generate general pharmaceutical-looking responses, but it could also:
misunderstand drug names
confuse medications
generate unsupported explanations
produce generic responses
After Fine-Tuning
The model became more aligned with the structure and terminology of the drug-interaction training data.
It learned patterns such as:
text
1User:
2Does Drug A interact with Drug B?
34Model:
5Interaction: <interaction category>
67<supporting pharmaceutical text>
However, the V1 model can still produce incorrect or mismatched evidence.
This is an important limitation of using fine-tuning alone for precise pharmaceutical knowledge retrieval.
⚠️ Limitations
This model is a learning/research prototype.
It should not be considered a reliable medical information system.
Known limitations include:
Small model size: 0.5B parameters
Limited V1 drug coverage
Limited training compute
Possible hallucinations
Possible incorrect drug-pair associations
Fine-tuning does not guarantee exact factual retrieval
Training data may contain inconsistencies
No retrieval/database verification layer
No clinical validation
No medical professional verification of generated responses
The model should therefore not be used for diagnosis, prescribing, dosage decisions, or clinical decision-making.
Always verify drug-interaction information using authoritative pharmaceutical references and qualified healthcare professionals.
💻 Quick Start
1. Install dependencies
pip install torch transformers peft
Depending on the environment, compatible versions of the Hugging Face ecosystem may also be required.
2. Download the base model
The model is based on:
Qwen/Qwen2.5-0.5B-Instruct
The LoRA repository contains the adapter rather than a complete copy of the base model.
1💊 You: Does Zolpidem Tartrate interact with Itraconazole?
23🤖 Assistant:
45Interaction: increase drug exposure.
67Itraconazole increased the exposure of zolpidem...
The LoRA adapter modifies the behavior of the base model toward patterns learned from the pharmaceutical training examples.
🧪 Why LoRA?
Full fine-tuning would require updating the entire model.
LoRA instead trains a relatively small number of additional parameters while keeping the original model largely frozen.
This makes it particularly useful for:
learning experiments
smaller compute environments
domain adaptation
rapid prototyping
parameter-efficient fine-tuning
🛣️ Future Work
Possible future versions could explore:
V2 — Larger Dataset
Increase drug coverage beyond the V1 top-100 selection.
V3 — Retrieval-Augmented Generation
Introduce a retrieval layer so that the model can retrieve the exact drug-interaction record instead of relying entirely on information encoded during fine-tuning.
This project was created as a hands-on exploration of LLMs, pharmaceutical datasets, data preprocessing, and parameter-efficient fine-tuning.
The project builds upon the capabilities of:
Qwen
Hugging Face Transformers
Hugging Face Datasets
PEFT / LoRA
⚖️ Disclaimer
Educational and research purposes only.
This model is not a medical device and has not been clinically validated.
Generated responses may be inaccurate, incomplete, or misleading. Do not use this model as a substitute for professional medical advice, prescribing information, official drug labels, or validated drug-interaction databases.
For real-world medical decisions, consult qualified healthcare professionals and authoritative pharmaceutical references.
👥 Contributors
This project was a collaborative effort combining pharmaceutical research and AI engineering.
