You can deploy this model and start earning money today!

Gaykar/
Phi2-drug_data

Name: Phi2-drug_data
Rating: 2.5 (24 reviews)

Views

No views yet

Model Card for Model ID

This model is a LoRA-based fine-tuned variant of Microsoft Phi-2, designed to generate concise, medical-style textual descriptions of drugs. Given a drug name as input, the model produces a short, single-paragraph description following an instruction-style prompt format.

The training pipeline consists of two stages:

Continued Pretraining (CPT) on domain-relevant medical and pharmaceutical text to adapt the base model to the language and terminology of the domain.

Supervised Fine-Tuning (SFT) using structured drug name–description pairs to guide the model toward consistent formatting and domain-specific writing style.

This model is intended strictly for educational and research purposes and must not be used for real-world medical, clinical, or decision-making applications.

Model Details

Model Description

This model is a parameter-efficient fine-tuned version of the Microsoft Phi-2 language model, adapted to generate concise medical drug descriptions from drug names. The training pipeline consists of two stages:

Continued Pretraining (CPT) to adapt the base model to drug and medical terminology.
Supervised Fine-Tuning (SFT) using instruction-style input–output pairs.

LoRA adapters were used during fine-tuning to reduce memory usage and training cost while preserving base model knowledge.

Developed by: Atharva Gaykar
Funded by: Not applicable
Shared by: Atharva Gaykar
Model type: Causal Language Model (LoRA-adapted)
Language(s) (NLP): English
License: CC-BY-NC 4.0
Finetuned from model: microsoft/phi-2

Uses

This model is designed to generate concise medical-style descriptions of drugs given their names.

Direct Use

Educational demonstrations of instruction-following language models
Academic research on medical-domain adaptation
Experimentation with CPT + SFT pipelines
Studying hallucination behavior in domain-specific LLMs

The model should only be used in non-production, educational, or research settings.

Out-of-Scope Use

This model is not designed or validated for:

Medical diagnosis or treatment planning
Clinical decision support systems
Dosage recommendations or prescribing guidance
Patient-facing healthcare applications
Professional medical, pharmaceutical, or regulatory use
Any real-world deployment where incorrect medical information could cause harm

Bias, Risks, and Limitations

This model was developed solely for educational purposes and must not be used in real-world medical or clinical decision-making.

Known Limitations

May hallucinate incorrect drug indications or mechanisms
Generated descriptions may be incomplete or outdated
Does not verify outputs against authoritative medical sources
Does not understand patient context, dosage, or drug interactions
Output quality is sensitive to prompt phrasing

Risks

Misinterpretation of outputs as medical advice
Overconfidence in fluent but inaccurate responses
Potential propagation of misinformation if misused

Recommendations

Always verify outputs using trusted medical references
Use only in controlled, non-production environments
Clearly disclose limitations in any downstream use
Avoid deployment in safety-critical or healthcare systems

How to Get Started with the Model

This repository contains LoRA adapter weights, not a full model.

Example usage (conceptual):

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2")
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Gaykar/Phi2-drug_data")

model.eval()


import torch

# Drug to evaluate
drug_name = "Paracetamol"

# Build evaluation prompt
eval_prompt = (
    "Generate exactly ONE sentence describing the drug.\n"
    "Do not include headings or extra information.\n\n"
    f"Drug Name: {drug_name}\n"
    "Description:"
)

# Tokenize prompt
model_input = tokenizer(
    eval_prompt,
    return_tensors="pt"
).to(model.device)

# Generate output (greedy decoding)
with torch.no_grad():
    output = model.generate(
        **model_input,
        do_sample=False,
        num_beams=1,             # Greedy decoding (This decision is critical for this model because it operates in the medical domain, where factual consistency and determinism are more important than linguistic diversity.)
        max_new_tokens=120,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.eos_token_id
    )

# Remove prompt tokens
prompt_length = model_input["input_ids"].shape[1]
generated_tokens = output[0][prompt_length:]

# Decode generated text only
generated_text = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True
).strip()

# Enforce single-sentence output
if "." in generated_text:
    generated_text = generated_text.split(".")[0] + "."

print(" DRUG NAME:", drug_name)
print(" MODEL GENERATED DESCRIPTION:")
print(generated_text)

#Example output
DRUG NAME (EVAL): Paracetamol

MODEL GENERATED DESCRIPTION:
Paracetamol (acetaminophen) is a non-narcotic analgesic and antipyretic used to relieve mild to moderate pain and reduce fever.

Training Details

Training Data

Dataset: Gaykar/DrugData
Structured drug name–description pairs
Used for both CPT (domain adaptation) and SFT (instruction following)

Training Procedure

Continued Pretraining (CPT)

The base model was further trained on domain-relevant medical and drug-related text to improve familiarity with terminology and style. CPT focused on next-token prediction without instruction formatting.

Supervised Fine-Tuning (SFT)

After CPT, the model was fine-tuned using instruction-style prompts to generate concise medical descriptions from drug names.

Training Hyperparameters

CPT Hyperparameters

Hyperparameter	Value
Batch size (per device)	1
Effective batch size	8
Epochs	4
Learning rate	2e-4
Precision	FP16
Optimizer	Paged AdamW (8-bit)
Logging steps	10
Checkpoint saving	Every 500 steps
Checkpoint limit	2

SFT Hyperparameters

Hyperparameter	Value
Batch size (per device)	4
Gradient accumulation	1
Effective batch size	4
Epochs	5
Learning rate	2e-5
LR scheduler	Linear
Warmup ratio	6%
Weight decay	1e-4
Max gradient norm	1.0
Precision	FP16
Optimizer	Paged AdamW (8-bit)
Checkpoint saving	Every 50 steps
Checkpoint limit	2
Experiment tracking	Weights & Biases

Evaluation

Testing Data

Drug names sampled from the same dataset were used for evaluation. Outputs were assessed for factual correctness using an external LLM-based evaluation approach.

Metrics

Evaluation Method: LLM-as-a-Judge (Chatgpt -Web seacrch available. )

Binary classification: Factually Correct / Hallucinated
Three evaluation batches

Results

Batch 1

Category	Count	Percentage
Total Drugs Evaluated	25	100%
Factually Correct	24	96%
Hallucinated / Failed	1	4%

Batch 2

Category	Count	Percentage
Total Drugs Evaluated	25	100%
Factually Correct	22	88%
Hallucinated / Failed	3	12%

Batch 3

Category	Count	Percentage
Total Drugs Evaluated	22	100%
Factually Correct	15	68%
Hallucinated / Failed	0	0%

Summary

Since this model was fine-tuned (SFT+CPT) using LoRA rather than full-parameter fine-tuning, eliminating hallucinations entirely is challenging. While LoRA enables efficient training and strong instruction-following behavior, it does not fully overwrite the base model’s internal knowledge. Despite this limitation, the model performs well for educational and research-oriented drug description generation tasks.

Environmental Impact

Hardware Type: NVIDIA T4 GPU
Hours used: Not recorded
Cloud Provider: Google Colab
Compute Region: Not specified
Carbon Emitted: Not estimated

Technical Specifications

Model Architecture and Objective

Base model: Microsoft Phi-2
Objective: Instruction-following text generation
Adaptation method: LoRA (PEFT)

Compute Infrastructure

Hardware

NVIDIA T4 GPU

Software

Transformers
PEFT
PyTorch

Model Card Contact

Atharva Gaykar

Framework Versions

PEFT 0.18.0