The model was trained as a causal language model on cleaned pharma-domain text extracted from a PDF corpus. Its objective is next-token prediction in the pharmaceutical domain.

Training pipeline

PDF text extraction
Text cleaning and normalization
Paragraph dataset creation
Hugging Face dataset conversion
Tokenization
Token packing into 512-token blocks
QLoRA fine-tuning
Adapter saving

Intended use

This adapter is intended for:

pharmaceutical domain adaptation
text continuation
downstream instruction tuning
educational and research experiments

Limitations

Not instruction fine-tuned
Not a chatbot
Not medically validated
Not for clinical decision-making

How to load

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
adapter_id = "ssuvetha/pharma-tinyllama-domain-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()