Views
No views yet
TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T on a small pharma-domain corpus,
using QLoRA (4-bit, nf4), then merging the LoRA adapter back into the base weights.1TinyLlama-1.1B (base)
2 ↓ Stage 1: Non-Instruction FT (this model)
3pharma-tinyllama-non-instruction-merged ← you are here
4 ↓ Stage 2: Instruction FT (SFT)
5pharma-tinyllama-instruction-merged
6 ↓ Stage 3: Preference Tuning (DPO)
7pharma-tinyllama-dpo-merged| Parameter | Value |
|---|---|
| Base model | TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T |
| Method | QLoRA (4-bit nf4, double quant) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Target modules | q/k/v/o_proj, gate/up/down_proj |
| Block size | 512 tokens |
| Epochs | 3 |
| Batch size | 1 (grad accum 8, effective = 8) |
| Learning rate | 2e-4 |
| Warmup ratio | 0.03 |
| Weight decay | 0.01 |
| Optimizer | paged_adamw_32bit |
| Environment | Google Colab T4 GPU |
SivaSai8143/pharma-finetuning-data (config: raw).| Artifact | Link |
|---|---|
| Stage 1 LoRA adapter | pharma-tinyllama-non-instruction-lora-adapter |
| Stage 2 merged model | pharma-tinyllama-instruction-merged |
| Stage 3 merged model | pharma-tinyllama-dpo-merged |
| Training notebook | llm-finetuning-playbook |
| Dataset | pharma-finetuning-data |
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3model_id = "SivaSai8143/pharma-tinyllama-non-instruction-merged"
4tokenizer = AutoTokenizer.from_pretrained(model_id)
5model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
6
7prompt = "Metformin is a widely used antihyperglycemic agent that works by"
8inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
9outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7)
10print(tokenizer.decode(outputs[0], skip_special_tokens=True))Note: This is a continued-pretraining model — it generates text continuations, not structured Q&A responses. For instruction-following behaviour, use the Stage 2 or Stage 3 models.