Views
No views yet
meta-llama/Llama-3.1-8B-Instruct, trained on the MedQuAD (Medical Question Answering Dataset) using QLoRA for factual and educational biomedical question answering.| Setting | Value |
|---|---|
| Base model | meta-llama/Llama-3.1-8B-Instruct |
| Fine-tuning method | QLoRA (4-bit NF4 quantization) |
| LoRA rank / alpha | r = 8 / α = 16 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Dropout | 0.1 |
| Precision | FP16 mixed |
| Batch size (effective) | 8 (2 × grad accum = 4) |
| Learning rate | 2e-4 |
| Optimizer | AdamW (β₁ = 0.9, β₂ = 0.95) |
| Scheduler | Linear decay, warmup = 3 % |
| Max sequence length | 1024 |
| Epochs | 2 |
| GPU | NVIDIA T4 (15 GB VRAM) |
| Training time | ≈ 8 hours |
| Final loss (train/val) | 0.76 / 0.75 |