Fine-tuned Llama 3.1 8B on Clinical Reasoning Dataset using research-optimized hyperparameters based on Med42 clinical LLM study and "LoRA Without Regret" findings.

Dataset

Source: Clinical Reasoning Data (crd_medprompt_unsloth.json)
Examples: 10,081
Format: Structured reasoning with "Key facts → Mechanism → Why-not alternatives → "
Avg Tokens: ~700 per example

Research-Optimized Configuration

Parameter	Value	Research Basis
Learning Rate	0.0001	Med42: 1e-4 optimal for clinical
LoRA Rank	64	Higher rank for complex patterns
LoRA Alpha	128	2× rank standard
LoRA Dropout	0.05	Regularization for structured output
Target Modules	All (MLP + Attention)	MLP critical for reasoning
Epochs	3	Structured formats need more exposure
Batch Size	16	Research-optimal for LoRA
Warmup	5%	Stability for complex learning
RSLoRA	Enabled	Rank-stabilized for r≥64

Training Results

Training Time: 2.66 hours
Final Loss: 0.5773

Part of Experiment

Testing hypothesis: Clinical reasoning data improves general reasoning capabilities.

kinzakhan1/CRD_V7 - 100% Clinical (this model)
kinzakhan1/SRD_V7 - 100% Standard reasoning
kinzakhan1/MIXED_V7 - 50/50 mix

kinzakhan1/CRD_V7

CRD_V7 - Clinical Reasoning Model (Research-Optimized)

Overview

Dataset

Research-Optimized Configuration

Training Results

Part of Experiment

kinzakhan1/
CRD_V7