Model	Accuracy	Notes
Mistral 7B Instruct v0.2 (zero-shot)	44.68%	Evaluated on MedMCQA validation set (4,183 examples)
MedLlama-India-70B	~74% (AutoScientist medical win rate)	Llama 3.3 70B fine-tuned

Benchmark: openlifescienceai/medmcqa validation set — 4,183 questions across 21 medical subjects.

Training Details

Parameter	Value
Base model	meta-llama/Llama-3.3-70B-Instruct
Method	LoRA (PEFT)
LoRA rank	64
LoRA alpha	128
Target modules	all-linear
Epochs	3
LR scheduler	cosine
Warmup ratio	0.05
Gradient clipping	1
Training platform	Adaption Labs AutoScientist
Data platform	Adaption Labs Adaptive Data
Training examples	38,000+
Source dataset	openlifescienceai/medmcqa

AutoScientist Training Metrics

Win rate: 20 → 80 (+300%)
Medical win rate: 26 → 74 (+185%)
Final training loss: 0.73
Training steps: 624
Loss curve: clean convergence, no overfitting

Adaptive Data Processing

Source: MedMCQA (170,000 questions)
Quality grade: B → A (28.9 → 57.7 percentile)
Recipes: Reasoning Traces, Hallucination Mitigation, Prompt Rephrase, Metadata Injection, House Special
Localization: India (English)
Blueprint: AIIMS/NEET-PG medical examiner persona

Dataset

xenkrypt/MedLlama-India-Dataset

38,000+ instruction-formatted NEET-PG and AIIMS-PG questions processed via Adaption Labs Adaptive Data with reasoning traces, hallucination mitigation, and Indian medical context localization.

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("xenkrypt/MedLlama-India-70B")
model = PeftModel.from_pretrained(base, "xenkrypt/MedLlama-India-70B")

prompt = """### Instruction:
You are a medical expert for AIIMS/NEET-PG examinations.
Answer this multiple choice question.

Question: Most common cause of mitral stenosis?
A) Rheumatic fever
B) Infective endocarditis
C) Congenital
D) SLE

### Response:
The correct answer is"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Limitations

LoRA adapter — requires base model meta-llama/Llama-3.3-70B-Instruct
Trained on exam MCQs — validate before clinical use
Not a substitute for professional medical advice

Demo

https://huggingface.co/spaces/xenkrypt/MedLlama-India-Demo

Submitted To

Adaption Labs AutoScientist Hackathon — Healthcare, Part 1, July 2025