Qwen3-4B MedQuAD QLoRA
This repository contains an experimental QLoRA adapter fine-tuned from Qwen/Qwen3-4B using the MedQuAD medical question-answering dataset.
The goal of this project is to explore parameter-efficient fine-tuning of Qwen3-4B for medical question answering using Hugging Face Transformers, PEFT, TRL, and bitsandbytes.
Important: This is an experimental research model. It is not intended for diagnosis, treatment decisions, or clinical use.
Model Details
- Base model: Qwen/Qwen3-4B
- Fine-tuning method: QLoRA / PEFT
- Quantization: 4-bit NF4
- Double quantization: Enabled
- LoRA rank (r): 8
- LoRA alpha: 16
- LoRA dropout: 0.05
- Target modules: All linear layers
- Training framework: Hugging Face TRL SFTTrainer
- Hardware: NVIDIA Tesla T4 (~15 GB VRAM)
This repository contains a LoRA adapter, not a fully merged standalone Qwen3-4B model.
Training Data
The model was fine-tuned using the MedQuAD dataset:
Dataset: Hmehdi515/MedQuad
For this initial experiment, a small subset of 96 training examples was used.
Training examples were filtered to:
- contain a non-empty medical question
- contain an answer between approximately 20 and 160 words
The examples were converted to the Qwen3 chat format using:
- System message
- User medical question
- Assistant reference answer
Thinking mode was disabled during formatting and generation.
System Prompt
The initial experiment used the following instruction:
Answer the medical question clearly and concisely. When a question requires personal medical advice, recommend consulting a qualified clinician.
Training Configuration
| Parameter | Value |
|---|
| Base model | Qwen/Qwen3-4B |
| Training examples | 96 |
| Epochs | 2 |
| Learning rate | 2e-4 |
| Batch size | 1 |
| Gradient accumulation | 8 |
| Maximum sequence length | 384 |
| LoRA rank | 8 |
| LoRA alpha | 16 |
| LoRA dropout | 0.05 |
| Quantization | 4-bit NF4 |
| Double quantization | Yes |
| Gradient checkpointing | Yes |
| GPU | NVIDIA Tesla T4 |
The observed average training loss was approximately:
1.72
Evaluation
The fine-tuned adapter was evaluated on 8 examples selected from the MedQuAD validation split.
Metric
A token-level F1 score was calculated between the generated answer and the MedQuAD reference answer.
Current Result
Average Token-F1: 28.2
Individual evaluation results varied significantly.
Some common medical topics produced reasonable answers, while several rare or highly specific conditions resulted in inaccurate or hallucinated information.
For example, the model produced incorrect medical information when answering a question about Singleton-Merten syndrome.
Important Evaluation Limitation
Token-F1 measures lexical overlap between the generated and reference answers.
It does not measure:
- clinical correctness
- factual accuracy
- hallucination rate
- diagnostic accuracy
- treatment safety
Therefore, the Token-F1 score should not be interpreted as evidence that the model is medically reliable.
Intended Use
This adapter is intended for:
- QLoRA experimentation
- PEFT experimentation
- learning Hugging Face fine-tuning workflows
- medical QA research prototypes
- comparing base and fine-tuned language models
- studying hallucination behavior in medical QA
Limitations
This is an early experimental fine-tune trained on only 96 MedQuAD examples.
The model may:
- hallucinate medical information
- generate incorrect symptoms
- generate incorrect prevalence statistics
- provide inaccurate treatment information
- confuse similar diseases
- generate incorrect inheritance patterns
- fail to answer the specific type of question asked
- produce plausible-sounding but factually incorrect statements
The underlying Qwen3-4B model already contains general medical knowledge. Small-scale fine-tuning does not guarantee that this knowledge becomes more accurate and may potentially degrade performance on some questions.
A direct comparison against the original Qwen3-4B model is planned.
Safety
This adapter must not be used as a substitute for a physician or other qualified healthcare professional.
Do not use this model for:
- medical diagnosis
- treatment selection
- medication decisions
- emergency medical decisions
- clinical decision support without independent validation
Medical information generated by the model should be independently verified using authoritative medical sources.
Future Work
Planned experiments include:
- Benchmarking the original Qwen3-4B model on the same validation questions.
- Comparing base Qwen3-4B against the QLoRA adapter.
- Increasing training data from 96 examples to several thousand examples.
- Improving sampling across different medical question types.
- Improving the system prompt to reduce unsupported claims.
- Evaluating factual accuracy in addition to Token-F1.
- Measuring hallucination rates.
- Adding retrieval-augmented generation (RAG) using trusted medical sources.
- Evaluating the model on a larger held-out test set.
Experimental Status
Version: v1
This version should be considered a proof-of-concept QLoRA experiment rather than a production medical model.
Current benchmark:
| Model | Token-F1 |
|---|
| Qwen3-4B Base | Not tested yet |
| Qwen3-4B + MedQuAD QLoRA v1 | 28.2 |
The base-model benchmark will be added after evaluation on the same validation examples.
Disclaimer
This repository is provided for research and educational purposes only.
The model can produce false or misleading medical information.
It is not a medical device and should not be relied upon for diagnosis, treatment, or other healthcare decisions.
