Gemma 3 4B — Bilingual (Amharic + English) Medical QA
Fine-tuned LoRA adapters on top of google/gemma-3-4b-it for medical
question-answering in Amharic and English. This is the lightweight
counterpart to the 12B model, trained on the same bilingual medical QA dataset
and intended for lower-cost inference.
⚠️ Medical disclaimer. This model is a research prototype. Its outputs are
AI-generated and may be incomplete or incorrect. It is not a medical device
and must not be used for diagnosis or treatment decisions. Always consult a
qualified healthcare professional. Outputs should be reviewed by a clinician
before any downstream use.
Model details
Base model
google/gemma-3-4b-it
Method
QLoRA (4-bit NF4) supervised fine-tuning
Adapter type
LoRA (PEFT)
Languages
Amharic (am), English (en)
Domain
Medical / health question answering
License
Gemma (inherits base model terms)
Developed by
Walelign Tewabe Sewunetie (PhD) & Surafel L. Tilahun (PhD)
Trained on a bilingual medical QA dataset with parallel English and Amharic
question–answer pairs. Each source row contributes two training examples
(one English pair, one Amharic pair).
Dataset provenance: content compiled from public health websites/articles,
medical textbooks and guidelines, expert/clinician-written material, and an
existing medical QA dataset. Amharic produced by professional translation.
Medical content reviewed by a qualified health professional. See the dataset
card for full details.
Training results
Training loss trajectory
Training loss dropped sharply and continued to decline through training:
Progress
Training loss
Token accuracy
Start (epoch ~0.03)
7.24
0.345
Epoch ~2.92
0.896
0.762
Epoch ~2.95
0.836
0.778
Epoch ~2.99
0.902
0.762
Per-epoch eval progress
Epoch
Eval loss
Eval token accuracy
1
1.4533
0.6455
2
1.3772
0.6670
3
1.4402
0.6672
Eval loss reached its minimum at epoch 2 (1.3772) and rose slightly at
epoch 3 (1.4402), while training loss continued to fall — indicating the onset
of mild overfitting in the final epoch. The end-of-epoch-2 checkpoint
(checkpoint-576) may generalize slightly better than the final checkpoint.
The final adapter at the repo root corresponds to epoch 3 (checkpoint-864);
users who prefer the best-validation checkpoint can load checkpoint-576.
Comparison with the 12B model
For reference, the larger Gemma 3 12B version
trained on the same data reached a best eval loss of 1.224 and accuracy 0.696.
This 4B model trades some quality for substantially smaller size and lower
inference cost.
Best-epoch metric
This model (4B)
12B version
Eval loss
1.377
1.224
Eval token accuracy
0.667
0.696
Checkpoints
Checkpoint
Corresponds to
checkpoint-576
End of epoch 2 (best eval)
checkpoint-864
End of epoch 3 (final — repo root)
How to use
These are LoRA adapters — load them on top of the base model.
Training data was clinician-reviewed, but model outputs are not. The
dataset was reviewed by a health professional; the model can still generate
answers that differ from its training data, so its generated outputs have not
been individually validated and may be incorrect.
Smaller model, lower ceiling. At 4B parameters this model is less capable
than the 12B version; expect somewhat weaker reasoning and fluency, especially
on complex or rare medical questions.
Domain-narrow. Medical knowledge comes primarily from the small
fine-tuning dataset (~2,300 pairs).
Treatment/dosage caution. Answers about treatment, medication, or
pregnancy should be treated with particular caution.
Inherits base-model limitations and biases from google/gemma-3-4b-it.
Intended use
Research and educational exploration of bilingual (Amharic/English) medical QA,
and lower-cost deployment where the 12B model is too expensive to serve. Not for
clinical, diagnostic, or treatment use.
License
Derived from google/gemma-3-4b-it and subject to the
Gemma Terms of Use.
Citation
bibtex
1@misc{sewunetie_tilahun_gemma3_4b_amharic_medqa,
2 title = {Gemma 3 4B Bilingual Amharic-English Medical QA (LoRA)},
3 author = {Sewunetie, Walelign Tewabe and Tilahun, Surafel L.},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/Walelign/Gemma_3_4B}}
6}
