This is a LoRA (Low-Rank Adaptation) adapter for the OpenMeditron/Meditron3-8B model, fine-tuned for medical multiple-choice question answering with structured JSON output generation.

Model Details

Base Model

Model: OpenMeditron/Meditron3-8B
Architecture: Llama-based medical language model
Parameters: 8B parameters
Precision: BFloat16

LoRA Configuration

Rank (r): 64
Alpha: 128
Dropout: 0.1
Target Modules: v_proj, q_proj, o_proj, k_proj, down_proj, gate_proj, up_proj
Task Type: Causal Language Modeling

Training Details

Dataset

Source: asanchez75/medical_textbooks_mcq
Domain: Medical multiple-choice questions
Language: Primarily French medical content
Format: JSON-structured input/output pairs
Size: 1,481 examples (1,184 train, 148 validation, 149 test)

Training Configuration

Epochs: 3
Learning Rate: 2e-5
Batch Size: 4 (per device)
Gradient Accumulation: 4 steps
Effective Batch Size: 16
Sequence Length: 2048 tokens
Hardware: NVIDIA A100 SXM4 40GB

Performance

Final Test Loss: 0.7995
Training Time: ~18.5 minutes (1,107 seconds)
Memory Usage: 23.2GB peak (A100 40GB)
LoRA Memory Usage: 7.59GB additional for training

Usage

Loading the Adapter

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "OpenMeditron/Meditron3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("OpenMeditron/Meditron3-8B")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "YOUR_HF_USERNAME/REPO_NAME")

Inference Example

python

import json

# Input format (medical context text)
input_text = "L'hypertension artérielle essentielle est une maladie chronique caractérisée par une pression artérielle élevée. Le traitement de première intention comprend les modifications du mode de vie et les médicaments antihypertenseurs."

# Format prompt using the same structure as training
prompt_prefix = "<|user|>
Context:
"
prompt_suffix = "

Generate ONE valid multiple-choice question based strictly on the context above. Output ONLY the valid JSON object representing the question.
MCQ JSON:<|end|>
<|assistant|>
"

# Generate response
formatted_prompt = prompt_prefix + input_text + prompt_suffix
inputs = tokenizer(formatted_prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.0)
    
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

Expected Output Format

json

{
    "question": "Quel est le traitement de première intention de l'hypertension artérielle essentielle?",
    "options": {
        "A": "Inhibiteurs de l'ECA",
        "B": "Bêta-bloquants",
        "C": "Diurétiques thiazidiques",
        "D": "Antagonistes calciques"
    },
    "correct_answer": "A",
    "explanation": "Les inhibiteurs de l'ECA sont recommandés en première intention pour le traitement de l'hypertension artérielle essentielle selon les guidelines internationales."
}

Model Architecture

This adapter targets the following modules in the Meditron3-8B model:

Query projection (q_proj)
Key projection (k_proj)
Value projection (v_proj)
Output projection (o_proj)
Gate projection (gate_proj)
Up projection (up_proj)
Down projection (down_proj)

Limitations and Biases

Domain Specific: Optimized for French medical content
MCQ Format: Designed for structured multiple-choice questions
Medical Focus: Performance may vary on non-medical content
Language: Primarily trained on French medical terminology

Citation

If you use this model, please cite:

bibtex

@misc{meditron3-8b-mcq-lora,
    title={Meditron3-8B LoRA Adapter for Medical MCQ JSON Generation},
    author={Your Name},
    year={2025},
    publisher={Hugging Face},
    url={https://huggingface.co/YOUR_USERNAME/REPO_NAME}
}

License

This adapter is released under the Apache-2.0 license, consistent with the base Meditron3-8B model.

asanchez75/meditron3-8b-mcq-lora

Meditron3-8B LoRA Adapter for Medical MCQ JSON Generation