🧬 Clinical Trial Retrieval-Augmented Generation (RAG) 🔬
Retrieval-Augmented Generation (RAG) system for oncology clinical trial question answering using Gemma-3-4B-IT, FAISS, and Qwen3-Embedding-0.6B.
Table of Contents
INTRODUCTION
Clinical trial data is updated in a frequent and irregular manner, making it challenging for clinicians and researchers to stay current on
developing therapeutics. Trial records hosted on
ClinicalTrials.gov are updated in a continuous and
ad hoc manner, following no fixed schedule. While public, this data is not easily accessible by LLMs or search engines, and in combination
with the complex domain terminology, makes it easy to misrepresent this information. Furthermore, commonly used LLMs rely primarily on trained
information, leading to stale information being included in responses and an inability to answer with accurate information.
This leads to increased work for physicians and researchers who must manually read through trial records to be aware of scientific
advancements and inform the best clinical decisions. An automated tool designed to reduce the manual labor to monitor clinical trials
would save time for busy clinical teams and allow for more accessible trial enrollment. This efficiency matters in fast-paced areas
of research like oncology, where new treatments are showing up frequently, and patients are interested in being enrolled in clinical trials.
Here, I take a strong base model (
Gemma-3-4B-IT) and build out the retrieval side of the RAG using FAISS and Qwen3 embeddings to allow
the model to access updated trial information, without the need for retraining. RAG improved accuracy on clinical trial test set from 0.17
without retrieval to 0.89 with retrieval and outperformed two similar size comparison models on this task (0.50 and 0.26).
This code uses a subset of oncology clinical trials to show how in practice, an advanced system could be built out for instant and
widespread clinical trial information retrieval.
Key Result: Retrieval improved clinical trial question answering from 0.168 to 0.888, achieving Recall@3 = 0.975 on the held-out clinical trial evaluation set without retraining the base language model.
DATA
To build a document store, I downloaded 600 oncology clinical trials using keyword search via
API from
ClinicalTrials.gov. After deduplicating and reformatting the data into a Pandas data frame retaining the most
important fields, 410 unique trials remained. Gemini API was used to generate
161 question/answer pairs, tied
to 131 unique trials, meaning some trials have more than one associated question.
Since this is a RAG task, there is no direct test/train split; all 410 trials
were used for the document retrieval store, and the 161 question test set was held out for final RAG
evaluation. The 410 trials are in the
clintrials_documents.csv file and the 161 question/trial/answer sets
are in
clintrials_test.csv file.
During model building, a discrepancy was identified that initially the RAG did not have access
to the eligibility information in the trial, which has been fixed by directly re-embedding the eligibility
data. The Gemini API prompt and a sample test data point is shown below.
Gemini Prompt
1prompt = f"""
2 You are an oncology expert. Analyze these 200 clinical trials and generate exactly 20 realistic clinical questions.
3
4 Each question must be something a medical doctor or scientific researcher would actually ask (e.g., matching a patient based on a cancer type, trial phase, or specific eligibility rule).
5
6 Spread the questions out evenly across different trials in this specific list.
7
8 For each item in your JSON output, provide:
9 1. "target_id": The exact NCT ID or Trial Index.
10 2. "question": The complex clinical query.
11 3. "ground_truth_summary": An ideal 2-4 sentence summary answering the question based strictly on this trial.
12
13 CRITICAL SUMMARY REQUIREMENTS:
14 - The summary must include the drug or intervention, the disease or condition being studied, the clinical trial phase, the recruitment/trial status, and the primary objective of the study.
15 - Mention other important details if available and relevant, such as the study design, sponsor, or key eligibility criteria.
16 - Write in clear, professional language and strictly do not exceed 4 sentences.
17
18 Here are the trials:
19 {trials_text}
20"""
Example Evaluation Sample
| Target ID | Question | Ground Truth Summary |
|---|
| NCT00310063 | What are the eligibility criteria for young cancer patients receiving chemotherapy to participate in a study evaluating acupressure for preventing nausea and vomiting? | This study evaluates acupressure in preventing nausea and vomiting in young cancer patients receiving chemotherapy. Patients must be receiving in-patient primary oncology care at Brenner Children's Hospital and be on specific chemotherapy agents like alkylating agents or high-dose cytarabine. The primary objective is to assess the effectiveness of acupressure as a supportive care intervention. |
METHODOLOGY
Model Architecture
This RAG architecture was built using the baseline model
Gemma-3-4B-IT. I previously tested three
embedding and distance metric combinations of Qwen3-Embedding-0.6B and MiniLM-L6-v2 embeddings with cosine
and Euclidean distance metrics, and ultimately selected
Qwen3-Embedding-0.6B with cosine similarity based
on retrieval accuracy.
Prior to implementing RAG, the model's pre-document retrieval performance was assessed to gather a baseline.
Then, documents were split into
800-token chunks with 10% chunk overlap to reduce the likelihood of
the model losing relevant information split across document chunks. FAISS-based vector retrieval was implemented,
and model performance was measured with the top-
k retrieved documents added to the prompt. I tested both
k=3 and k=5 as retrieval depths with the initial thought that k=5 would improve retrieval accuracy,
however, it had no beneficial effects on datasets other than HotPotQA thus I chose k=3 for my final
implementation across all datasets.
Final configuration
- Generator: Gemma-3-4B-IT
- Embedding model: Qwen3-Embedding-0.6B
- Similarity metric: Cosine similarity
- Vector database: FAISS
- Chunk size: 800 tokens
- Chunk overlap: 10%
- Retrieval depth: k = 3
Experiments
Both document retrieval and answer accuracy were assessed, with answer accuracy being measured by an
F1-similar metric that compares the model generated responses to the ‘gold’ answer.
Additionally, I tracked refusal rates, where the model declines to answer or cites missing documents,
to compare where the model did not retrieve the true document (as shown by recall) to when the
model believed it did not retrieve the true document.
Two comparison models were evaluated,
Qwen2.5-3B-Instruct and
Phi-3.5-mini-instruct,
on their ability to answer the questions in the absence of document retrieval, to test whether
the RAG implementation improved performance relative to other strong base models of similar size.
Lastly, I tested my RAG architectures across three benchmarks to assess its ability to generalize
to other question/document/answer sets beyond the clinical trial task.
Note, the full implementation and evaluation pipeline is available as train_eval_pipeline.py in this repository.
To summarize, I conducted four experiments:
- Does RAG improve the performance of the baseline model to answer clinical trial specific questions?
- Does the RAG model answer clinical trials questions in comparison to other strong base models?
- Does the RAG model generalize well to other document/answer sets?
- How does the RAG model recognize the limits of its retrieval? (Recall vs Refusal)
EVALUATION
To evaluate model performance, the base model, RAG model, and comparison models were tested using the
clinical trial test data and three Hugging Face benchmark datasets:
Natural Questions,
HotPotQA, and
BioASQ.
Natural Questions consists of real user queries to Google and Wikipedia pages used to answer those questions,
intended to test general knowledge and information retrieval. HotPotQA is a benchmark requiring models to synthesize data across multiple documents by using multi-hop reasoning. Lastly, I chose a benchmark focused on retrieval and answer summarization in the biomedical domain using BioASQ to evaluate technical evidence-based responses. This dataset pairs questions with relevant clinical passages aimed to emulate PubMed abstracts.
Performance on test data was evaluated against two other base models,
Qwen2.5-3B-Instruct and
Phi-3.5-mini-instruct, based on being openly available instruction tune models
of a similar size.
The metrics used to evaluate model performance were Recall@3 for document retrieval and an F1-style metric
that computes precision and recall based on overlap between generated response and correct answer, filtering out
un-important ‘stop’ words. This F1 approach was chosen to balance penalizing overly verbose answers with providing
enough relevant information.
RAG implementation of Gemma-3-4B-IT shows significant improvement in accuracy over the base model performance.
Notably, the model had exceptional Recall@3 (0.975) and minimal refusals (2 out of 161 test questions).
This RAG architecture had moderate generalizability. It generalized well to Natural Questions, which had 0.54
answer accuracy, high recall of 0.956, and minimal refusals (84 out of 2474 test questions). The model failed to generalize well
to HotPotQA and BioASQ, likely due to the multi-hop reasoning and biomedical domain specificity required.
Furthermore, RAG significantly outperformed the other two pre-retrieval models on the clinical trial task.
Across datasets, the comparison models did perform interestingly well on benchmark tasks without document retrieval,
such as Qwen2.5-3B-Instruct scored 0.357 on BioASQ.
Accuracy by Dataset and Model
| Dataset | Your Model (RAG) | Base Model (Gemma-3-4B-IT, no RAG) | Qwen2.5-3B-Instruct | Phi-3.5-mini-instruct |
|---|
| Natural Questions | 0.540 | 0.256 | 0.123 | 0.230 |
| HotPotQA | 0.421 | 0.260 | 0.135 | 0.277 |
| BioASQ | 0.366 | 0.190 | 0.357 | 0.285 |
| Custom Clinical Test | 0.888 | 0.168 | 0.503 | 0.261 |
Retrieval Recall@3 (RAG Model Only)
| Dataset | Recall@3 |
|---|
| Natural Questions | 0.956 |
| HotPotQA | 0.653 |
| BioASQ | 0.283 |
| Custom Clinical Test | 0.975 |
Refusal Rate (RAG Model)
| Dataset | Total Refusals / Evaluated | Refusals Despite Correct Doc Retrieved |
|---|
| Natural Questions | 84 / 2474 (3.4%) | 48 |
| HotPotQA | 569 / 7405 (7.7%) | 380 |
| BioASQ | 236 / 4719 (5.0%) | 5 |
| Custom Clinical Test | 2 / 161 (1.2%) | 1 |
INTENDED USE AND USAGE
The intended use for this code and this architecture is for researchers or physicians who want quick answers about current clinical trials.
It is not meant to be used as medical advice, and all information should be used under the guidance of a clinical professional.
The code below loads the saved model + FAISS index and answers new questions in the same manner.
1import torch
2from huggingface_hub import hf_hub_download
3from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
4from langchain_community.vectorstores import FAISS
5from langchain_huggingface import HuggingFaceEmbeddings
6
7HF_id = "clairesull/onc_clintrial_RAG"
8
9# Load the generator/base model
10tokenizer = AutoTokenizer.from_pretrained(HF_id, subfolder="generator")
11tokenizer.padding_side = "left"
12tokenizer.pad_token_id = tokenizer.pad_token_id or tokenizer.eos_token_id
13model = AutoModelForCausalLM.from_pretrained(HF_id, subfolder="generator", device_map="auto", torch_dtype=torch.bfloat16)
14pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=250, do_sample=False, return_full_text=False)
15
16# Load the FAISS retrieval index for clinical trials
17# Alternatively, can load the FAISS index for benchmarks: faiss_nq, faiss_hotpotqa, faiss_bioasq
18hf_hub_download(repo_id=HF_id, filename="faiss_clinical_trials/index.faiss")
19faiss_dir = hf_hub_download(repo_id=HF_id, filename="faiss_clinical_trials/index.pkl").rsplit("/", 1)[0]
20embedding_model = HuggingFaceEmbeddings(model_name="Qwen/Qwen3-Embedding-0.6B")
21vector_db = FAISS.load_local(faiss_dir, embedding_model, allow_dangerous_deserialization=True)
22
23# Retrieve relevant trial documents and generate an answer
24question = "What are the eligibility criteria for a Phase 0 study involving Temsirolimus?"
25docs = vector_db.similarity_search(question, k=3)
26context = "\nExtracted documents:\n" + "".join(f"Document {i} (Source: {d.metadata.get('source', 'unknown')}):::\n{d.page_content}\n" for i, d in enumerate(docs))
27
28clinical_specific_instruction = (
29 "The documents above may describe different clinical trials. First, internally "
30 "identify which document (by Trial ID) is most relevant to the question. Then, "
31 "without restating which document you chose, answer the following question using "
32 "only that document with a 2-4 sentence summary. The summary must include the drug "
33 "or intervention, the disease or condition being studied, the clinical trial phase, "
34 "the recruitment/trial status, and the primary objective of the study."
35)
36
37raw_prompt = f"Context:\n{context}\n\n{clinical_specific_instruction}\nQuestion: {question}\nAnswer:"
38prompt = tokenizer.apply_chat_template([{"role": "user", "content": raw_prompt}], tokenize=False, add_generation_prompt=True)
39
40output = pipe(prompt)[0]["generated_text"].strip()
41print(output)
PROMPT FORMAT
For clinical trial data, inform the model to retrieve the most relevant document, then answer the question using only that document.
Tell the model to format a 2–4 sentence answer and include relevant pieces of information including the drug, the patient condition, phase of the trial, primary outcomes, etc.
For all other benchmarking datasets (Natural Questions HotPotQA and BioASQ), provide simple instructions to answer the question directly using the provided context.
1clinical_specific_instruction = (
2 "The documents above may describe different clinical trials. First, internally "
3 "identify which document (by Trial ID) is most relevant to the question. Then, "
4 "without restating which document you chose, answer the following question using "
5 "only that document with a 2-4 sentence summary. The summary must include the drug "
6 "or intervention, the disease or condition being studied, the clinical trial phase, "
7 "the recruitment/trial status, and the primary objective of the study. Where "
8 "relevant, mention the study design, sponsor, or key eligibility criteria."
9)
1general_instruction = (
2 "Answer the following question directly and concisely, strictly using the context above."
3)
OUTPUT FORMAT
The expected output is a 2–4 sentence response that summarizes the relevant clinical trial and answers the question.
The format should be a plain text summary. The response should mention things such as the therapeutic or intervention, the patient disease, the study phase or status.
Example Question
What are the age and health requirements for pediatric patients with Acute Lymphoblastic Leukemia to participate in a quality of life study during maintenance chemotherapy?
Example Response
Document 0 (NCT03132948) outlines the age and health requirements for pediatric patients with Acute Lymphoblastic Leukemia (ALL) participating in a quality of life study during maintenance chemotherapy. The study includes children aged 8-18 years who are able to speak and read English and have no contraindications to moderate physical exercise, as determined by their pediatric oncologist. Furthermore, participants must have parental consent and patient assent, and not have any documented psychiatric or neurological disorders that would interfere with study participation.
LIMITATIONS
-
Benchmark HotPotQA assesses multi-hop reasoning and requires connecting information from two separate documents, retrieval of 0.65 indicates the retrieval often missed one of the two documents needed to answer the question. A higher retrieval number of k=5 was shown to slightly improve performance, but not significantly. Failure to retrieve enough relevant documents limits the ability of the RAG to answer the question accurately.
-
BioASQ recall was very low (0.28), which inevitably impacted accuracy, however, accuracy is slightly retained (0.37), like the accuracy of the Qwen2.5-3B model (0.35) which had no document retrieval. This suggests the base models can answer these questions correctly without retrieval knowledge, reducing the impact of this benchmark in RAG evaluation.
-
Experiments were run to test different answer-level accuracy metrics: count of number of correct words over estimated accuracy, a stricter metric (BERTScore semantic matching) underestimated accuracy, the final F1 metrics was best option, but could still be improved by a LLM judge.
-
Mismatch in eligibility data, the Gemini API had access to the full eligibility for each trial to generate the ‘gold’ response whereas the RAG had only the trial summary, not necessarily including the eligibility. This was remedied by adding eligibility information into the embedded text, but the gold answer and model generated answer were not produced based on identical prompts.
