SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
1from sentence_transformers import SentenceTransformer
23# Download from the 🤗 Hub4model = SentenceTransformer("Gaykar/all-MiniLM-L6-medical-rag")5# Run inference6queries =[7"What is (are) multiple sulfatase deficiency ?",8]9documents =[10'Multiple sulfatase deficiency is a condition that mainly affects the brain, skin, and skeleton. Because the signs and symptoms of multiple sulfatase deficiency vary widely, researchers have split the condition into three types: neonatal, late-infantile, and juvenile. The neonatal type is the most severe form, with signs and symptoms appearing soon after birth. Affected individuals have deterioration of tissue in the nervous system (leukodystrophy), which can contribute to movement problems, seizures, developmental delay, and slow growth. They also have dry, scaly skin (ichthyosis) and excess hair growth (hypertrichosis). Skeletal abnormalities can include abnormal side-to-side curvature of the spine (scoliosis), joint stiffness, and dysostosis multiplex, which refers to a specific pattern of skeletal abnormalities seen on x-ray. Individuals with the neonatal type typically have facial features that can be described as "coarse." Affected individuals may also have hearing loss, heart malformations, and an enlarged liver and spleen (hepatosplenomegaly). Many of the signs and symptoms of neonatal multiple sulfatase deficiency worsen over time. The late-infantile type is the most common form of multiple sulfatase deficiency. It is characterized by normal cognitive development in early childhood followed by a progressive loss of mental abilities and movement (psychomotor regression) due to leukodystrophy or other brain abnormalities. Individuals with this form of the condition do not have as many features as those with the neonatal type, but they often have ichthyosis, skeletal abnormalities, and coarse facial features. The juvenile type is the rarest form of multiple sulfatase deficiency. Signs and symptoms of the juvenile type appear in mid- to late childhood. Affected individuals have normal early cognitive development but then experience psychomotor regression; however, the regression in the juvenile type usually occurs at a slower rate than in the late-infantile type. Ichthyosis is also common in the juvenile type of multiple sulfatase deficiency. Life expectancy is shortened in individuals with all types of multiple sulfatase deficiency. Typically, affected individuals survive only a few years after the signs and symptoms of the condition appear, but life expectancy varies depending on the severity of the condition and how quickly the neurological problems worsen.',11'There is no cure for OPCA. The disorder is slowly progressive with death usually occurring approximately 20 years after onset.',12'Spinal cord infarction is a stroke either within the spinal cord or the arteries that supply it. It is caused by arteriosclerosis or a thickening or closing of the major arteries to the spinal cord. Frequently spinal cord infarction is caused by a specific form of arteriosclerosis called atheromatosis, in which a deposit or accumulation of lipid-containing matter forms within the arteries. Symptoms, which generally appear within minutes or a few hours of the infarction, may include intermittent sharp or burning back pain, aching pain down through the legs, weakness in the legs, paralysis, loss of deep tendon reflexes, loss of pain and temperature sensation, and incontinence.',13]14query_embeddings = model.encode_query(queries)15document_embeddings = model.encode_document(documents)16print(query_embeddings.shape, document_embeddings.shape)17# [1, 384] [3, 384]1819# Get the similarity scores for the embeddings20similarities = model.similarity(query_embeddings, document_embeddings)21print(similarities)22# tensor([[0.7917, 0.0896, 0.0186]])
The image below shows the difference between base model and fine tuned model:-
image
If a base model gives a 0.80 to a correct answer and a 0.75 to a wrong one, the retriever might easily get confused by a small amount of noise. In our fine-tuned model, if the correct answer stays at 0.80 but the wrong ones drop to 0.20, you have created a massive Discriminative Gap. This ensures:
Robustness: Even if a "negative" answer shares similar keywords, the model now knows they aren't semantically related to that specific question.
Cleaner RAG: Your LLM receives exactly the right context without "distractor" chunks that could cause hallucinations.
Training Details
Training Dataset
📊 Dataset Creation Pipeline
This dataset was created purely for academic and learning purposes to demonstrate skills in data collection, preprocessing, and LLM-based data generation within the medical NLP domain.
⚠️ No proprietary or copyrighted text is redistributed in raw form.
The overall pipeline consists of two stages:
1️⃣ Data Collection (Source Material)
Medical information related to brain tumors and human health was gathered from openly accessible educational and public medical resources. These sources were used only as intermediate context to generate derived question–answer pairs.
Sources Used
Public medical websites
Example: Mayo Clinic
Used only to understand structure and terminology (no verbatim content stored)
~6000 samples (already structured as question–answer pairs)
📌 Important Note
No textbook or website content is stored, shared, or redistributed in original form.
All source material was used only as temporary input to generate transformed outputs.
2️⃣ Data Formatting & Generation
To convert unstructured medical text into structured data, an LLM-assisted pipeline was implemented using LangChain and the Groq API.
Workflow
Extracted medical text chunks from websites and PDFs
Passed extracted text as context to an LLM
Prompted the LLM to generate high-quality question–answer pairs
Discarded:
Non-medical content
Questions without valid answers
Low-information or irrelevant text
Stored only the generated Q&A pairs in JSON format
Prompt Design
python
1prompt = PromptTemplate(2 input_variables=["page_content"],3 template="""
4You are a medical AI assistant.
56Given the following medical text, generate high-quality question and answer pairs.
7Ignore any non-medical information.
8IMP: Ignore if the data only contains questions without answers. Do not generate questions from such data.
910Rules:
11- Use ONLY the provided content
12- Ignore sentences that do not contain meaningful medical information
13- Do NOT hallucinate
14- If no useful information exists, return an empty list
15- Output MUST be valid JSON only
1617Output format:
18[
19 {
20 "question": "...",
21 "answer": "..."
22 }
23]
2425Medical Text:
26{page_content}
27"""28)
📦 Final Dataset Format
The final dataset contains only synthesized question–answer pairs, structured as:
json
1{2"question":"What is a pituitary tumor?",3"answer":"A pituitary tumor is an abnormal growth in the pituitary gland that can affect hormone production."4}
No raw source text
No copyrighted paragraphs
Fully transformed content
🎓 Intended Use
This project is intended to:
Demonstrate data extraction and preprocessing skills
Showcase LLM-assisted dataset generation
Support academic research and experimentation
Enable model fine-tuning and evaluation
❌ Not intended for:
Commercial redistribution
Reproducing copyrighted material
Clinical or diagnostic use
⚖️ Ethical & Legal Considerations
All source materials are either open-access or used under educational fair use
The dataset contains only derived, non-verbatim content
This repository does not claim ownership over original source materials
If any content is found to violate usage policies, it will be removed immediately
image
Unnamed Dataset
Size: 6,460 training samples
Columns: question and answer
Approximate statistics based on the first 1000 samples:
question
answer
type
string
string
details
min: 6 tokens
mean: 14.62 tokens
max: 43 tokens
min: 3 tokens
mean: 156.11 tokens
max: 256 tokens
Samples:
question
answer
What type of brain tumors are children likely to have?
Primary brain tumors
What is (are) Non 24 hour sleep wake disorder ?
Non 24 hour sleep wake disorder refers to a steady pattern of one- to two-hour delays in sleep onset and wake times in people with normal living conditions. This occurs because the period of the person's sleep-wake cycle is longer than 24 hours. The condition most commonly affects people who are blind, due to an impaired sense of light-dark cycles. Non 24 hour sleep wake disorder can also affect sighted people. The cause of the disorder in these cases is incompletely understood, but studies suggest melatonin levels play a role.
Name two common symptoms of diphtheria.
Slight fever and sore throat, and the development of a tough membrane in the throat.
1@inproceedings{reimers-2019-sentence-bert,
2 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
3 author = "Reimers, Nils and Gurevych, Iryna",
4 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
5 month = "11",
6 year = "2019",
7 publisher = "Association for Computational Linguistics",
8 url = "https://arxiv.org/abs/1908.10084",
9}
MultipleNegativesRankingLoss
bibtex
1@misc{henderson2017efficient,
2 title={Efficient Natural Language Response Suggestion for Smart Reply},
3 author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
4 year={2017},
5 eprint={1705.00652},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
