You can deploy this model and start earning money today!

mlai-dante/
medical-llm

Name: medical-llm
Availability: InStock

safetensors

Views

No views yet

Medical LLM - Fine-tuned Medical LLM

A specialized language model fine-tuned on medical terminology to provide accurate and contextual explanations of medical terms. Built on Llama 3.1 8B using LoRA (Low-Rank Adaptation) for efficient training.

Project Overview

This project fine-tunes Meta's Llama 3.1 8B model to understand and explain medical terminology from the Wiki Medical Terms dataset. The model is trained to take a medical term as input and generate a comprehensive explanation of its meaning.

Example:

Input: "The medical term is: Urodynia\nThe meaning of the term is:"
Output: "Urodynia is a painful condition that affects the urinary system..."

Dataset

Source: gamino/wiki_medical_terms
Type: Medical terminology with definitions
Training Strategy: Custom tokenization with weighted loss focusing on output generation

Technical Stack

Base Model: Meta Llama 3.1 8B
Training Framework: Tinker (Efficient LoRA training)
Fine-tuning Method: LoRA (Low-Rank Adaptation)
- Rank: 32
- Alpha: 32
- Target Modules: all-linear
- Dropout: 0
Evaluation: LLM-as-a-Judge using OpenAI GPT models
Model Hub: Hugging Face (mlai-dante/medical-llm)

Project Structure

medical-llm/
├── checkpoints/          # Saved model checkpoints (LoRA adapters)
├── scripts/              # Training and utility scripts
├── dataset.ipynb         # Main notebook with training and evaluation
├── figureMetrics.png     # Evaluation metrics visualization
└── README.md            # Project documentation

Getting Started

Prerequisites

bash

pip install tinker
pip install datasets
pip install huggingface-hub
pip install openai
pip install python-dotenv
pip install pandas numpy matplotlib

Environment Setup

Create a .env file with your API keys:

env

TINKER_API_KEY=your_tinker_api_key
OPENAPI_KEY=your_openai_api_key

Training

The training process includes:

Data Loading & Preprocessing
- Load the wiki_medical_terms dataset
- Custom tokenization with weighted loss
- Input prompts are weighted at 0, outputs at 1
Training Loop
- 20 training iterations
- Adam optimizer with learning rate: 1e-4
- Cross-entropy loss
- Loss decreases from ~0.84 to ~0.0003
Checkpoint Saving
- Models are saved as LoRA adapters
- Compatible with Hugging Face PEFT library

Performance

The model shows strong convergence:

Initial Loss: 0.839
Final Loss: 0.0003 (after 20 iterations)
Evaluation: Using reference vs. fine-tuned accuracy scores

Evaluation

The project uses LLM-as-a-Judge evaluation methodology:

Metrics Assessed:
- Reference accuracy (1-10 scale)
- Fine-tuned accuracy (1-10 scale)
- Factual errors identification
- Hallucination detection
Judge Model: OpenAI GPT (gpt-5-nano-2025-08-07)
Evaluation Set: Random sample of 0.8% of the dataset

Usage Example

python

from tinker import types
import tinker

# Initialize client
service_client = tinker.ServiceClient()
training_client = service_client.create_lora_training_client(
    base_model="meta-llama/Llama-3.1-8B"
)

# Get sampling client
sample_client = training_client.save_weights_and_get_sampling_client(
    name="medical-001-8B"
)

# Run inference
tokenizer = training_client.get_tokenizer()
prompt = types.ModelInput.from_ints(
    tokenizer.encode("The medical term is: Urodynia\nThe meaning of the term is:")
)
params = types.SamplingParams(max_tokens=200, temperature=0.8)
results = sample_client.sample(prompt=prompt, sampling_params=params, num_samples=1).result()

print(tokenizer.decode(results.sequences[0].tokens))

Model on Hugging Face

The fine-tuned model is available on Hugging Face Hub: mlai-dante/medical-llm

You can load and use it with the PEFT library:

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
model = PeftModel.from_pretrained(base_model, "mlai-dante/medical-llm")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

Training Details

Optimizer: Adam
Learning Rate: 1e-4
Loss Function: Cross-entropy with weighted tokens
Training Iterations: 20
Architecture: LoRA adapters only (efficient parameter updates)

Key Features

Efficient fine-tuning using LoRA (only ~0.1% of parameters trained)
Medical domain specialization
Comprehensive evaluation using LLM-as-a-Judge
Easy deployment via Hugging Face Hub
Fast inference with Tinker framework

Results

Training converges rapidly with loss dropping significantly:

Iteration 1: 0.839
Iteration 5: 0.232
Iteration 10: 0.005
Iteration 20: 0.000371

The model generates coherent and medically relevant explanations for various medical terms.

Future Improvements

Expand evaluation dataset
Add multi-turn conversation capabilities
Incorporate medical literature citations
Support for medical term relationships and synonyms
Multilingual medical terminology support

License

This project uses Meta's Llama 3.1 model, which is subject to Meta's license agreement. Please review the Llama 3.1 License before use.

Acknowledgments

Dataset: gamino/wiki_medical_terms
Base Model: Meta Llama 3.1 8B
Training Framework: Tinker
Evaluation: OpenAI GPT models

Author

Brian Temu

For questions, issues, or contributions, please open an issue on the repository.