This repository hosts the LoRA (Low-Rank Adaptation) weights trained during Phase 3 of my final degree thesis. This adapter is injected into the medical large language model BioGPT-Large to establish a fully functional Multimodal Vision-Language Model (VLM) capable of autonomously drafting clinical radiology reports from chest X-rays.

The core achievement of this model is the successful mapping of spatial visual features (extracted via a custom DenseNet121) into the semantic space of BioGPT. This architecture generates highly accurate diagnostic descriptions while explicitly mitigating the normality bias (linguistic prior) and domain misalignment commonly found in standard LLMs.

🏗️ Architecture & 3-Phase Pipeline

The system is built upon a rigorous 3-phase engineering pipeline designed to ensure clinical reliability and visual explainability:

Phase 1 (Clinical Visual Encoder): A customized DenseNet121 backbone (pretrained on CheXpert) utilizing Weight Surgery. The standard Global Average Pooling was replaced to output a 10x10 spatial grid (100 continuous visual tokens). Long-tail class imbalance was mitigated using asymmetric weighted loss functions (BCE + Pos-Weight).
Phase 2 (Multimodal Bridge & Geometric Alignment): An MLP projector assisted by 2D Positional Embeddings connects the visual encoder to the frozen LLM. This phase successfully reconstructs 2D topological affinity from a 1D sequence, trained on the OpenI (Indiana University) dataset.
Phase 3 (Domain Adaptation via LoRA & Focal Loss): To break the LLM's inherent "clinical blindness", LoRA adapters (r=32, α=64) were applied to the self-attention layers of BioGPT-Large. The standard Cross-Entropy was replaced with an Autoregressive Focal Loss (γ=2.0, α=1.0) to concentrate mathematical gradients exclusively on predicting rare pathological vocabulary.

📊 Evaluation & Clinical Metrics

The model has undergone strict auditing using a Zero-Shot NLP extraction pipeline (BART-Large-MNLI) and ablation studies to validate clinical fidelity against Ground Truth reports.

Visual Dependency Index (Macro VDI): 0.8333 (Proves the model relies on the radiograph, not on linguistic hallucinations).
Semantic Quality: BERTScore F1 of 0.8774 and SBERT Similarity of 0.6314.
Lexical Metrics (Metrological Dissociation): BLEU-4 (0.0811) and ROUGE-L (0.2711). These rigid indices suffer a logical collapse due to human clinical synonymy, but are heavily compensated by the strong semantic convergence mentioned above.
Diagnostic Net Yield (Micro F1-Score): 60.03% (Stable population-adjusted convergence) / Macro F1-Score: 51.45%.
Top Clinical Captures: Pulmonary Opacity (F1: 0.7617), Consolidation (F1: 0.7286), and Atelectasis (F1: 0.7151).

📂 How to Use This Model (Multimodal Inference)

Because this repository contains only the LoRA adapter weights, you must load the base BioGPT model along with the complete visual projection architecture defined in the notebooks/Modulo_II_Puente_Texto.ipynb notebook.

Crucial Inference Parameters: For optimal autoregressive generation without attentional loops ("stuttering"), the model requires Nucleus Sampling evaluation.

python

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# 1. Load Tokenizer and frozen BioGPT base model
tokenizer = AutoTokenizer.from_pretrained("microsoft/biogpt")
model_base = AutoModelForCausalLM.from_pretrained("microsoft/biogpt")

# 2. Load the LoRA Adapter from this Hub repository
model = PeftModel.from_pretrained(model_base, "RafaelCarrillo/vlm-biogpt-chestxray")

# 3. Integrate with Visual Projector and Generate (Requires Full Pipeline)
# Generation kwargs calibrated for this specific architecture:
"""
outputs = model.generate(
    inputs_embeds=multimodal_embeddings, # 100 Visual Tokens + Text Prompt
    max_new_tokens=128,
    do_sample=True,
    top_p=0.9,
    temperature=0.7,
    repetition_penalty=1.25
)

🔗 Related Resources

💻 Source Code & Architectural Documentation (GitHub): TFG-Vision-Language-Clinical-Reports
📊 Datasets & Full Weights (Google Drive): External Volume Access (Heavy Files)