You can deploy this model and start earning money today!

rishii100/
knowledge-guided-vqa-models

pytorch

vqa

visual-question-answering

Views

No views yet

Knowledge-Guided VQA Models

This repository contains trained model weights for Knowledge-Guided Visual Question Answering on the OK-VQA dataset.

These models explore different strategies for injecting external LLM-derived knowledge (visual focus cues and semantic knowledge) into a frozen Vision Transformer (ViT) + Flan-T5 pipeline for open-ended VQA.

🏗️ Architecture

Component	Model
Vision Encoder	`google/vit-base-patch16-224`
Text Encoder	CLIP Text (`openai/clip-vit-base-patch16`)
Decoder (Answer Generator)	`google/flan-t5-small`
Dataset	OK-VQA (train + val split)

📦 Available Models

1. `best_baseline_model.pth` (309 MB)

Strategy: No external knowledge. Standard ViT → CLS projection → T5 decoder.
Purpose: Serves as the baseline for comparison.

2. `best_fusion_model.pth` (313 MB)

Strategy: Late Fusion. Concatenates ViT CLS token with CLIP-encoded guidance vector, then projects to T5.
Knowledge Source: LLM-generated visual focus + semantic knowledge via CLIP text encoder.

3. `best_attention_model.pth` (325 MB)

Strategy: Attention Modification. Injects per-layer guidance projections into the top 6 ViT self-attention layers.
Knowledge Source: LLM-generated visual focus + semantic knowledge via CLIP text encoder.

🔧 Usage

python

import torch
from transformers import ViTModel, AutoTokenizer, AutoModelForSeq2SeqLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load base models
t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
t5 = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small").to(device)
vit = ViTModel.from_pretrained("google/vit-base-patch16-224").to(device)

# Define your model class (see notebook for full definitions)
# Then load weights:
model = YourModelClass().to(device)
model.load_state_dict(torch.load("best_baseline_model.pth", map_location=device))

📊 Training Details

Optimizer: Adam (lr=3e-5)
Early Stopping: Patience = 3-4 epochs
Max Epochs: 15
Batch Size: 8
Evaluation Metrics: Exact Match, Edit Similarity, Token F1, BERTScore

📄 Citation

If you use these models in your research, please cite:

bibtex

@misc{knowledge-guided-vqa-2026,
  title={Knowledge-Guided Visual Question Answering with Adaptive Vision-Language Integration},
  author={Aneerban},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/rishii100/knowledge-guided-vqa-models}
}

📜 License

MIT License

rishii100/knowledge-guided-vqa-models