This repository contains a pre-trained DECAP model from the Patch-ioner framework, presented in the paper "One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework". Patch-ioner is designed for dense image captioning and controllable visual description.

🎯 Model Overview

Model Type: DECAP
Configuration: mlp_noise.k.yaml
Vision Backbone: dinov2_vitb14_reg
Language Model: GPT-2
Input Resolution: 518x518
Prefix Size: 768

DeCap Configuration

Memory Bank: Not used
Projection Type: coco
Linear Talk2DINO: False

📊 Performance

Task	METEOR	CIDEr	SPICE
Image Captioning	0.215	0.655	0.155
Narratives	11.500	29.300	12.300

📈 Detailed Results

Image Captioning Results

METEOR: 0.2147
CIDEr: 0.6553
SPICE: 0.1551
BLEU_4: 0.1957
ROUGE_L: 0.4542
CLIP-S: 0.7090

Narratives Results

METEOR: 11.5000
CIDEr: 29.3000
SPICE: 12.3000
BLEU_4: 3.0000
ROUGE_L: 24.7000
CLIP-S: 66.7000

🚀 Quick Start (Patch-ioner library)

python

from patch_ioner import load_model, Patchioner

# Load the model
config_path = "config.yaml"
model = load_model(config_path)

# Run inference
image_path = "your_image.jpg"
results = model.forward(image_path)
print(results)

🚀 Quick Start (Transformers library)

python

from transformers import AutoModel

MODEL_ID = "Ruggero1912/Patch-ioner_talk2dino_capdec_COCO_Captions" # Example model ID

model = AutoModel.from_pretrained(MODEL_ID, trust_remote_code=True)
# Further usage would follow the Hugging Face Transformers pattern

📁 Repository Contents

config.yaml: Model configuration file
coco_karpathy-009.pt: Pre-trained model weights
README.md: This file

🔧 Installation

pip install git+https://github.com/Ruggero1912/Patch-ioner

💡 Usage Examples

Refer to the Patch-ioner repository for updated usage examples.

🎛️ Model Configuration

Prefix Size: 768
Memory Bank Size: 0
Normalization: True

📈 Training Details

Training Dataset: COCO Captions
Training Epochs: TBD
Batch Size: TBD
Learning Rate: TBD
Optimizer: AdamW

📚 Citation

If you use this model in your research, please cite our paper, refer to the Project Page for updated citation template.

🤝 Contributing

We welcome contributions to improve the Patch-ioner framework. Please see the main repository for contribution guidelines.

📄 License

See the main repository for detailed license information.

🐛 Issues and Support

For issues related to this model or the Patch-ioner framework, please:

Check the main repository for existing issues
Open a new issue with detailed information about your problem
Contact the authors.

🔗 Related Models

Explore other Patch-ioner model configurations:

Patch-ioner_mlp - MLP-based DeCap model
Patch-ioner_viecap - VieCap controllable captioning
Patch-ioner_clipcap - ClipCap integration

More models available in Ruggero1912's models

This model is part of the Patch-ioner framework for dense image captioning and controllable visual description.