This model is a fine-tuned version of Pixtral-12B on the GroundCap dataset for grounded image captioning. It generates detailed image descriptions with explicit grounding tags that link textual descriptions to specific visual elements in the image. The model was trained on the GroundCap dataset and uses a novel tag system to ground objects (<gdo>), actions (<gda>), and locations (<gdl>) to specific regions in images.
Model Details
Model Description
Developed by: Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos (INESC-ID Lisboa and Instituto Superior Técnico, Universidade de Lisboa)
Model type: Fine-tuned Pixtral-12B model for grounded image captioning
Language(s): English
License: Creative Commons Attribution 4.0
Finetuned from model: mistral-community/pixtral-12b
The model is designed for generating grounded image captions that explicitly link textual descriptions to visual elements using three types of grounding tags:
<gdo> for objects
<gda> for actions
<gdl> for locations
Each tag maintains object identity through unique IDs, enabling consistent reference tracking throughout the caption.
Content management systems needing verifiable image captions
Visual question answering systems
Image retrieval systems
Out-of-Scope Use
The model is not designed for:
General image classification
Object detection (requires separate object detection pipeline)
Video captioning
Non-English language captioning
How to Get Started with the Model
Input Format
The model expects input in the following format:
You are an AI assistant that can see and understand images. I will provide you with an image and the detected objects in it along with their positions and dimensions in the format [id, x,y,width,height].
[DETECTIONS]
[sky-0: 0.41,0.00,0.20,0.15]
[sky-1: 0.62,0.00,0.26,0.10]
[wall-0: 0.01,0.02,0.35,0.86]
[person-0: 0.38,0.35,0.12,0.40]
[person-1: 0.45,0.35,0.08,0.39]
[wall-1: 0.39,0.10,0.35,0.48]
[person-2: 0.71,0.29,0.20,0.51]
[wall-2: 0.75,0.03,0.24,0.88]
[person-3: 0.00,0.57,0.22,0.42]
[handbag-0: 0.21,0.75,0.11,0.23]
[person-4: 0.26,0.48,0.20,0.52]
[floor-wood-0: 0.40,0.59,0.60,0.41]
[/DETECTIONS]
[IMG]
Example Output
The model will generate a grounded caption using three types of tags:
<gdo> for objects
<gda> for actions
<gdl> for locations
Example output:
In this scene, a group of individuals is gathered in what appears to be a <gdl class="wall" wall-0 wall-1 wall-2>rugged, makeshift shelter</gdl>. The <gdl class="wall" wall-0 wall-1 wall-2>walls</gdl> are constructed from <gdl class="wall" wall-0 wall-1 wall-2>rough materials</gdl>, giving the space a temporary and utilitarian feel. The <gdl class="sky" sky-0 sky-1>sky</gdl> is visible in the background, suggesting that the shelter is partially open or lacks a complete roof.
Several <gdo class="person" person-0 person-1 person-2 person-3 person-4>people</gdo> are present in the scene, each engaged in their own activities. <gdo class="person" person-0>One individual</gdo> <gda class="sit" person-0>sits</gda> on the <gdl class="floor-wood" floor-wood-0>ground</gdl>, while <gdo class="person" person-1>another person</gdo> <gda class="sit" person-1>is seated</gda> nearby. <gdo class="person" person-2>Another person</gdo> <gda class="sit" person-2>is also sitting</gda> on the <gdl class="floor-wood" floor-wood-0>ground</gdl>, and <gdo class="person" person-3>a fourth individual</gdo> <gda class="sit" person-3>is seated</gda> as well. <gdo class="person" person-4>An additional person</gdo> <gda class="sit" person-4>is sitting</gda> close by.
The <gdo class="handbag" handbag-0>handbag</gdo> is placed on the <gdl class="floor-wood" floor-wood-0>ground</gdl> near one of the individuals, suggesting they might have brought some personal belongings with them. The overall atmosphere of the scene is one of simplicity and resilience, with the individuals making the best of their surroundings in this temporary shelter.
Bias, Risks, and Limitations
The model was trained on movie scenes from MovieNet, which may introduce biases in terms of scene composition, lighting, and camera angles
Performance may vary for real-world images that differ significantly from movie scenes
The model relies on pre-detected objects and their bounding boxes, Mask2Former was used for object detection in the original paper
Recommendations
Use in conjunction with a robust object detection system
Verify grounding accuracy for critical applications
Consider the movie-centric nature of the training data when applying to other domains
Training Details
Training Data
The model was trained on the GroundCap dataset, which contains:
52,016 images from 77 movies
344 human-annotated captions
52,016 automatically generated captions
Training Procedure
The training followed a two-stage approach:
Stage 1:
Training on 52,016 automatically generated captions
1@article{Oliveira2025GroundCapAV,
2title={GroundCap: A Visually Grounded Image Captioning Dataset},
3author={Daniel A. P. Oliveira and Louren{ç}o Teodoro and David Martins de Matos},
4year={2025},
5url={https://api.semanticscholar.org/CorpusID:276450057}6}
Model Card Authors
Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos
