ConMe (Confuse Me) is a benchmark dataset specifically designed to assess the capabilities of Vision-Language Models (VLMs) in Compositional Reasoning (CR). CR involves understanding the significance of attributes, relations, and word order in the context of images and text. This dataset is crafted by leveraging VLMs to generate 'hard CR Q&A' tasks, presenting a unique challenge by focusing on generating questions that expose the weaknesses of current VLMs in processing complex visual and language cues.
Methodology
The dataset is built using a novel data generation pipeline where VLMs converse with each other to collaboratively identify and generate the most challenging CR tasks. This approach not only pushes the boundaries of what modern VLMs can interpret and respond to but also ensures that the questions are difficult yet relevant, avoiding outliers and improbable scenarios often seen in other datasets.
question: Challenging questions generated through our VLM conversational pipeline.
answer: Expert-validated answers to these questions.
base_question: The unmodified base question from which the challenging one is derived.
correct_option: The correct option for the modified question.
incorrect_option: The incorrect option that closely matches the complexity of the correct one.
question_format (HUMAN_FILTERED partition): The structural format of the question, which has been tailored through LLM-based analysis tool to pinpoint specific types of reasoning challenges, leveraging a natural language description of error taxonomies.
question_topic (HUMAN_FILTERED partition): Specifies the conceptual topic or domain of the question, categorized automatically by our LLM-based analysis tool, which aligns the questions with human-specified error taxonomies, providing insights into common VLM mistakes.
Download
To download the ConMe dataset, please run the following command:
git clone https://huggingface.co/conme/ConMe
Code
The code used to generate the dataset and to analyze the different partitions can be accessed through this Github Repository: Link.
Intended Uses
This dataset is intended for advanced training and evaluating models specializing in visual question answering and compositional reasoning. It aims to significantly enhance model performance by providing a rigorously challenging set of questions and scenarios, proven to reduce CR performance by up to 33% compared to previous benchmarks.
Source and Licensing
This dataset utilizes images from the COCO validation dataset 2017, which are used under Creative Commons License. The additional data is generated and manually validated by experts and is licensed under Apache-2.0.
