tuandunghcmut/

vlmeval

🆕 News

• [2024-12-02] Supported VisOnlyQA, a benchmark for evaluating the visual perception capabilities 🔥🔥🔥
• [2024-11-26] Supported Ovis1.6-Gemma2-27B, thanks to runninglsy 🔥🔥🔥
• [2024-11-25] Create a new flag VLMEVALKIT_USE_MODELSCOPE. By setting this environment variable, you can download the video benchmarks supported from modelscope 🔥🔥🔥
• [2024-11-25] Supported VizWiz benchmark 🔥🔥🔥
• [2024-11-22] Supported the inference of MMGenBench, thanks lerogo 🔥🔥🔥
• [2024-11-22] Supported Dynamath, a multimodal math benchmark comprising of 501 SEED problems and 10 variants generated based on random seeds. The benchmark can be used to measure the robustness of MLLMs in multi-modal math solving 🔥🔥🔥
• [2024-11-21] Integrated a new config system to enable more flexible evaluation settings. Check the Document or run python run.py --help for more details 🔥🔥🔥
• [2024-11-21] Supported QSpatial, a multimodal benchmark for Quantitative Spatial Reasoning (determine the size / distance, e.g.), thanks andrewliao11 for providing the official support 🔥🔥🔥
• [2024-11-21] Supported MM-Math, a new multimodal math benchmark comprising of ~6K middle school multi-modal reasoning math problems. GPT-4o-20240806 achieces 22.5% accuracy on this benchmark 🔥🔥🔥
• [2024-11-16] Supported OlympiadBench, a new multimodal benchmark comprising olympiad-level math and physics questions 🔥🔥🔥
• [2024-11-16] Supported WildVision, a new subjective multimodal benchmark derived from multi-modal arena data 🔥🔥🔥
• [2024-11-13] Supported MIA-Bench, a multimodal instruction-following benchmark 🔥🔥🔥

🏗️ QuickStart

📊 Datasets, Models, and Evaluation Results

Evaluation Results

Supported Benchmarks

• By default, all evaluation results are presented in OpenVLM Leaderboard.
• Abbrs: MCQ: Multi-choice question; Y/N: Yes-or-No Questions; MTT: Benchmark with Multi-turn Conversations; MTI: Benchmark with Multi-Image as Inputs.

Supported Models

• Please use transformers==4.33.0 for: Qwen series, Monkey series, InternLM-XComposer Series, mPLUG-Owl2, OpenFlamingo v2, IDEFICS series, VisualGLM, MMAlaya, ShareCaptioner, MiniGPT-4 series, InstructBLIP series, PandaGPT, VXVERSE.
• Please use transformers==4.36.2 for: Moondream1.
• Please use transformers==4.37.0 for: LLaVA series, ShareGPT4V series, TransCore-M, LLaVA (XTuner), CogVLM Series, EMU2 Series, Yi-VL Series, MiniCPM-[V1/V2], OmniLMM-12B, DeepSeek-VL series, InternVL series, Cambrian Series, VILA Series, Llama-3-MixSenseV1_1, Parrot-7B, PLLaVA Series.
• Please use transformers==4.40.0 for: IDEFICS2, Bunny-Llama3, MiniCPM-Llama3-V2.5, 360VL-70B, Phi-3-Vision, WeMM.
• Please use transformers==4.44.0 for: Moondream2, H2OVL series.
• Please use transformers==4.45.0 for: Aria.
• Please use transformers==latest for: LLaVA-Next series, PaliGemma-3B, Chameleon series, Video-LLaVA-7B-HF, Ovis series, Mantis series, MiniCPM-V2.6, OmChat-v2.0-13B-sinlge-beta, Idefics-3, GLM-4v-9B, VideoChat2-HD, RBDash_72b, Llama-3.2 series, Kosmos series.

• Please use torchvision>=0.16 for: Moondream series and Aria

• Please use pip install flash-attn --no-build-isolation for: Aria

🛠️ Development Guide

• All Contributions will be acknowledged in the report.
• Contributors with 3 or more major contributions (implementing an MLLM, benchmark, or major feature) can join the author list of VLMEvalKit Technical Report on ArXiv. Eligible contributors can create an issue or dm kennyutc in VLMEvalKit Discord Channel.

🎯 The Goal of VLMEvalKit

1. Provide an easy-to-use, opensource evaluation toolkit to make it convenient for researchers & developers to evaluate existing LVLMs and make evaluation results easy to reproduce.
2. Make it easy for VLM developers to evaluate their own models. To evaluate the VLM on multiple supported benchmarks, one just need to implement a single generate_inner() function, all other workloads (data downloading, data preprocessing, prediction inference, metric calculation) are handled by the codebase.

1. Reproduce the exact accuracy number reported in the original papers of all 3rd party benchmarks. The reason can be two-fold:
   1. VLMEvalKit uses generation-based evaluation for all VLMs (and optionally with LLM-based answer extraction). Meanwhile, some benchmarks may use different approaches (SEEDBench uses PPL-based evaluation, eg.). For those benchmarks, we compare both scores in the corresponding result. We encourage developers to support other evaluation paradigms in the codebase.
   2. By default, we use the same prompt template for all VLMs to evaluate on a benchmark. Meanwhile, some VLMs may have their specific prompt templates (some may not covered by the codebase at this time). We encourage VLM developers to implement their own prompt template in VLMEvalKit, if that is not covered currently. That will help to improve the reproducibility.

🖊️ Citation