Model type: Matryoshka Multimodal Models (M3) allow using to explicitly control visual granularities (the number of visual toknes per sample) at time time. Also, the model itself serves as a metric for image/dataset complexity. M3s is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on visual conversation data. It is an auto-regressive language model, based on the transformer architecture.

Model date: llava-v1.5-7b-m3 was trained in May 2024. Paper

Paper or resources for more information: https://matryoshka-mm.github.io/

Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

Where to send questions or comments about the model: https://github.com/mu-cai/matryoshka-mm/issues

Primary intended uses: The primary use of M3 is research on large multimodal models and chatbots.

Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.

Matryoshka Multimodal Models (M3) achieves strong performance even using 1 or 9 visual tokens per image.