You can deploy this model and start earning money today!

cnxup/
LLaVA-NeXT-8B-MLA-stage1-rope32

Name: LLaVA-NeXT-8B-MLA-stage1-rope32
Rating: 2.5 (5 reviews)

safetensors

llava_next

2601.11464

Views

No views yet

Research Paper: "MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models"

Model Description

This is a Stage 1 checkpoint for the MHA2MLA-VLM conversion process. This model serves as an intermediate checkpoint trained with partial RoPE (rope_dim=32) and is specifically designed to provide weight initialization for Stage 2 MHA2MLA-VLM models through SVD decomposition.

Usage

This Stage 1 model is primarily used to generate SVD initialization weights for Stage 2 MLA models:

Step 1: Download this Stage 1 checkpoint
Step 2: For MHA2MLA-VLM models using Partial-RoPE MKL method, Download the MKL file.
Step 3: Download and use the corresponding Stage 2 MHA2MLA-VLM models for inference. Refer to each Stage 2 model's README for detailed inference instructions. Choose from different d_kv dimensions:
- LLaVA-NeXT-8B-MLA-stage2-rope32-d_kv_32
- LLaVA-NeXT-8B-MLA-stage2-rope32-d_kv_64
- LLaVA-NeXT-8B-MLA-stage2-rope32-d_kv_128

Citation

bibtex

@misc{fan2026mha2mlavlmenablingdeepseekseconomical,
      title={MHA2MLA-VLM: Enabling DeepSeek's Economical Multi-Head Latent Attention across Vision-Language Models}, 
      author={Xiaoran Fan and Zhichao Sun and Tao Ji and Lixing Shen and Tao Gui},
      year={2026},
      eprint={2601.11464},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.11464}, 
}

cnxup/LLaVA-NeXT-8B-MLA-stage1-rope32

Model Description

Usage

Citation

cnxup/
LLaVA-NeXT-8B-MLA-stage1-rope32