A frozen Whisper-large-v3-turbo encoder is grafted onto Qwen2-VL-7B-Instruct through learned linear projections. The same frozen encoder serves two roles: it encodes the video's music track and the user's question, which is posed as synthesized speech rather than text. The language model fuses visual frames, music, and question audio entirely through its pretrained self-attention — there is no cross-modal attention module and no learned fusion network.

The only new modality-specific modules are two linear projectors. Whisper and the Qwen2-VL base weights stay frozen throughout; AVQA Stage 2 additionally trains LoRA adapters (r=64) on the LLM's attention layers.

Paper: https://arxiv.org/abs/2608.11329
Code: https://github.com/MKDehdashti/Qwen2-vl-audio
Developed by: Maryam Dehdashti (Inference Matter Labs)
Model type: multimodal (video + music audio + spoken question -> generated text)
Finetuned from: Qwen/Qwen2-VL-7B-Instruct

Not an omni-modal model. There is no text-question path in the deployed configuration and no speech output. The point of the work is that a modular graft is a competitive alternative to omni-modal systems on this task.

Results

MUSIC-AVQA, evaluated on the 7,402-pair available-video test subset (about 20% of the benchmark's videos were no longer downloadable), trained on 8,000 pairs.

Model	Test acc.
Qwen2.5-Omni-7B zero-shot (audio-matched)	56.82%
Qwen2.5-Omni-7B fine-tuned (matched inputs/hyperparameters)	80.91%
PANNs-8 (ablation)	66.87%
PANNs-32 (ablation)	69.90%
Whisper-60s-compressed (ablation)	70.50%
Whisper-30s	95.91%
Qwen-MusicAVQA-7B (Whisper-60s-chunked)	97.31%

The headline row is the seed-42 representative run; across three full retraining seeds the result is 96.0% +/- 3.9%. Per-modality for the headline model: Audio 97.4%, Audio-Visual 97.3%, Visual 97.3%.

Central finding. Accuracy tracks how much fine-grained local temporal information the audio representation preserves. At a matched 32-token budget, a stride-pooled Whisper frame sequence beats a globally pooled PANNs vector by 26 points — even though PANNs sees at least as much audio and uses a 50x larger projector.

Robustness. On the rephrased MUSIC-AVQA-R benchmark, accuracy is 96.5% (head) and 95.6% (tail) on 3,000-question samples of each split, providing evidence that accuracy is not solely dependent on the original question templates.

ASR. The same architecture reaches 4.85% WER on the full LibriSpeech test-clean (normalized); that checkpoint is what the AVQA track initializes from.

The fine-tuned Omni comparison matches data, inputs, audio duration, and Stage-2 hyperparameters, but the systems differ in backbone and adaptation; read it as a system-level comparison rather than an isolated encoder comparison.

Published MUSIC-AVQA baselines use the full official splits; these numbers use the available-video subset and 8,000 training pairs, so the two are not directly comparable. See the paper.

Efficiency

Both encoders are frozen and music features are cached, so the complete two-stage AVQA run takes about 5 A100-hours on a single A100 80GB. Stage 1 trains only the 4.6M-parameter music projector (0.05% of the model's parameters) and already reaches 96.0% with the LLM frozen.

Repository layout

The repo root is the merged headline model — Stage-1 projector and Stage-2 LoRA already merged, so it loads in one line. Everything else is organized by track.

(root)                       Qwen-MusicAVQA-7B, merged  -  97.31%
asr/
  merged_stage2/             ASR Stage-2 merge (4.85% WER); the AVQA starting point
  stage1_only/               ASR Stage 1
  lora_stage2/  lora_stage3/ ASR LoRA adapters
avqa/
  init/                      merged ASR model + untrained music projector (see its README)
  headline/
    stage1/                  music projector trained, LLM frozen  -  96.0%
    stage2_qproj_frozen/     LoRA, question projector frozen      -  97.31%  (the root model)
    stage2_qproj_tuned/      LoRA, question projector tuned       -  95.49%
  seeds/seed1234|seed2026/{stage1,stage2}/     the 96.0% +/- 3.9% runs
  ablations/
    panns8/  panns32/                          pooled PANNs vector
    whisper32/  whisper32_full/                Whisper-30s, Whisper-60s-compressed
    whisper_fullres/  whisper_fullres_varlen/  whisper_fullres_ts/
    whisper_fullres_notts/  whisper_fullres_notts_matched/   text-question variants
  comparison/qwen2.5-omni/   fine-tuned Qwen2.5-Omni baseline

headline/stage1/ is shared: the two Stage-2 variants were both trained from it, differing only in whether the question projector stayed frozen. Ablation folder names are the internal W&B experiment_tag strings; the GitHub README maps them to the descriptive configuration names used in the paper (e.g. whisper32 = Whisper-30s, whisper32_full = Whisper-60s-compressed).

All precomputed audio features are in MayaKD/qwen2-vl-audio-data — whisper_features_fullres/ (headline model), whisper_features_full/ (60 s-compressed), whisper_features/ (30 s), whisper_features_fullres_varlen/, panns_features/, clap_features/, and tts_questions_r/ (MUSIC-AVQA-R). Download these rather than recomputing: they derive from the MUSIC-AVQA source videos, which are not redistributed.

Video frames and TTS questions are not published — too large to upload. Regenerate them with src/avqa/video_precompute.py (needs the source videos) and src/avqa/tts_preprocess.py (needs no video — the question text ships with the code); see the GitHub README's preprocessing section.

Usage

These checkpoints require the custom transformers fork that adds the Qwen2VLAudio model classes; they will not load with stock transformers.

bash

git clone --recurse-submodules https://github.com/MKDehdashti/Qwen2-vl-audio
cd Qwen2-vl-audio
pip install -e ./transformers && pip install -e ./qwen-vl-utils
pip install -r requirements.txt

The merged model is at the repo root:

python

from transformers.models.qwen2_vl.modeling_qwen2_vl import Qwen2VLDualAudioForConditionalGeneration
model = Qwen2VLDualAudioForConditionalGeneration.from_pretrained(
    "MayaKD/qwen2-vl-audio", torch_dtype="bfloat16")

Music features are precomputed offline and passed as music_features; question audio is encoded live by the same frozen Whisper encoder. See the repository README for preprocessing and the training entry points.

To reproduce training rather than run inference, start from asr/merged_stage2/ — that is the checkpoint every AVQA run initializes from, not the root.

Limitations

Evaluated only on MUSIC-AVQA (plus MUSIC-AVQA-R for rephrasing robustness), with a closed 42-answer vocabulary, so exact-match accuracy does not measure open-ended generation.
The official splits share source videos across train and test, so evaluation measures held-out video-question pairs rather than generalization to unseen videos.
Questions are delivered as synthesized speech; a text-question variant underperforms and is discussed in the paper.
Music features are precomputed, so inference on a new video needs a separate encoding pass.
Each ablation row is a single training run; only the headline configuration has three seeds.
No music-domain audio encoder (e.g. MERT) was evaluated.

License

Code (GitHub repository): MIT.
Model weights: derived from Qwen2-VL-7B-Instruct (Apache-2.0) and Whisper-large-v3-turbo (MIT); the upstream terms carry over to these checkpoints, and the repository is labelled Apache-2.0 accordingly.
MUSIC-AVQA data is subject to its own terms; no benchmark videos are redistributed here.

Citation

bibtex

@article{dehdashti2026qwenmusicavqa,
  title   = {Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA},
  author  = {Dehdashti, Maryam},
  journal = {arXiv preprint arXiv:2608.11329},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.11329}
}

Contact: dehdashti@inferencematter.ai