ASID-Captioner-3B is an audiovisual captioning model (based on Qwen2.5-Omni) fine-tuned for attribute-structured and quality-verified video understanding. It is designed to generate fine-grained captions that cover both visual and audio signals, with controllable prompting over multiple attributes.

[🏠 Homepage] [📖 Arxiv Paper] [🤗 Models & Datasets] [💻 Code]

Introduction

Modern video MLLMs often describe long and complex audiovisual content with a single caption, which can be incomplete (missing audio or camera details), unstructured, and weakly controllable.

ASID-Captioner-3B is trained to follow attribute-specific instructions and produce more organized, fine-grained descriptions. It is built upon Qwen2.5-Omni and fine-tuned on ASID-1M, which provides structured supervision over multiple attributes (scene, characters, objects, actions, narrative elements, speech, camera, emotions) with quality verification and refinement.

Key Features

Audiovisual captioning: uses both video frames and audio (when available).
Attribute-structured instruction following: supports prompts targeting specific attributes (e.g., speech-only, camera-only).
High-quality supervision: trained on attribute-structured, quality-verified instructions from ASID-1M.
Standard Transformers interface: load with transformers and the Qwen2.5-Omni processor/model classes.

What’s in this repo

Typical files include:

config.json
generation_config.json
preprocessor_config.json
chat_template.jinja
added_tokens.json / special_tokens_map.json
model-*.safetensors and model.safetensors.index.json

Prompting (recommended)

ASID-Captioner-3B works best with explicit attribute prompts, for example:

Describe the scene in the video in detail. Write your answer as one coherent paragraph.
Describe the characters in the video in detail. Write your answer as one coherent paragraph.
Provide a comprehensive description of all the content in the video, leaving out no details, and naturally covering the scene, characters, objects, actions, narrative elements, speech, camera, and emotions in a single coherent account.

Usage (minimal, single GPU)

Install

pip install -U transformers accelerate

Optional: faster attention

If you want faster attention (optional), install FlashAttention2 following its official instructions.

You must also have qwen_omni_utils.process_mm_info available in your environment (same as your reference script).

Run inference

python

import os
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info

# Constants (same spirit as reference)
VIDEO_MAX_PIXELS = 401408        # 512*28*28
VIDEO_TOTAL_PIXELS = 20070400    # 512*28*28*50
USE_AUDIO_IN_VIDEO = True

# Some pipelines use this env var
os.environ["VIDEO_MAX_PIXELS"] = str(VIDEO_TOTAL_PIXELS)

model_id = "AudioVisual-Caption/ASID-Captioner-3B"

model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",  # optional; remove if not available
    low_cpu_mem_usage=True,
)
model.disable_talker()

processor = Qwen2_5OmniProcessor.from_pretrained(model_id)

file_path = "/path/to/video.mp4"
prompt = "Provide a comprehensive description of all the content in the video, leaving out no details, and naturally covering the scene, characters, objects, actions, narrative elements, speech, camera, and emotions in a single coherent account."

conversation = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "video", "video": file_path, "max_pixels": VIDEO_MAX_PIXELS},
            {"type": "text", "text": prompt},
        ],
    },
]

text = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=False,
)

# IMPORTANT: reference-style multimodal extraction
audios, images, videos = process_mm_info(
    conversation,
    use_audio_in_video=USE_AUDIO_IN_VIDEO,
)

inputs = processor(
    text=text,
    audio=audios,
    images=images,
    videos=videos,
    return_tensors="pt",
    padding=True,
    use_audio_in_video=USE_AUDIO_IN_VIDEO,
)

device = "cuda"
inputs = inputs.to(device).to(model.dtype)

with torch.no_grad():
    text_ids = model.generate(
        **inputs,
        use_audio_in_video=USE_AUDIO_IN_VIDEO,
        do_sample=False,
        thinker_max_new_tokens=4096,
        repetition_penalty=1.1,
        use_cache=True,
    )

decoded = processor.batch_decode(
    text_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]

answer = decoded.split("\nassistant\n")[-1].strip()
print(answer)

Notes (important)

If you do not use process_mm_info, you may get missing/incorrect audiovisual inputs in some environments.
use_audio_in_video=True enables audio-conditioned captioning when your runtime supports extracting audio from the video container.
thinker_max_new_tokens is used in the reference script. If your environment does not recognize it, replace with max_new_tokens.

Training Data

This model is fine-tuned using ASID-1M (attribute-structured and quality-verified audiovisual instructions). Dataset: AudioVisual-Caption/ASID-1M

Citation

If you use our model in your research, please cite our paper:

bibtex

@article{li2026asid,
    title={Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions},
    author={Li, Yunheng and Zhang, Hengrui and Guo, Meng-Hao and Gao, Wenzhao and Jia, Shaoyong and Jiao, Shaohui and Hou, Qibin and Cheng, Ming-Ming},
    journal={arXiv preprint arXiv:2602.13013},
    year={2026}
}

Contact

Please open a Discussion on the Hugging Face page for usage questions or issues.