You can deploy this model and start earning money today!

suimu/
InstructAV2AV

Name: InstructAV2AV
Availability: InStock

audio-video

audio-video-editing

instruction-guided-editing

Views

No views yet

InstructAV2AV: Instruction-Guided Audio-Video Joint Editing

Haojie Zheng^{1, 2}, Yixin Yang², Siqi Yang², Shuchen Weng^{1, 2*}, Boxin Shi^2*
^*Corresponding authors
¹BAAI, ²Peking University

Overview

InstructAV2AV is an instruction-guided framework for joint audio-video editing. Given a source video with audio and a natural-language instruction, the model edits the visual and audio content together while preserving the unedited properties of the source as much as possible.

This repository contains six task-specific InstructAV2AV checkpoints. They share the same architecture and inference pipeline, but are fine-tuned for different editing objectives, including general editing, content insertion and removal, visual identity preservation, voice preservation, and joint identity-and-voice preservation.

The inference and training code is available in the InstructAV2AV GitHub repository. The corresponding training dataset is available as InsAVE-80K.

Video Demo

For the best experience, please enable audio.

Checkpoints

Checkpoint	Editing type	Description	Example instruction
`general.safetensors`	General editing	Edits appearance, scenes, actions, speech, and sound.	`Make the horse dark brown with a white saddle.`
`insertion.safetensors`	Content insertion	Adds an object or other content to the source video.	`Add a dark vintage sedan driving from the right to the left.`
`removal.safetensors`	Content removal	Removes an object and its associated audiovisual content.	`Remove the chipmunk standing on the stone surface among the peanuts.`
`clone_id.safetensors`	Identity cloning	Preserves a person's visual identity while editing other visual or audio attributes.	`Keep the person's appearance, change the timbre to a man, and change the spoken words to <S>I understand, but I think we need to consider.<E>.`
`clone_voice.safetensors`	Voice cloning	Preserves the speaker's timbre while editing the video or spoken content.	`Keep the timbre, change the person's appearance, and change the spoken words to <S>I came here to tell you that you should go.<E>.`
`clone_id_voice.safetensors`	Identity and voice cloning	Preserves both the person's visual identity and the speaker's timbre.	`Keep the person's identity and voice, and change the spoken words to <S>This is more than just art, it's a statement.<E>.`

Select the checkpoint that most closely matches the requested edit and pass its path through --finetune-path.

Installation

The checkpoints must be used with the InstructAV2AV codebase.

bash

git clone https://github.com/suimuc/InstructAV2AV.git
cd InstructAV2AV

conda create -n instructav2av python=3.10 -y
conda activate instructav2av

pip install torch==2.6.0 torchvision torchaudio
pip install -r requirements.txt
pip install flash_attn --no-build-isolation
pip install -e .

The provided installation instructions target CUDA 12.1.

Download

Download all six InstructAV2AV checkpoints:

bash

huggingface-cli download suimu/InstructAV2AV \
  --local-dir ckpts/InstructAV2AV

The inference pipeline also requires the Wan2.2 text encoder and video VAE, together with the MMAudio audio VAE and vocoder:

bash

# Text encoder and video VAE
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
  --include \
    "google/*" \
    "models_t5_umt5-xxl-enc-bf16.pth" \
    "Wan2.2_VAE.pth" \
  --local-dir ckpts/Wan2.2-TI2V-5B

# Audio VAE and vocoder
huggingface-cli download hkchengrex/MMAudio \
  --include \
    "ext_weights/best_netG.pt" \
    "ext_weights/v1-16.pth" \
  --local-dir ckpts/MMAudio

The resulting checkpoint layout should include:

text

ckpts/
├── InstructAV2AV/
│   ├── clone_id.safetensors
│   ├── clone_id_voice.safetensors
│   ├── clone_voice.safetensors
│   ├── general.safetensors
│   ├── insertion.safetensors
│   └── removal.safetensors
├── MMAudio/
│   └── ext_weights/
│       ├── best_netG.pt
│       └── v1-16.pth
└── Wan2.2-TI2V-5B/
    ├── google/
    ├── models_t5_umt5-xxl-enc-bf16.pth
    └── Wan2.2_VAE.pth

Inference

The unified command-line entry point is scripts/edit.py. The source video must contain an audio track unless a separate audio file is supplied with --source-audio.

Single-video editing

bash

python scripts/edit.py \
  --source-video assets/input.mp4 \
  --instruction "Keep the person's identity and change the spoken words to <S>This is more than just art, it's a statement.<E>." \
  --finetune-path ckpts/InstructAV2AV/clone_id_voice.safetensors \
  --output outputs/edited.mp4

To use a separate audio track:

bash

python scripts/edit.py \
  --source-video assets/input.mp4 \
  --source-audio assets/input.wav \
  --instruction "Make the horse dark brown with a white saddle." \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output outputs/edited.mp4

Prompt enhancement

When specifying new spoken content, place the target utterance between <S> and <E> markers:

Change the spoken words to <S>This is the new sentence.<E>.

For better editing quality and text-to-video alignment, append a detailed description of the desired final video immediately after the editing command. The enhanced prompt can describe the subject's appearance, clothing, actions, expressions, scene, and target speech. For example:

bash

python scripts/edit.py \
  --source-video assets/input1.mp4 \
  --instruction "Change the man into a young woman with brown hair, wearing a gray blazer over a light pink top and a necklace with a heart-shaped pendant, and saying, <S>I really think we should give it another chance.<E>. A young woman with long, wavy brown hair and fair skin is engaged in a conversation. She is wearing a gray blazer over a light pink top and has a necklace with a heart-shaped pendant. Her facial expressions change throughout the sequence, showing a range of emotions that suggest she is either explaining something earnestly or reacting to a conversation, and says <S>I really think we should give it another chance.<E>. The setting appears to be indoors, with a dimly lit, blurred background that suggests a social environment, possibly a bar or restaurant. The focus remains on the woman's face, capturing her reactions and engagement in the dialogue." \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output outputs/edited.mp4

Gradio demo

After downloading all six checkpoints and the dependency weights, launch the interactive demo with:

python scripts/demo.py --share

Default Inference Configuration

The released configuration uses 121 frames at 704 × 1280 resolution, 16 kHz audio, 50 sampling steps, and bfloat16 inference. CUDA is required. See ovi/configs/inference/inference_av_edit.yaml in the code repository for all options, including guidance scales, seed, sequence parallelism, and CPU offloading.

InsAVE-80K Dataset

The training dataset is available at suimu/InsAVE-80K.

Training

The unified training entry point is scripts/train.py, with configuration in ovi/configs/train/train_av_edit.yaml. The trainer accepts CSV, JSON, or JSONL manifests with the following canonical fields:

Field	Description
`source_video`	Original video
`source_audio`	Original audio
`target_video`	Edited video
`target_audio`	Edited audio
`instruction`	Editing instruction

Before training, set the checkpoint directory, initialization checkpoint, and dataset manifest in the training configuration:

yaml

ckpt_dir: ./ckpts
finetune_path: ./ckpts/InstructAV2AV/general.safetensors

dataset:
  metadata_path: ./data/InsAVE-80K/path/to/manifest.csv

The manifest and initialization checkpoint can also be overridden from the command line:

bash

accelerate launch \
  --config_file ovi/configs/train/accelerate_config.yaml \
  scripts/train.py \
  --config-file ovi/configs/train/train_av_edit.yaml \
  --data-manifest data/InsAVE-80K/path/to/manifest.csv \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output-dir outputs/train_av_edit

The provided Accelerate configuration uses DeepSpeed ZeRO-2 and is configured for eight processes. Change num_processes in ovi/configs/train/accelerate_config.yaml to match the available GPUs.

The training objective is selected from the modality flags:

`has_video`	`has_audio`	Training objective
`true`	`true`	Joint audio-video editing
`true`	`false`	Video-only editing
`false`	`true`	Audio-only editing

Related Resources

Citation

If you find InstructAV2AV or InsAVE-80K useful in your research, please cite:

bibtex

@article{instructav2av2026,
  title={InstructAV2AV: Instruction-Guided Audio-Video Joint Editing},
  author={Zheng, Haojie and Yang, Yixin and Yang, Siqi and Weng, Shuchen and Shi, Boxin},
  journal={arXiv preprint arXiv:2605.18467},
  year={2026}
}

License

The project is released under the Apache License 2.0. Users must also comply with the licenses and terms of the underlying pretrained models and any input media or datasets they use.