Views
No views yet
TL; DR: AnyTalker is an audio-driven framework for generating multi-person talking videos. It features a flexible multi-stream structure to scale identities while ensuring seamless inter-identity interactions.
| Input Image | Generated Video |
|---|---|
 | |
 | |
 |
conda create -n AnyTalker python=3.10
conda activate AnyTalker
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126pip install -r requirements.txtpip install ninja
pip install flash_attn==2.8.1 --no-build-isolation1# Ubuntu / Debian
2apt-get install ffmpeg1# CentOS / RHEL
2yum install ffmpeg ffmpeg-devel1# Conda (no root required)
2conda install -c conda-forge ffmpeg⚠️ Note: If you install FFmpeg via conda and encounter the errorUnknown encoder 'libx264', or if the following command does not list libx264:ffmpeg -encoders | grep libx264you can install a specific conda-forge build that includes libx264 support:conda install -c conda-forge ffmpeg=7.1.0Reference: bytedance/LatentSync#60
| Models | Download Link | Notes |
|---|---|---|
| Wan2.1-Fun-V1.1-1.3B-InP | 🤗 Huggingface | Base model |
| wav2vec2-base | 🤗 Huggingface | Audio encoder |
| AnyTalker-1.3B | 🤗 Huggingface | Our weights |
1# curl -LsSf https://hf.co/cli/install.sh | bash
2hf download alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP --local-dir ./checkpoints/Wan2.1-Fun-1.3B-Inp
3hf download facebook/wav2vec2-base-960h --local-dir ./checkpoints/wav2vec2-base-960h
4hf download zzz66/AnyTalker-1.3B --local-dir ./checkpoints/AnyTalkercheckpoints/
├── Wan2.1-Fun-V1.1-1.3B-InP
├── wav2vec2-base-960h
└── AnyTalker1#!/bin/bash
2export CUDA_VISIBLE_DEVICES=0
3python generate_a2v_batch_multiID.py \
4 --ckpt_dir="./checkpoints/Wan2.1-Fun-1.3B-Inp" \
5 --task="a2v-1.3B" \
6 --size="832*480" \
7 --batch_gen_json="./input_example/customize_your_input_here.json" \
8 --batch_output="./outputs" \
9 --post_trained_checkpoint_path="./checkpoints/AnyTalker/1_3B-single-v1.pth" \
10 --sample_fps=24 \
11 --sample_guide_scale=4.5 \
12 --offload_model=True \
13 --base_seed=44 \
14 --dit_config="./checkpoints/AnyTalker/config_af2v_1_3B.json" \
15 --det_thresh=0.15 \
16 --mode="pad" \
17 --use_half=True sh infer_a2v_1_3B_batch.sh--offload_model: Whether to offload the model to CPU after each model forward, reducing GPU memory usage.
--det_thresh: detection threshold for the InsightFace model; a lower value improves performance on abstract-style images.
--sample_guide_scale: recommended value is 4.5; applied to both text and audio.
--mode: select "pad" if every audio input track has already been zero-padded to a common length; select "concat" if you instead want the script to chain each speaker’s clips together and then zero-pad the non-speaker segments to reach a uniform length.
--use_half: Whether to enable half-precision (FP16) inference for faster acceleration.
python -m pip install -U yt-dlp1cd ./benchmark
2python download.pybenchmark/
├── audio_left # Audio for left speaker (zero-padded to full length)
├── audio_right # Audio for right speaker (zero-padded to full length)
├── speaker_duration.json # Start/end timestamps for each speaker
├── interact_11.mp4 # Example video
└── frames # Reference image supplied as the first video frame1# single video
2python calculate_interactivity.py --video interact_11.mp4
3
4# entire directory
5python calculate_interactivity.py --dir ./your_dirspeaker_duration.json.@article{zhong2025anytalker,
title={AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement},
author={Zhong, Zhizhou and Ji, Yicheng and Kong, Zhe and Liu, YiYing and Wang, Jiarui and Feng, Jiasun and Liu, Lupeng and Wang, Xiangyi and Li, Yanjia and She, Yuqing and Qin, Ying and Li, Huan and Mao, Shuiyang and Liu, Wei and Luo, Wenhan},
journal={arXiv preprint},
year={2025}
}