Views
No views yet
git clone git@github.com:Omni-Avatar/OmniAvatar.git
cd OmniAvatarpip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
# Optional to install flash_attn to accelerate attention computation
pip install flash_attn| Models | Download Link | Notes |
|---|---|---|
| Wan2.1-T2V-14B | 🤗 Huggingface | Base model for 14B |
| OmniAvatar model 14B | 🤗 Huggingface | Our LoRA and audio condition weights |
| Wav2Vec | 🤗 Huggingface | Audio encoder |
1mkdir pretrained_models
2pip install "huggingface_hub[cli]"
3huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./pretrained_models/Wan2.1-T2V-14B
4huggingface-cli download facebook/wav2vec2-base-960h --local-dir ./pretrained_models/wav2vec2-base-960h
5huggingface-cli download OmniAvatar/OmniAvatar-14B --local-dir ./pretrained_models/OmniAvatar-14B1OmniAvatar
2├── pretrained_models
3│ ├── Wan2.1-T2V-14B
4│ │ ├── ...
5│ ├── OmniAvatar-14B
6│ │ ├── config.json
7│ │ └── pytorch_model.pt
8│ └── wav2vec2-base-960h
9│ ├── ...1# 480p only for now
2torchrun --standalone --nproc_per_node=1 scripts/inference.py --config configs/inference.yaml --input_file examples/infer_samples.txtexamples/infer_samples.txt, and its format is [prompt]@@[img_path]@@[audio_path]. The recommended range for prompt and audio cfg is [4-6]. You can increase the audio cfg to achieve more consistent lip-sync.audio_scale=3 to control audio guidance separately. At this time, guidance_scale only controls prompts.num_steps range is [20-50], more steps bring higher quality. To use multi-gpu inference, just set sp_size=$GPU_NUM. To use TeaCache, you can set tea_cache_l1_thresh=0.14 , and the recommanded range is [0.05-0.15].use_fsdp=True and num_persistent_param_in_dit. An example command is as follows:torchrun --standalone --nproc_per_node=8 scripts/inference.py --config configs/inference.yaml --input_file examples/infer_samples.txt --hp=sp_size=8,max_tokens=30000,guidance_scale=4.5,overlap_frame=13,num_steps=25,use_fsdp=True,tea_cache_l1_thresh=0.14,num_persistent_param_in_dit=7000000000model_size | torch_dtype | GPU_NUM | use_fsdp | num_persistent_param_in_dit | Speed | Required VRAM |
|---|---|---|---|---|---|---|
| 14B | torch.bfloat16 | 1 | False | None (unlimited) | 16.0s/it | 36G |
| 14B | torch.bfloat16 | 1 | False | 7*10**9 (7B) | 19.4s/it | 21G |
| 14B | torch.bfloat16 | 1 | False | 0 | 22.1s/it | 8G |
| 14B | torch.bfloat16 | 4 | True | None (unlimited) | 4.8s/it | 14.3G |
30000 tokens for 480p videos. We found that using more tokens when inference can also have good results. You can try 60000, 80000. Overlap overlap_frame can be set as 1 or 13. 13 could have more coherent generation, but error propagation is more severe.[Description of first frame]- [Description of human behavior]-[Description of background (optional)]@misc{gan2025omniavatar,
title={OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation},
author={Qijun Gan and Ruizi Yang and Jianke Zhu and Shaofei Xue and Steven Hoi},
year={2025},
eprint={2506.18866},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2506.18866},
}