Views
No views yet
talker-t2av/
model.safetensors ← AR backbone (Qwen3-0.6B) + dual diffusion heads
+ Patch Transformer Encoder + Stop Predictor
config.json
chat_template.jinja
tokenizer.json
tokenizer_config.json
whisperx-vae/
model.ckpt ← WhisperX-VAE audio autoencoder
(32-d, 25 Hz; Whisper-Large-v3 encoder + DAC backbone)lia_x/ in the GitHub repo — only the
lia-x.pt weight file needs to be fetched separately from
wyhsirius/LIA-X. The WavLM-Large
fine-tuned speaker encoder (wavlm_large_finetune.pth) similarly ships
its code under speaker_verification/; only the .pth weights need to
be obtained from
Microsoft UniSpeech.1git clone https://github.com/zhenye234/Talker-T2AV.git
2cd Talker-T2AV
3
4# put the HF-hosted weights in place
5huggingface-cli download HKUSTAudio/Talker-T2AV --local-dir ./hf_weights
6export CHECKPOINT_DIR="$(pwd)/hf_weights/talker-t2av"
7export WHISPERVAE_CKPT="$(pwd)/hf_weights/whisperx-vae/model.ckpt"
8
9# the two extra weight files (code already vendored — no need to clone the repos)
10export LIAX_CKPT=/path/to/lia-x.pt
11export WAVLM_CKPT=/path/to/wavlm_large_finetune.pth
12
13python infer.py1@misc{ye2026talkert2avjointtalkingaudiovideo,
2 title={Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling},
3 author={Zhen Ye and Xu Tan and Aoxiong Yin and Hongzhan Lin and Guangyan Zhang and Peiwen Sun and Yiming Li and Chi-Min Chan and Wei Ye and Shikun Zhang and Wei Xue},
4 year={2026},
5 eprint={2604.23586},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2604.23586},
9}