TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting

Installation

git clone git@github.com:Fictionarry/TalkingGaussian.git --recursive

conda env create --file environment.yml
conda activate talking_gaussian
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
pip install tensorflow-gpu==2.8.0

Preparation

• Prepare face-parsing model and the 3DMM model for head pose estimation.
  bash scripts/prepare.sh
• Download 3DMM model from Basel Face Model 2009:
  bash

  1# 1. copy 01_MorphableModel.mat to data_util/face_tracking/3DMM/ 2# 2. run following 3cd data_utils/face_tracking 4python convert_BFM.py
• Prepare the environment for EasyPortrait:
  bash

  1# prepare mmcv 2conda activate talking_gaussian 3pip install -U openmim 4mim install mmcv-full==1.7.1 5 6# download model weight 7cd data_utils/easyportrait 8wget "https://n-ws-620xz-pd11.s3pd11.sbercloud.ru/b-ws-620xz-pd11-jux/easyportrait/experiments/models/fpn-fp-512.pth"

Usage

Important Notice

• This code is provided for research purposes only. The author makes no warranties, express or implied, as to the accuracy, completeness, or fitness for a particular purpose of the code. Use this code at your own risk.
• The author explicitly prohibits the use of this code for any malicious or illegal activities. By using this code, you agree to comply with all applicable laws and regulations, and you agree not to use it to harm others or to perform any actions that would be considered unethical or illegal.
• The author will not be responsible for any damages, losses, or issues that arise from the use of this code.
• Users are encouraged to use this code responsibly and ethically.

Video Dataset

Pre-processing Training Video

• Put training video under data/<ID>/<ID>.mp4.
  The video must be 25FPS, with all frames containing the talking person. The resolution should be about 512x512, and duration about 1-5 min.
• Run script to process the video.
  python data_utils/process.py data/<ID>/<ID>.mp4
• Obtain Action Units
  Run FeatureExtraction in OpenFace, rename and move the output CSV file to data/<ID>/au.csv.
• Generate tooth masks
  bash

  1export PYTHONPATH=./data_utils/easyportrait 2python ./data_utils/easyportrait/create_teeth_mask.py ./data/<ID>

Audio Pre-process

• DeepSpeech
  python data_utils/deepspeech_features/extract_ds_features.py --input data/<name>.wav # saved to data/<name>.npy

• HuBERT
  Similar to ER-NeRF, HuBERT is also available. Recommended for situations if the audio is not in English.
  Specify --audio_extractor hubert when training and testing.
  python data_utils/hubert.py --wav data/<name>.wav # save to data/<name>_hu.npy

Train

Test

Inference with target audio

Citation

@article{li2024talkinggaussian,
    title={TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting}, 
    author={Jiahe Li and Jiawei Zhang and Xiao Bai and Jin Zheng and Xin Ning and Jun Zhou and Lin Gu},
    journal={arXiv preprint arXiv:2404.15264},
    year={2024}
}

Acknowledgement