We propose a Masked 3D Diffusion Model (M3DDM) and a hybrid coarse-to-fine inference pipeline for video outpainting. Our method can not only generate high temporal consistency and reasonable outpainting results but also alleviate the problem of artifact accumulation in long video outpainting.
Make sure you have Anaconda or Miniconda installed before running this command. This is our testing environment, but it can also run on versions of PyTorch greater than 1.10.0.
Install the required dependencies from the requirements.txt file in this repository:
pip install -r requirements.txt
Downloads
Before you can run the project, you need to download the following:
Pre-trained Stable Diffusion Model Weights:
We used the VAE encoder and decoder inside Stable Diffusion Model. To get the pre-trained stable diffusion v1.5 weights, download them from the following link:
Our network architecture is based on modifications of the Stable Diffusion Model v1.5. To get the pre-trained model weights, download them from the following link:
target_ratio_list: This parameter specifies the aspect ratio for the output video. You can input a single value such as "1:1", "16:9", or "9:16", or you can input a list like "16:9,9:16". For better results, we recommend inputting a single value.
copy_original: This parameter specifies whether to replace the corresponding parts of the generated video with the original video. It is worth noting that the parts of the original video being replaced have already been resized to meet a resolution of 256.
Citation
If this repo is useful to you, please cite our paper.
bibtex
1@inproceedings{fan2023hierarchical,
2 title={Hierarchical Masked 3D Diffusion Model for Video Outpainting},
3 author={Fan, Fanda and Guo, Chaoxu and Gong, Litong and Wang, Biao and Ge, Tiezheng and Jiang, Yuning and Luo, Chunjie and Zhan, Jianfeng},
4 booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
5 pages={7890--7900},
6 year={2023}
7}
We heavily borrow the code from diffusers. Thanks for open-sourcing! We also gratefully acknowledge the Stable Diffusion for providing the SD1.5 Model Weights. Any third-party packages are owned by their respective authors and must be used under their respective licenses.
