(*Works done during the internship at Bytedance Intelligent Creation, ✝Project lead, ✉Corresponding author)
From University of Science and Technology of China, ByteDance Intelligent Creation and Yuanshi Inc.
🔆 Introduction
TL;DR: We present Mask²DiT, a novel dual-mask-based diffusion transformer designed for multi-scene long video generation. It enables both synthesizing a fixed number of scenes and auto-regressively expanding new scenes, advancing the scalability and continuity of long video synthesis.
⭐⭐ Fixed-Scene Video Generation.
Videos generated with a fixed number of scenes using Mask²DiT. Each scene maintains coherent appearance and motion across temporal boundaries.
⭐⭐ Auto-Regressive Scene Expansion.
Mask²DiT extends multi-scene narratives auto-regressively, producing long and coherent videos with evolving context.
📝 Changelog
[2025.10.15]: 🔥🔥 Release the code and checkpoint.
[2025.03.26]: 🔥🔥 Release the arxiv paper and project page.
🧩 Inference
We provide two inference pipelines for long video generation:
🎬 Fixed-Scene Generation — generate videos with a fixed number of scenes.
🔄 Auto-Regressive Scene Expansion — expand scenes continuously based on previous context.
1️⃣ Prepare Pretrained Model
Download the pretrained model from Hugging Face and place it under:
./models/
2️⃣ Environment Setup
We recommend using a virtual environment to install the required dependencies. You can create a virtual environment using conda as follows:
This mode auto-regressively extends the video while maintaining global temporal consistency, storing the expanded video under samples/mask2dit-cogvideox-5b-autoregressive-scene-expansion.
🧑🏫 Training
1️⃣ Prepare Training Data
Please prepare your datasets following the provided examples:
datasets/pretrain.csv → used for pretraining
datasets/sft.json → used for supervised fine-tuning (SFT)
💡 You can modify these template files to fit your own dataset paths and captions.
2️⃣ Pretraining
We pretrain Mask²DiT using the provided datasets/pretrain.csv. Use the following script to start pretraining:
After pretraining, we fine-tune Mask²DiT using the datasets/sft.json. Use the following script to start SFT:
bash scripts/cogvideox_fun/train_mask2dit_sft.sh
🙏 Acknowledgement
This project is built upon the open-source repository
VideoX-Fun.
We sincerely thank the original authors for their excellent work and open-source contributions.
Bibtex
If you find our work useful for your research, welcome to cite our work using the following BibTeX:
bibtex
1@inproceedings{qi2025mask,
2 title={Mask\^{} 2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation},
3 author={Qi, Tianhao and Yuan, Jianlong and Feng, Wanquan and Fang, Shancheng and Liu, Jiawei and Zhou, SiYu and He, Qian and Xie, Hongtao and Zhang, Yongdong},
4 booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
5 pages={18837--18846},
6 year={2025}
7}
