Fun-CineForge contains an end-to-end dataset pipeline for producing large-scale dubbing datasets and an MLLM-based dubbing model designed for diverse cinematic scenes. Using this pipeline, we constructed the first large-scale Chinese television dubbing dataset CineDub-CN, which includes rich annotations and diverse scenes. In monologue, narration, dialogue, and multi-speaker scenes, our dubbing model consistently outperforms state-of-the-art methods in terms of audio quality, lip-sync, timbre transition, and instruction following.
If you want to produce your own data,
we recommend that you refer to the following requirements to collect the corresponding movies or television series.
Video source: TV dramas or movies, non documentaries, with more monologues or dialogue scenes, clear and unobstructed faces (such as without masks and veils).
Speech Requirements: Standard pronunciation, clear articulation, prominent human voice. Avoid materials with strong dialects, excessive background noise, or strong colloquialism.
Image Requirements: High resolution, clear facial details, sufficient lighting, avoiding extremely dark or strong backlit scenes.
How to use
[1] Standardize video format and name; trim the beginning and end of long videos; extract the audio from the trimmed video. (default is to trim 10 seconds from both the beginning and end.)
[3] VideoClipper. For long videos, VideoClipper is used to obtain sentence-level subtitle files and clip the long video into segments based on timestamps. Now it supports bilingualism in both Chinese and English. Below is an example in Chinese. It is recommended to use gpu acceleration for English.
Video duration limit and check for cleanup. (Without --execute, only pre-deleted files will be printed. After checking, add --execute to confirm the deletion.)
[4] Speaker Diarization. Multimodal active speaker recognition obtains RTTM files; identifies the speaker's facial frames, extracts frame-level speaker face and lip raw data.
[5] Multimodal CoT Correction. Based on general-purpose MLLMs, the system uses audio, ASR text, and RTTM files as input. It leverages Chain-of-Thought (CoT) reasoning to extract clues and corrects the results of the specialized models. It also annotates character age, gender, and vocal timbre. Experimental results show that this strategy reduces the CER from 4.53% to 0.94% and the speaker diarization error rate from 8.38% to 1.20%, achieving quality comparable to or even better than manual transcription. Adding the --resume enables breakpoint COT inference to prevent wasted resources from repeated COT inferences. Now supports both Chinese and English.
shell
1python cot.py --root_dir datasets/clean/zh --lang zh --provider google --model gemini-3-pro-preview --api_key xxx --resume
2python cot.py --root_dir datasets/clean/en --lang en --provider google --model gemini-3-pro-preview --api_key xxx --resume
The construction of the dataset retrieval file will read all production data, perform bidirectional verification of script content and speaker separation results.
We've open-sourced the inference code and the infer.sh script, and provided some test cases in the data folder for your experience. Inference requires a consumer-grade GPU. Run the following command:
shell
1cd exps
2bash infer.sh
The API for multi-speaker dubbing from raw videos and SRT scripts is under development ...
Recent Updates 🚀
2025/12/18: Fun-CineForge dataset pipeline toolkit is online! 🔥
2026/01/19: Chinese demo samples and CineDub-CN dataset samples released. 🔥
2026/01/25: Fix some environmental and operational issues.
2026/02/09: Optimized the data pipeline and added support for English videos.
2026/03/05: English demo samples and CineDub-EN dataset samples released. 🔥
2026/03/16: Open source inference code and checkpoints. 🔥
Publication 📚
If you use our dataset or code, please cite the following paper:
@misc{liu2026funcineforgeunifieddatasettoolkit,
title={FunCineForge: A Unified Dataset Toolkit and Model for Zero-Shot Movie Dubbing in Diverse Cinematic Scenes},
author={Jiaxuan Liu and Yang Xiang and Han Zhao and Xiangang Li and Zhenhua Ling},
year={2026},
eprint={2601.14777},
archivePrefix={arXiv},
primaryClass={cs.CV},
}
Comminicate 🍟
The Fun-CineForge open-source project is developed and maintained by the Tongyi Lab Speech Team and a student from NERCSLIP, University of Science and Technology of China.
We welcome you to participate in discussions on Fun-CineForge GitHub Issues or contact us for collaborative development.
For any questions, you can contact the developer.
⭐ Hope you will support Fun-CineForge. Thank you.
Disclaimer
This repository contains research artifacts:
⚠️ Currently not a commercial product of Tongyi Lab.
⚠️ Released for academic research / cutting-edge exploration purposes
⚠️ CineDub Dataset samples are subject to specific license terms.
