1Zhejiang University 2Tencent Hunyuan 3Nanyang Technological University *Equal Contribution §Corresponding Authors †Project Leader
Work done during Kaihang Pan's internship at Tencent Hunyuan
🔥🔥🔥 News
📌 OmniWeaving is developed by the HunyuanVideo team and is built upon the latest HunyuanVideo-1.5 as the backbone. If you find our work useful, please consider giving this repository a like ❤️ and citing our paper~
🚀 April 3, 2026: We release the code
and model weights of OmniWeaving.
We propose OmniWeaving, an omni-level video generation model featuring powerful multimodal composition and reasoning-informed capabilities. By leveraging a massive-scale pretraining dataset that encompasses diverse compositional and reasoning-augmented scenarios, OmniWeaving learns to temporally bind interleaved text, multi-image, and video inputs while acting as an intelligent agent to infer complex user intentions for sophisticated video creation.
Furthermore, we introduce IntelligentVBench, the first comprehensive benchmark designed to rigorously assess next-level intelligent unified video generation. Extensive experiments demonstrate that OmniWeaving achieves SoTA performance among open-source unified models.
🏗 Model Architecture
Following the paper, OmniWeaving is built as an integrated MLLM + MMDiT + VAE framework for unified free-form video generation. The MLLM serves as the semantic parser for interleaved text, images, and video inputs, mapping them into a high-level semantic space and forwarding its hidden states through an MLP connector. The VAE acts as the visual tokenizer, compressing visual inputs into low-level latents, while the MMDiT uses these semantic conditions together with latent noise to generate semantically aligned, high-fidelity videos.
On this basis, we further introduce two extra improvements tailored for advanced reasoning and composition.
(1) Activating Thinking Mode of the MLLM: Direct MLLM encoding of interleaved visual-text inputs often yields semantic ambiguity due to weak intra-correlations and unclear video creation intents. We elevate the MLLM from a passive feature extractor to an active reasoner. By activating the thinking mode to generate intermediate reasoning steps, it autonomously deduces a semantically precise, enhanced prompt. The hidden states of this enhanced prompt are then forwarded alongside the original MLLM features to condition the MMDiT, effectively bridging the cognitive gap between abstract user intent and pixel-level generation.
(2) Hidden States DeepStacking: Compositional video generation involving multiple subjects or intricate scenes often relies on both low- and high-level semantic representations. Drawing inspiration from the DeepStacking mechanism in Qwen3-VL, we extract hidden states from a broader range of intermediate MLLM layers to capture a rich semantic spectrum spanning from fine-grained details to high-level abstractions. An MLP connector projects these multi-level features into the MMDiT embedding space. These projected features are then directly added to the corresponding hidden states within the first three layers of the MMDiT conditioning branch, effectively injecting multi-granular semantic guidance into the generative process.
OmniWeaving Architecture
Figure 1. Overview of the OmniWeaving architecture, which consists of an MLLM for multimodal understanding and an MMDiT for generation.
🚀 Supported Tasks
OmniWeaving is flexible in its input and output configurations, supporting a wide range of unified video generation tasks:
Task
Input Type
Output
Description
Demo Input
Demo Output
Text-to-Video (T2V)
Text 📝
Video 🎬
Generating a video from text prompts.
First-Frame-to-Video (I2V)
Image 🖼 + Text 📝
Video 🎬
Generating a video based on the first frame.
Key-Frames-to-Video
2 × Images 🖼 + Text 📝
Video 🎬
Generating a video conditioned on start and end frames.
Video-to-Video Editing
Video 🎬 + Text 📝
Video 🎬
Instruction-based video manipulation and stylization.
Reference-to-Video
Image 🖼 + Text 📝
Video 🎬
Single-subject reference-driven video generation.
Compositional Multi-Image-to-Video
2–4 × Images 🖼 + Text 📝
Video 🎬
Multi-subject compositional video generation.
Text-Image-Video-to-Video
Video 🎬 + Image 🖼 + Text 📝
Video 🎬
Generating a video conditioned on text, image, and video inputs.
Reasoning-Augmented Video Generation
Image(s) 🖼 + Text 📝
Reasoning 💭 + Video 🎬
Reasoning over user intent before generating the video.
OmniWeaving is built upon HunyuanVideo-1.5. The way to install dependencies is similar to HunyuanVideo-1.5. Specifically, you should install basic dependencies:
pip install -r requirements.txt
Additionally, install the attention libraries as needed (we use Flash Attention in practice):
Flash Attention: Install for faster inference and reduced GPU memory consumption. See Flash Attention for details.
Flex-Block-Attention: Required only for sparse attention to achieve faster inference:
Single- or multi-subject reference-driven video generation.
editing
Video-to-Video Editing
Instruction-based video manipulation and stylization.
tiv2v
Text-Image-Video-to-Video
Generate a video conditioned on text, image, and video inputs.
Among these, t2v, i2v, and interpolation can optionally enable thinking mode (--think) for Reasoning-Augmented Video Generation, where the MLLM first reasons over user intent before generating the video.
Common Configuration
All tasks share the following hyperparameters (configured at the top of generate.sh):
bash
1N_INFERENCE_GPU=82SEED=03ASPECT_RATIO=16:9
4MODEL_PATH=/path/to/OmniWeaving
5SAGE_ATTN=false ### Use Flash Attention6### SAGE_ATTN=true ### Use SageAttention7SPARSE_ATTN=false
8OVERLAP_GROUP_OFFLOADING=false
9ENABLE_CACHE=false
10CACHE_TYPE=deepcache
Tips: If your GPU memory is limited and you encounter OOM errors, try:
If you have limited CPU memory, disable overlapped group offloading by setting OVERLAP_GROUP_OFFLOADING=false.
Task-Specific Inference Scripts
1. Text-to-Video (t2v)
Generate a video from a text prompt.
bash
1PROMPT="Put Your Prompt Here"2NEGATIVE_PROMPT="overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion"3OUTPUT_PATH=./outputs/t2v.mp4
4torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \5 --task t2v \6 --prompt "$PROMPT"\7 --negative_prompt "$NEGATIVE_PROMPT"\8 --aspect_ratio $ASPECT_RATIO\9 --seed $SEED\10 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\11 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\12 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\13 --output_path $OUTPUT_PATH\14 --model_path $MODEL_PATH\15# --think \ # Optional: enable reasoning-augmented generation (see note below)
The --think flag activates the MLLM's thinking mode, in which it reasons over user intent and generates an enriched prompt before video generation. The --think flag is supported by t2v, i2v, and interpolation tasks.
2. First-Frame-to-Video (i2v)
Animate a first-frame image into a video guided by a text prompt.
Generate a video featuring one or more reference subjects. Provide one or more reference images via --ref_image_paths.
bash
1PROMPT="Put Your Prompt Here"2# Supports 1–4 reference images.3# For best results with multiple images, use the same aspect ratio across all images,4# as they will be center-cropped to match the size of the first image.5REF_IMAGE_PATHS=(/path/to/img1.png /path/to/img2.png ... /path/to/img4.png)# up to 4 input images6OUTPUT_PATH=./outputs/reference2v.mp4
7torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \8 --task reference2v \9 --prompt "$PROMPT"\10 --ref_image_paths "${REF_IMAGE_PATHS[@]}"\11 --aspect_ratio $ASPECT_RATIO\12 --seed $SEED\13 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\14 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\15 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\16 --output_path $OUTPUT_PATH\17 --model_path $MODEL_PATH
5. Video-to-Video Editing (editing)
Edit an existing video according to the text instruction (e.g., style transfer, object replacement).
bash
1PROMPT="Put Your Prompt Here"2CONDITION_VIDEO_PATH=/path/to/source_video.mp4
3OUTPUT_PATH=./outputs/editing.mp4
4# If you have pre-extracted VAE latents for the condition video, pass them via5# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.6torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \7 --task editing \8 --prompt "$PROMPT"\9 --condition_video_paths $CONDITION_VIDEO_PATH\10 --aspect_ratio $ASPECT_RATIO\11 --seed $SEED\12 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\13 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\14 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\15 --output_path $OUTPUT_PATH\16 --model_path $MODEL_PATH\17# --condition_video_latents_path /path/to/latents.pt # Optional: skip VAE encoding by providing pre-extracted latents
6. Text-Image-Video-to-Video (tiv2v)
Edit a video while incorporating reference subject images (e.g., insert a character from a reference image into a source video).
bash
1PROMPT="Put Your Prompt Here"2CONDITION_VIDEO_PATH=/path/to/source_video.mp4
3# Only one reference image is supported for tiv2v.4# For best results, use a reference image whose aspect ratio is close to the output video's aspect ratio.5REF_IMAGE_PATHS=(/path/to/ref_image.png)6OUTPUT_PATH=./outputs/tiv2v.mp4
78# If you have pre-extracted VAE latents for the condition video, pass them via9# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.10torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \11 --task tiv2v \12 --prompt "$PROMPT"\13 --condition_video_paths $CONDITION_VIDEO_PATH\14 --ref_image_paths "${REF_IMAGE_PATHS[@]}"\15 --aspect_ratio $ASPECT_RATIO\16 --seed $SEED\17 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\18 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\19 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\20 --output_path $OUTPUT_PATH\21 --model_path $MODEL_PATH\22# --condition_video_latents_path /path/to/latents.pt # Optional: skip VAE encoding by providing pre-extracted latents
Other Optional Arguments
The arguments below can be appended to any of the task commands above for further customization:
Argument
Type
Default
Description
--negative_prompt
str
""
Negative prompt for video generation. Default is empty. Setting a negative prompt (e.g., 'overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion') can improve quality, especially for tasks like t2v.
--num_inference_steps
int
50
Number of denoising steps
--video_length
int
81
Number of frames to generate
--fps
int
Auto
Output FPS (default: 16 for ≤81 frames, 24 for >81 frames)
--dtype
str
bf16
Data type: bf16 or fp32
--offloading
bool
true
Enable CPU offloading
--group_offloading
bool
None
Enable group offloading (auto-enabled with offloading)
--pipeline_config
str
omniweaving
Pipeline configuration preset that controls guidance_scale and flow_shift. Available presets: omniweaving (guidance_scale=6.0, flow_shift=7.0), omniweaving2 (guidance_scale=6.0, flow_shift=5.0).
Tuning guidance_scale / flow_shift: You can switch presets via --pipeline_config (e.g., --pipeline_config omniweaving2). If the available presets do not meet your needs, you can add a new key to the PIPELINE_CONFIGS dict in hyvideo/commons/__init__.py with your desired values. We recommend guidance_scale=6.0 with flow_shift=5.0 or 7.0.
📚 Citation
If you find our work helpful, please consider giving us a like ❤️ on this repo and citing our papers as follows:
OmniWeaving
bibtex
1@article{pan2026omniweaving,
2 title={OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning},
3 author={Pan, Kaihang and Tian, Qi and Zhang, Jianwei and Kong, Weijie and Xiong, Jiangfeng and Long, Yanxin and Zhang, Shixue and Qiu, Haiyi and Wang, Tan and Lv, Zheqi and others},
4 journal={arXiv preprint arXiv:2603.24458},
5 year={2026}
6}
HunyuanVideo 1.5
bibtex
1@article{wu2025hunyuanvideo,
2 title={Hunyuanvideo 1.5 technical report},
3 author={Wu, Bing and Zou, Chang and Li, Changlin and Huang, Duojun and Yang, Fang and Tan, Hao and Peng, Jack and Wu, Jianbing and Xiong, Jiangfeng and Jiang, Jie and others},
4 journal={arXiv preprint arXiv:2511.18870},
5 year={2025}
6}
