FP8 Quantized Checkpoints
These are FP8-quantized versions of the original OmniWeaving checkpoints, converted from bf16 to torch.float8_e4m3fn precision.
The transformer achieves greater compression because many non-floating-point tensors (int/bool indices, embeddings) are kept at full precision, while the text encoder is dominated by float weights.
Performance
Metric bf16 (original) FP8
Inference speed Baseline Same (weights auto-cast to bf16 at load time)
Video quality Reference Near-identical (minimal precision loss)
Loading RAM ~48 GB peak ~16 GB peak
Inference VRAM ~50-60 GB ~50-60 GB (unchanged)
Who Should Use This
Users with limited system RAM who can't load the full bf16 checkpoint
Users who want faster download times (16 GB vs 48 GB)
Users who want a smaller disk footprint
Who Should NOT Use This
Users expecting lower VRAM usage — use --quantize_4bit instead
Users who need bit-exact reproducibility with the original bf16 model
How to Use
Replace the original checkpoint files with these FP8 versions:
ckpts/
├── text_encoder/ckpt/text_encoder_model.safetensors ← replace with FP8 version
└── transformer/diffusion_pytorch_model.safetensors ← replace with FP8 version
Or rename the _fp8 files to match the originals. The loading code is identical — no code changes needed.
OmniWeaving Logo
icon
OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning
1Zhejiang University 2Tencent Hunyuan 3Nanyang Technological University *Equal Contribution §Corresponding Authors †Project Leader
Work done during Kaihang Pan's internship at Tencent Hunyuan
🔥🔥🔥 News
📌 OmniWeaving is developed by the HunyuanVideo team and is built upon the latest HunyuanVideo-1.5 as the backbone. If you find our work useful, please consider giving this repository a like ❤️ and citing our paper~
🚀 April 3, 2026: We release the code
and model weights of OmniWeaving.
While proprietary systems such as Seedance-2.0 have achieved remarkable success in omni-capable video generation, open-source alternatives significantly lag behind. To bridge this gap, we propose
OmniWeaving
OmniWeaving, an omni-level video generation model featuring powerful multimodal composition and reasoning-informed capabilities. By leveraging a massive-scale pretraining dataset that encompasses diverse compositional and reasoning-augmented scenarios, OmniWeaving learns to temporally bind interleaved text, multi-image, and video inputs while acting as an intelligent agent to infer complex user intentions for sophisticated video creation.
Furthermore, we introduce IntelligentVBench, the first comprehensive benchmark designed to rigorously assess next-level intelligent unified video generation. Extensive experiments demonstrate that OmniWeaving achieves SoTA performance among open-source unified models.
🏗 Model Architecture
Following the paper, OmniWeaving is built as an integrated MLLM + MMDiT + VAE framework for unified free-form video generation. The MLLM serves as the semantic parser for interleaved text, images, and video inputs, mapping them into a high-level semantic space and forwarding its hidden states through an MLP connector. The VAE acts as the visual tokenizer, compressing visual inputs into low-level latents, while the MMDiT uses these semantic conditions together with latent noise to generate semantically aligned, high-fidelity videos.
On this basis, we further introduce two extra improvements tailored for advanced reasoning and composition.
(1) Activating Thinking Mode of the MLLM: Direct MLLM encoding of interleaved visual-text inputs often yields semantic ambiguity due to weak intra-correlations and unclear video creation intents. We elevate the MLLM from a passive feature extractor to an active reasoner. By activating the thinking mode to generate intermediate reasoning steps, it autonomously deduces a semantically precise, enhanced prompt. The hidden states of this enhanced prompt are then forwarded alongside the original MLLM features to condition the MMDiT, effectively bridging the cognitive gap between abstract user intent and pixel-level generation.
(2) Hidden States DeepStacking: Compositional video generation involving multiple subjects or intricate scenes often relies on both low- and high-level semantic representations. Drawing inspiration from the DeepStacking mechanism in Qwen3-VL, we extract hidden states from a broader range of intermediate MLLM layers to capture a rich semantic spectrum spanning from fine-grained details to high-level abstractions. An MLP connector projects these multi-level features into the MMDiT embedding space. These projected features are then directly added to the corresponding hidden states within the first three layers of the MMDiT conditioning branch, effectively injecting multi-granular semantic guidance into the generative process.
OmniWeaving Architecture
Figure 1. Overview of the OmniWeaving architecture, which consists of an MLLM for multimodal understanding and an MMDiT for generation.
🚀 Supported Tasks
OmniWeaving is flexible in its input and output configurations, supporting a wide range of unified video generation tasks:
Task
Input Type
Output
Description
Demo Input
Demo Output
Text-to-Video (T2V)
Text 📝
Video 🎬
Generating a video from text prompts.
First-Frame-to-Video (I2V)
Image 🖼 + Text 📝
Video 🎬
Generating a video based on the first frame.
Key-Frames-to-Video
2 × Images 🖼 + Text 📝
Video 🎬
Generating a video conditioned on start and end frames.
Video-to-Video Editing
Video 🎬 + Text 📝
Video 🎬
Instruction-based video manipulation and stylization.
Reference-to-Video
Image 🖼 + Text 📝
Video 🎬
Single-subject reference-driven video generation.
Compositional Multi-Image-to-Video
2–4 × Images 🖼 + Text 📝
Video 🎬
Multi-subject compositional video generation.
Text-Image-Video-to-Video
Video 🎬 + Image 🖼 + Text 📝
Video 🎬
Generating a video conditioned on text, image, and video inputs.
Reasoning-Augmented Video Generation
Image(s) 🖼 + Text 📝
Reasoning 💭 + Video 🎬
Reasoning over user intent before generating the video.
OmniWeaving is built upon HunyuanVideo-1.5. The way to install dependencies is similar to HunyuanVideo-1.5. Specifically, you should install basic dependencies:
pip install -r requirements.txt
Additionally, install the attention libraries as needed (we use Flash Attention in practice):
Flash Attention: Install for faster inference and reduced GPU memory consumption. See Flash Attention for details.
Flex-Block-Attention: Required only for sparse attention to achieve faster inference:
Single- or multi-subject reference-driven video generation.
editing
Video-to-Video Editing
Instruction-based video manipulation and stylization.
tiv2v
Text-Image-Video-to-Video
Generate a video conditioned on text, image, and video inputs.
Among these, t2v, i2v, and interpolation can optionally enable thinking mode (--think) for Reasoning-Augmented Video Generation, where the MLLM first reasons over user intent before generating the video.
Common Configuration
All tasks share the following hyperparameters (configured at the top of generate.sh):
bash
1N_INFERENCE_GPU=82SEED=03ASPECT_RATIO=16:9
4MODEL_PATH=/path/to/OmniWeaving
56SAGE_ATTN=false ### Use Flash Attention7### SAGE_ATTN=true ### Use SageAttention8SPARSE_ATTN=false
9OVERLAP_GROUP_OFFLOADING=false
10ENABLE_CACHE=false
11CACHE_TYPE=deepcache
Tips: If your GPU memory is limited and you encounter OOM errors, try:
If you have limited CPU memory, disable overlapped group offloading by setting OVERLAP_GROUP_OFFLOADING=false.
Task-Specific Inference Scripts
1. Text-to-Video (t2v)
Generate a video from a text prompt.
bash
1PROMPT="Put Your Prompt Here"2NEGATIVE_PROMPT="overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion"3OUTPUT_PATH=./outputs/t2v.mp4
45torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \6 --task t2v \7 --prompt "$PROMPT"\8 --negative_prompt "$NEGATIVE_PROMPT"\9 --aspect_ratio $ASPECT_RATIO\10 --seed $SEED\11 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\12 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\13 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\14 --output_path $OUTPUT_PATH\15 --model_path $MODEL_PATH\16# --think \ # Optional: enable reasoning-augmented generation (see note below)
The --think flag activates the MLLM's thinking mode, in which it reasons over user intent and generates an enriched prompt before video generation. The --think flag is supported by t2v, i2v, and interpolation tasks.
2. First-Frame-to-Video (i2v)
Animate a first-frame image into a video guided by a text prompt.
Generate a video featuring one or more reference subjects. Provide one or more reference images via --ref_image_paths.
bash
1PROMPT="Put Your Prompt Here"2# Supports 1–4 reference images.3# For best results with multiple images, use the same aspect ratio across all images,4# as they will be center-cropped to match the size of the first image.5REF_IMAGE_PATHS=(/path/to/img1.png /path/to/img2.png ... /path/to/img4.png)# up to 4 input images6OUTPUT_PATH=./outputs/reference2v.mp4
78torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \9 --task reference2v \10 --prompt "$PROMPT"\11 --ref_image_paths "${REF_IMAGE_PATHS[@]}"\12 --aspect_ratio $ASPECT_RATIO\13 --seed $SEED\14 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\15 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\16 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\17 --output_path $OUTPUT_PATH\18 --model_path $MODEL_PATH
5. Video-to-Video Editing (editing)
Edit an existing video according to the text instruction (e.g., style transfer, object replacement).
bash
1PROMPT="Put Your Prompt Here"2CONDITION_VIDEO_PATH=/path/to/source_video.mp4
3OUTPUT_PATH=./outputs/editing.mp4
45# If you have pre-extracted VAE latents for the condition video, pass them via6# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.7torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \8 --task editing \9 --prompt "$PROMPT"\10 --condition_video_paths $CONDITION_VIDEO_PATH\11 --aspect_ratio $ASPECT_RATIO\12 --seed $SEED\13 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\14 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\15 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\16 --output_path $OUTPUT_PATH\17 --model_path $MODEL_PATH\18# --condition_video_latents_path /path/to/latents.pt # Optional: skip VAE encoding by providing pre-extracted latents
6. Text-Image-Video-to-Video (tiv2v)
Edit a video while incorporating reference subject images (e.g., insert a character from a reference image into a source video).
bash
1PROMPT="Put Your Prompt Here"2CONDITION_VIDEO_PATH=/path/to/source_video.mp4
3# Only one reference image is supported for tiv2v.4# For best results, use a reference image whose aspect ratio is close to the output video's aspect ratio.5REF_IMAGE_PATHS=(/path/to/ref_image.png)6OUTPUT_PATH=./outputs/tiv2v.mp4
78# If you have pre-extracted VAE latents for the condition video, pass them via9# --condition_video_latents_path /path/to/latents.pt to skip VAE encoding at inference.10torchrun --nproc_per_node=$N_INFERENCE_GPU generate.py \11 --task tiv2v \12 --prompt "$PROMPT"\13 --condition_video_paths $CONDITION_VIDEO_PATH\14 --ref_image_paths "${REF_IMAGE_PATHS[@]}"\15 --aspect_ratio $ASPECT_RATIO\16 --seed $SEED\17 --sparse_attn $SPARSE_ATTN --use_sageattn $SAGE_ATTN\18 --enable_cache $ENABLE_CACHE --cache_type $CACHE_TYPE\19 --overlap_group_offloading $OVERLAP_GROUP_OFFLOADING\20 --output_path $OUTPUT_PATH\21 --model_path $MODEL_PATH\22# --condition_video_latents_path /path/to/latents.pt # Optional: skip VAE encoding by providing pre-extracted latents
Other Optional Arguments
The arguments below can be appended to any of the task commands above for further customization:
Argument
Type
Default
Description
--negative_prompt
str
""
Negative prompt for video generation. Default is empty. Setting a negative prompt (e.g., 'overexposed, low quality, deformation, a poor composition, bad hands, bad teeth, bad eyes, bad limbs, distortion') can improve quality, especially for tasks like t2v.
--num_inference_steps
int
50
Number of denoising steps
--video_length
int
81
Number of frames to generate
--fps
int
Auto
Output FPS (default: 16 for ≤81 frames, 24 for >81 frames)
--dtype
str
bf16
Data type: bf16 or fp32
--offloading
bool
true
Enable CPU offloading
--group_offloading
bool
None
Enable group offloading (auto-enabled with offloading)
--pipeline_config
str
omniweaving
Pipeline configuration preset that controls guidance_scale and flow_shift. Available presets: omniweaving (guidance_scale=6.0, flow_shift=7.0), omniweaving2 (guidance_scale=6.0, flow_shift=5.0).
Tuning guidance_scale / flow_shift: You can switch presets via --pipeline_config (e.g., --pipeline_config omniweaving2). If the available presets do not meet your needs, you can add a new key to the PIPELINE_CONFIGS dict in hyvideo/commons/__init__.py with your desired values. We recommend guidance_scale=6.0 with flow_shift=5.0 or 7.0.
📚 Citation
If you find our work helpful, please consider giving us a like ❤️ on this repo and citing our papers as follows:
OmniWeaving
bibtex
1@article{pan2026omniweaving,
2 title={OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning},
3 author={Pan, Kaihang and Tian, Qi and Zhang, Jianwei and Kong, Weijie and Xiong, Jiangfeng and Long, Yanxin and Zhang, Shixue and Qiu, Haiyi and Wang, Tan and Lv, Zheqi and others},
4 journal={arXiv preprint arXiv:2603.24458},
5 year={2026}
6}
HunyuanVideo 1.5
bibtex
1@article{wu2025hunyuanvideo,
2 title={Hunyuanvideo 1.5 technical report},
3 author={Wu, Bing and Zou, Chang and Li, Changlin and Huang, Duojun and Yang, Fang and Tan, Hao and Peng, Jack and Wu, Jianbing and Xiong, Jiangfeng and Jiang, Jie and others},
4 journal={arXiv preprint arXiv:2511.18870},
5 year={2025}
6}
