High-performance Variational Autoencoder (VAE) component for the WAN (World Anything Now) video generation system. This VAE provides efficient latent space encoding and decoding for video content, enabling high-quality video generation with reduced computational requirements.
Model Description
The WAN22-VAE is a specialized variational autoencoder designed for video content processing in the WAN video generation pipeline. It compresses video frames into a compact latent representation and reconstructs them with high fidelity, enabling efficient text-to-video and image-to-video generation workflows.
Key Capabilities
Video Compression: Efficient encoding of video frames into latent space representations
High Fidelity Reconstruction: Accurate decoding back to pixel space with minimal quality loss
Temporal Coherence: Maintains consistency across video frames during encoding/decoding
Memory Efficient: Reduces VRAM requirements during video generation inference
Compatible Pipeline Integration: Seamlessly integrates with WAN video generation models
Technical Highlights
Optimized architecture for temporal video data processing
Supports various frame rates and resolutions
Low latency encoding/decoding for real-time applications
Precision-optimized for stable inference on consumer hardware
Repository Contents
wan22-vae/
└── vae/
└── wan/
└── wan22-vae.safetensors # 1.34 GB - Main VAE model weights
Total Repository Size: ~1.4 GB
File Details
File
Size
Description
wan22-vae.safetensors
1.34 GB
WAN22 VAE model weights in safetensors format
Hardware Requirements
Minimum Requirements
VRAM: 2 GB (VAE inference only)
System RAM: 4 GB
Disk Space: 1.5 GB free space
GPU: CUDA-compatible GPU (NVIDIA) or compatible accelerator
Recommended Specifications
VRAM: 4+ GB for comfortable operation with video generation pipeline
System RAM: 16+ GB
GPU: NVIDIA RTX 3060 or better
Storage: SSD for faster model loading
Performance Notes
VAE operations are typically memory-bound rather than compute-bound
Larger batch sizes require proportionally more VRAM
CPU inference is possible but significantly slower (30-50x)
Usage Examples
Basic Usage with Diffusers
python
1import torch
2from diffusers import AutoencoderKL
34# Load the WAN22 VAE5vae_path =r"E:\huggingface\wan22-vae\vae\wan"6vae = AutoencoderKL.from_pretrained(7 vae_path,8 torch_dtype=torch.float16
9)1011# Move to GPU12device ="cuda"if torch.cuda.is_available()else"cpu"13vae = vae.to(device)1415# Encode video frames to latent space16# video_frames: tensor of shape [batch, channels, height, width]17with torch.no_grad():18 latents = vae.encode(video_frames).latent_dist.sample()19 latents = latents * vae.config.scaling_factor
2021# Decode latents back to pixel space22with torch.no_grad():23 decoded_frames = vae.decode(latents / vae.config.scaling_factor).sample
Integration with WAN Video Generation Pipeline
python
1import torch
2from diffusers import DiffusionPipeline
34# Load WAN video generation pipeline with custom VAE5pipeline = DiffusionPipeline.from_pretrained(6"wan-model/wan-base",# Replace with actual WAN model path7 vae=vae,# Use the loaded WAN22-VAE8 torch_dtype=torch.float16
9)10pipeline = pipeline.to("cuda")1112# Generate video from text prompt13prompt ="A serene sunset over mountains with flowing clouds"14video_frames = pipeline(15 prompt=prompt,16 num_frames=24,17 height=512,18 width=512,19 num_inference_steps=5020).frames
Memory-Efficient Video Processing
python
1import torch
23# Enable memory-efficient attention for large videos4vae.enable_xformers_memory_efficient_attention()56# Process video in smaller chunks7defencode_video_chunks(video_tensor, chunk_size=8):8"""Encode video frames in chunks to reduce VRAM usage"""9 latents =[]10for i inrange(0, video_tensor.shape[0], chunk_size):11 chunk = video_tensor[i:i+chunk_size].to(device)12with torch.no_grad():13 chunk_latents = vae.encode(chunk).latent_dist.sample()14 latents.append(chunk_latents.cpu())15return torch.cat(latents, dim=0)
Custom Latent Space Manipulation
python
1import torch
2import numpy as np
34# Encode input video5latents = vae.encode(input_frames).latent_dist.sample()67# Apply transformations in latent space (e.g., interpolation)8latents_start = latents[0]9latents_end = latents[-1]1011# Create smooth interpolation between frames12interpolated_latents =[]13for alpha in np.linspace(0,1,16):14 interpolated =(1- alpha)* latents_start + alpha * latents_end
15 interpolated_latents.append(interpolated)1617# Decode interpolated latents18smooth_video = vae.decode(torch.stack(interpolated_latents)).sample
Model Specifications
Architecture Details
Model Type: Variational Autoencoder (VAE)
Architecture: Convolutional encoder-decoder with KL divergence regularization
Input Format: Video frames (RGB or grayscale)
Latent Dimensions: Compressed spatial resolution with channel expansion
Activation Functions: Mixed (SiLU, tanh for output)
Framework: PyTorch-based, compatible with Diffusers library
Parameters: ~335M parameters (1.34 GB in FP32)
Compression Ratio: Approximately 8x spatial compression per dimension
Supported Input Resolutions
Standard: 512x512, 768x768
Extended: 256x256 to 1024x1024 (depending on VRAM)
Aspect Ratios: Square and common video ratios (16:9, 4:3)
Performance Tips and Optimization
Memory Optimization
python
1# Enable gradient checkpointing for training (if fine-tuning)2vae.enable_gradient_checkpointing()34# Use float16 for inference to reduce VRAM usage5vae = vae.half()67# Process frames in batches8batch_size =4# Adjust based on available VRAM
Speed Optimization
python
1# Compile model with torch.compile (PyTorch 2.0+)2vae = torch.compile(vae, mode="reduce-overhead")34# Use channels_last memory format for better performance5vae = vae.to(memory_format=torch.channels_last)67# Enable TF32 on Ampere+ GPUs8torch.backends.cuda.matmul.allow_tf32 =True9torch.backends.cudnn.allow_tf32 =True
Quality vs Speed Trade-offs
High Quality: Use FP32 precision, larger batch sizes, disable tiling
Fast Inference: FP16 precision, smaller batches (1-2 frames), enable tiling
Best Practices
Always use safetensors format for security and compatibility
Monitor VRAM usage with torch.cuda.memory_allocated()
Clear cache between large operations: torch.cuda.empty_cache()
Use mixed precision training if fine-tuning the VAE
Validate reconstruction quality with perceptual metrics (LPIPS, SSIM)
License
This model is released under a custom WAN license. Please review the license terms before use:
Commercial Use: Subject to WAN license terms
Research Use: Generally permitted with attribution
Redistribution: Refer to original WAN model license
Modifications: Check license for derivative work permissions
For complete license details, refer to the original WAN model repository or license documentation.
Citation
If you use this VAE in your research or projects, please cite:
bibtex
1@misc{wan22-vae,
2 title={WAN22 VAE: Video Variational Autoencoder for WAN Video Generation},
3 author={WAN Model Team},
4 year={2024},
5 publisher={Hugging Face},
6 howpublished={\url{https://huggingface.co/wan-model/wan22-vae}}
7}
