Views
No views yet
| 10s, 768p, 24fps | 5s, 768p, 24fps | Image-to-video |
|---|---|---|
2024.11.13 🚀🚀🚀 We release the 768p miniFLUX checkpoint (up to 10s).We have switched the model structure from SD3 to a mini FLUX to fix human structure issues, please try our 1024p image checkpoint, 384p video checkpoint (up to 5s) and 768p video checkpoint (up to 10s). The new miniflux model shows great improvement on human structure and motion stability
2024.10.29 ⚡️⚡️⚡️ We release training code and new model checkpoints with FLUX structure trained from scratch.1git clone https://github.com/jy0205/Pyramid-Flow
2cd Pyramid-Flow
3
4# create env using conda
5conda create -n pyramid python==3.8.10
6conda activate pyramid
7pip install -r requirements.txt1from huggingface_hub import snapshot_download
2
3model_path = 'PATH' # The local directory to save downloaded checkpoint
4snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')1import torch
2from PIL import Image
3from pyramid_dit import PyramidDiTForVideoGeneration
4from diffusers.utils import load_image, export_to_video
5
6torch.cuda.set_device(0)
7model_dtype, torch_dtype = 'bf16', torch.bfloat16 # Use bf16 (not support fp16 yet)
8
9model = PyramidDiTForVideoGeneration(
10 'PATH', # The downloaded checkpoint dir
11 model_name="pyramid_flux",
12 model_dtype,
13 model_variant='diffusion_transformer_768p',
14)
15
16model.vae.enable_tiling()
17# model.vae.to("cuda")
18# model.dit.to("cuda")
19# model.text_encoder.to("cuda")
20
21# if you're not using sequential offloading bellow uncomment the lines above ^
22model.enable_sequential_cpu_offload()1prompt = "A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors"
2
3# used for 384p model variant
4# width = 640
5# height = 384
6
7# used for 768p model variant
8width = 1280
9height = 768
10
11with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
12 frames = model.generate(
13 prompt=prompt,
14 num_inference_steps=[20, 20, 20],
15 video_num_inference_steps=[10, 10, 10],
16 height=height,
17 width=width,
18 temp=16, # temp=16: 5s, temp=31: 10s
19 guidance_scale=7.0, # The guidance for the first frame, set it to 7 for 384p variant
20 video_guidance_scale=5.0, # The guidance for the other video latent
21 output_type="pil",
22 save_memory=True, # If you have enough GPU memory, set it to `False` to improve vae decoding speed
23 )
24
25export_to_video(frames, "./text_to_video_sample.mp4", fps=24)1# used for 384p model variant
2# width = 640
3# height = 384
4
5# used for 768p model variant
6width = 1280
7height = 768
8
9image = Image.open('assets/the_great_wall.jpg').convert("RGB").resize((width, height))
10prompt = "FPV flying over the Great Wall"
11
12with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
13 frames = model.generate_i2v(
14 prompt=prompt,
15 input_image=image,
16 num_inference_steps=[10, 10, 10],
17 temp=16,
18 video_guidance_scale=4.0,
19 output_type="pil",
20 save_memory=True, # If you have enough GPU memory, set it to `False` to improve vae decoding speed
21 )
22
23export_to_video(frames, "./image_to_video_sample.mp4", fps=24)guidance_scale parameter controls the visual quality. We suggest using a guidance within [7, 9] for the 768p checkpoint during text-to-video generation, and 7 for the 384p checkpoint.video_guidance_scale parameter controls the motion. A larger value increases the dynamic degree and mitigates the autoregressive generation degradation, while a smaller value stabilizes the video.@article{jin2024pyramidal,
title={Pyramidal Flow Matching for Efficient Video Generative Modeling},
author={Jin, Yang and Sun, Zhicheng and Li, Ningyuan and Xu, Kun and Xu, Kun and Jiang, Hao and Zhuang, Nan and Huang, Quzhe and Song, Yang and Mu, Yadong and Lin, Zhouchen},
jounal={arXiv preprint arXiv:2410.05954},
year={2024}
}