Views
No views yet
| 10s, 768p, 24fps | 5s, 768p, 24fps | Image-to-video |
|---|---|---|
COMING SOON ⚡️⚡️⚡️ Training code and new model checkpoints trained from scratch.2024.10.10 🚀🚀🚀 We release the technical report, project page and model checkpoint of Pyramid Flow.1from huggingface_hub import snapshot_download
2
3model_path = 'PATH' # The local directory to save downloaded checkpoint
4snapshot_download("rain1011/pyramid-flow-sd3", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')video_generation_demo.ipynb at this link. We further simplify it into the following two-step procedure. First, load the downloaded model:1import torch
2from PIL import Image
3from pyramid_dit import PyramidDiTForVideoGeneration
4from diffusers.utils import load_image, export_to_video
5
6torch.cuda.set_device(0)
7model_dtype, torch_dtype = 'bf16', torch.bfloat16 # Use bf16, fp16 or fp32
8
9model = PyramidDiTForVideoGeneration(
10 'PATH', # The downloaded checkpoint dir
11 model_dtype,
12 model_variant='diffusion_transformer_768p', # 'diffusion_transformer_384p'
13)
14
15model.vae.to("cuda")
16model.dit.to("cuda")
17model.text_encoder.to("cuda")
18model.vae.enable_tiling()1prompt = "A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors"
2
3with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
4 frames = model.generate(
5 prompt=prompt,
6 num_inference_steps=[20, 20, 20],
7 video_num_inference_steps=[10, 10, 10],
8 height=768,
9 width=1280,
10 temp=16, # temp=16: 5s, temp=31: 10s
11 guidance_scale=9.0, # The guidance for the first frame
12 video_guidance_scale=5.0, # The guidance for the other video latent
13 output_type="pil",
14 )
15
16export_to_video(frames, "./text_to_video_sample.mp4", fps=24)1image = Image.open('assets/the_great_wall.jpg').convert("RGB").resize((1280, 768))
2prompt = "FPV flying over the Great Wall"
3
4with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
5 frames = model.generate_i2v(
6 prompt=prompt,
7 input_image=image,
8 num_inference_steps=[10, 10, 10],
9 temp=16,
10 video_guidance_scale=4.0,
11 output_type="pil",
12 )
13
14export_to_video(frames, "./image_to_video_sample.mp4", fps=24)guidance_scale parameter controls the visual quality. We suggest using a guidance within [7, 9] for the 768p checkpoint during text-to-video generation, and 7 for the 384p checkpoint.video_guidance_scale parameter controls the motion. A larger value increases the dynamic degree and mitigates the autoregressive generation degradation, while a smaller value stabilizes the video.@article{jin2024pyramidal,
title={Pyramidal Flow Matching for Efficient Video Generative Modeling},
author={Jin, Yang and Sun, Zhicheng and Li, Ningyuan and Xu, Kun and Xu, Kun and Jiang, Hao and Zhuang, Nan and Huang, Quzhe and Song, Yang and Mu, Yadong and Lin, Zhouchen},
jounal={arXiv preprint arXiv:2410.05954},
year={2024}
}