You can deploy this model and start earning money today!

csbhr/
Vivid-VR

Name: Vivid-VR
Availability: InStock

Views

No views yet

Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration

📚 Paper | 🌐 Project Page | 💻 Code

For more quantitative results and visual results, go checkout our project page

🎬 Overview

🔧 Dependencies and Installation

Clone Repo

bash
1git clone https://github.com/csbhr/Vivid-VR.git 2cd Vivid-VR
Create Conda Environment and Install Dependencies

bash
1# create new conda env 2conda create -n Vivid-VR python=3.10 3conda activate Vivid-VR 4 5# install pytorch 6pip install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu121 7 8# install python dependencies 9pip install -r requirements.txt 10 11# install easyocr [Optional, for text fix] 12pip install easyocr 13pip install numpy==1.26.4 # numpy2.x maybe installed when installing easyocr, which will cause conflicts

Download Models

[Required] Download CogVideoX1.5-5B checkpoints from [huggingface].
[Required] Download cogvlm2-llama3-caption checkpoints from [huggingface].
- Please replace modeling_cogvlm.py with ./VRDiT/cogvlm2-llama3-caption/modeling_cogvlm.py to remove the dependency on pytorchvideo.
[Required] Download Vivid-VR checkpoints from [huggingface].
[Optional, for text fix] Download easyocr checkpoints [english_g2] [zh_sim_g2] [craft_mlt_25k].
[Optional, for text fix] Download Real-ESRGAN checkpoints [RealESRGAN_x2plus].
Put them under the ./ckpts folder.

The ckpts directory structure should be arranged as:

├── ckpts
│   ├── CogVideoX1.5-5B
│   │   ├── ...
│   ├── cogvlm2-llama3-caption
│   │   ├── ...
│   ├── Vivid-VR
│   │   ├── controlnet
│   │       ├── config.json
│   │       ├── diffusion_pytorch_model.safetensors
│   │   ├── connectors.pt
│   │   ├── control_feat_proj.pt
│   │   ├── control_patch_embed.pt
│   ├── easyocr
│   │   ├── craft_mlt_25k.pth
│   │   ├── english_g2.pth
│   │   ├── zh_sim_g2.pth
│   ├── RealESRGAN
│   │   ├── RealESRGAN_x2plus.pth

☕️ Quick Inference

Run the following commands to try it out:

shell

python VRDiT/inference.py \
    --ckpt_dir=./ckpts \
    --cogvideox_ckpt_path=./ckpts/CogVideoX1.5-5B \
    --cogvlm2_ckpt_path=./ckpts/cogvlm2-llama3-caption \
    --input_dir=/dir/to/input/videos \
    --output_dir=/dir/to/output/videos \
    --num_temporal_process_frames=121 \  # For long video inference, if video longer than num_temporal_process_frames, aggregate sampling will be enabled in the temporal dimension
    --upscale=0 \  # Optional, if set to 0, the short-size of output videos will be 1024
    --textfix \  # Optional, if given, the text region will be replaced by the output of Real-ESRGAN
    --save_images  # Optional, if given, the video frames will be saved

GPU memory usage:

For a 121-frame video, it requires approximately 43GB GPU memory.
If you want to reduce GPU memory usage, replace "pipe.enable_model_cpu_offload" with "pipe.enable_sequential_cpu_offload" in ./VRDiT/inference.py. GPU memory usage is reduced to 25GB, but the inference time is longer.
For the arg "--num_temporal_process_frames", smaller values require less GPU memory but increase inference time.

📧 Citation

If you find our repo useful for your research, please consider citing it:

bibtex

@article{bai2025vividvr,
   title={Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration}, 
   author={Bai, Haoran and Chen, Xiaoxu and Yang, Canqian and He, Zongyao and Deng, Sibin and Chen, Ying},
   journal={arXiv preprint arXiv:2508.14483},
   year={2025},
   url={https://arxiv.org/abs/2508.14483}
 }

📄 License

This repo is built based on diffusers v0.31.0, which is distributed under the terms of the Apache License 2.0.
CogVideoX1.5-5B models are distributed under the terms of the CogVideoX License.
cogvlm2-llama3-caption models are distributed under the terms of the CogVLM2 License and LLAMA3 License.
Real-ESRGAN models are distributed under the terms of the BSD 3-Clause License.
easyocr models are distributed under the terms of the JAIDED.AI Terms and Conditions.

csbhr/Vivid-VR