ToonCrafter_with_SketchGuidance This repository is an implementation that recreates the SketchGuidance feature of "ToonCrafter".
ToonCrafter with Sketch Guidance
Overview
This repository contains the ToonCrafter model with additional sketch guidance functionality. The model is hosted on GitHub and can be accessed via the following link:
Installation
Clone the Repository
To clone the repository and set up the environment, follow these steps:
1 git clone https://github.com/svjack/ToonCrafter_with_SketchGuidance && cd ToonCrafter_with_SketchGuidance
2 pip install -r requirements.txt
3 pip install --upgrade einops
4 pip install pyav == 13.1 .0
List Control Models
After cloning the repository, you can list the control models available:
Download Sketch Encoder
Download the sketch_encoder.ckpt file from the Hugging Face repository:
wget https://huggingface.co/Doubiiu/ToonCrafter/resolve/main/sketch_encoder.ckptCopy Sketch Encoder to Control Models
Move the downloaded sketch_encoder.ckpt file to the control_models directory:
cp sketch_encoder.ckpt control_modelsUsage
Run the Gradio App
Once the repository is cloned and the necessary files are in place, you can run the Gradio app using the following command:
This will start the Gradio interface, allowing you to interact with the ToonCrafter model with sketch guidance.
🧰 Models
Model Resolution GPU Mem. & Inference Time (A100, ddim 50steps) Checkpoint ToonCrafter_512 320x512 TBD (perframe_ae=True) Hugging Face SketchEncoder TBD TBD Hugging Face
Currently, ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.
⚙️ Setup
Install Environment via Anaconda (Recommended)
1 conda create -n tooncrafter python = 3.8 .5
2 conda activate tooncrafter
3 pip install -r requirements.txt
💫 Inference
1. Local Gradio demo
Download pretrained ToonCrafter_512 and put the model.ckpt in checkpoints/tooncrafter_512_interp_v1/model.ckpt.
Download pretrained SketchEncoder and put the model.ckpt in control_models/sketch_encoder.ckpt.
