warp-ai/

wuerstchen-prior-model-interpolated

Würstchen is a diffusion model, whose text-conditional model works in a highly compressed latent space of images. Why is this important? Compressing data can reduce computational costs for both training and inference by magnitudes. Training on 1024x1024 images is way more expensive than training on 32x32. Usually, other works make use of a relatively small compression, in the range of 4x - 8x spatial compression. Würstchen takes this to an extreme. Through its novel design, we achieve a 42x spatial compression. This was unseen before because common methods fail to faithfully reconstruct detailed images after 16x spatial compression. Würstchen employs a two-stage compression, what we call Stage A and Stage B. Stage A is a VQGAN, and Stage B is a Diffusion Autoencoder (more details can be found in the paper). A third model, Stage C, is learned in that highly compressed latent space. This training requires fractions of the compute used for current top-performing models, allowing also cheaper and faster inference.

The Prior is what we refer to as "Stage C". It is the text-conditional model, operating in the small latent space that Stage A and Stage B encode images into. During inference, its job is to generate the image latents given text. These image latents are then sent to Stages A & B to decode the latents into pixel space.

The interpolated model is our current best Prior (Stage C) checkpoint. It is an interpolation between our base model and the finetuned model. We created this interpolation because the finetuned model became too artistic and often only generates artistic images. The base model, however, usually is very photorealistic. As a result, we combined both by interpolating their weights by 50%, so the middle between the base and finetuned model (0.5 * base_weights + 0.5 * finetuned_weights). You can also interpolate the base model and the finetuned model as you want and maybe find an interpolation that fits your needs better than this checkpoint.

Würstchen was trained on image resolutions between 1024x1024 & 1536x1536. We sometimes also observe good outputs at resolutions like 1024x2048. Feel free to try it out. We also observed that the Prior (Stage C) adapts extremely fast to new resolutions. So finetuning it at 2048x2048 should be computationally cheap.

This pipeline should be run together with https://huggingface.co/warp-ai/wuerstchen:

Würstchen v2 Estimated Emissions Based on that information, we estimate the following CO2 emissions using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.