Experimental conversion of SDXL architecture to 16 channel latent space
This is an experimental pretrain on top of Rouwei-0.8 that works with 16 channel latent space and uses Flux ae.
Goals:
Achieve better details while maintaining low compute requirements and all existing knowledge and performance
Possibility of joint sampling with Flux/Chroma/Lumina and other models with same latent space
Current state:
Early alpha version, it is pretty raw. Images may contain extra noise and have artifacts in small details, level varies from neglectable to significant. Upscale, samplers/schedulers, styles, even prompt affect it.
Use of GAN upscale models in pixels space instead of latent upscale gives much smoother results, bumping base resolution higher helps too.
Epsilon prediction now, can be converted to vpred or anything in future.
Download these nodes (or just use "install missing nodes" using Comfy Manager)
Use SDXL 16ch loader node to load it, then work just like you used to with sdxl
DO NOT REMOVE Latent multyply NODES, latents should be scaled before and after processing just same as in regular SDXL inference. This step just isn't hidden yet.
If you're getting error mat1 and mat2 shapes cannot be multiplied (_x16 and 4x3) - disable the preview option for Ksampler. It happens because preview uses taesd vae designed for 4channel.
Other UI
Since the main difference is just shapes of tensors, used vae and latents scaling factor - it should be easy to implement support to any other UI.
Lora adapters, controlnet, ip-adapters, other things untested.
Joint sampling
Since the model operates in 16channel latent space similat to Flux, Chroma, Limina-image and some other, you can implement comlex workflows (if you have enough memory). This allows to utilize all knowlege of characters, styles, concepts from RouWei along with the performance of bigger models.
Here is an example workflow. Using just few (1..4) steps from Flux you create some rough basic composition. Then the latents come to 16channel sdxl model where denoised (skipping initial high noise timesteps).
It is the most simple approach, since you don't need to reconvert latents though series of vae or some adapters, you can change models on every denoising step without having any performance impact.
Just don't forget to upply Latenst multiply nodes.
Basically, no changes to default architecture. Just reinitializing if input and output layers to new size, then training with gradual unfreezing of blocks towards the middle.
Default SDXL latent scale factor of 0.13025 doesn't work well here, 0.6 is used for this release.
This is not the most optimal approach. Some changes to the outer layers of the model instead of direct use 'as is' should give improvement in future. If you have any thoughts or ideas about it - please share them.
Training
To train it (in current version) all you need is to change the number of in/out channels in UNET config and set scale factor to 0.6 instead of 0.13025. And probably check vae part to work properly.
(Code examples later)
I'm willing to help/cooperate:
Join Discord server where you can share your thoughts, proposals, requests, etc. Write to me directly here, on civitai or dm in discord.
Thanks:
Part of training was performed using google TPU and sponsored by OpenRoot-Compute
Personal: NeuroSenko
And many thanks to all fellow brothers who supported me before.
