The QLoRA fine-tuning process of icespice_lora_flux_nf4 takes inspiration from this post (https://huggingface.co/blog/flux-qlora). The training was executed on a local computer with 1000 timesteps and the same parameters as the link mentioned above, which took around 6 hours on 8GB VRAM 4060. The peak VRAM usage was around 7.7GB. To avoid running low on VRAM, both transformers and text_encoder were quantized. All the images generated here are using the below parameters

Height: 512
Width: 512
Guidance scale: 5
Num inference steps: 20
Max sequence length: 512
Seed: 0

Usage

python

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from transformers import T5EncoderModel

text_encoder_4bit = T5EncoderModel.from_pretrained(
    "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="text_encoder_2",torch_dtype=torch.float16,)

transformer_4bit = FluxTransformer2DModel.from_pretrained(
        "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="transformer",torch_dtype=torch.float16,)

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.float16,
                                    transformer=transformer_4bit,text_encoder_2=text_encoder_4bit)

pipe.load_lora_weights("je-suis-tm/icespice_lora_flux_nf4",
                       weight_name='pytorch_lora_weights.safetensors')

prompt="Surreal 4k, a beautiful elf princess called IceSpice with expressive black eyes and cosmic features. Her skin appears to be composed of intertwined bioluminescent particles, surrounded by neon lights and floating, colorful orbs in a fantastical forest environment, with exotic animals creating a mesmerizing and otherworldly atmosphere, cinematic composition. full body portrait"

image = pipe(
            prompt,
            height=512,
            width=512,
            guidance_scale=5,
            num_inference_steps=20,
            max_sequence_length=512,
            generator=torch.Generator("cpu").manual_seed(0),            
        ).images[0]

image.save("icespice_lora_flux_nf4.png")

Trigger words

You should use IceSpice to trigger the image generation.

Download model

Download them in the Files & versions tab.