The QLoRA fine-tuning process of aoi_sola_lora_flux_nf4 takes inspiration from this post (https://huggingface.co/blog/flux-qlora). The training was executed on a local computer with 1200 timesteps and the same parameters as the link mentioned above, which took around 7.5 hours on 8GB VRAM 4060. The peak VRAM usage was around 7.7GB. To avoid running low on VRAM, both transformers and text_encoder were quantized. The biggest challenge of training Japanese actresses is their photos used heavy filters to whiten and smoothen the skin. This practise severely distorts the training images which makes the result less convincing than Hollywood actresses. All the images generated here are using the below parameters

Height: 512
Width: 512
Guidance scale: 5
Num inference steps: 20
Max sequence length: 512
Seed: 0

Usage

python

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from transformers import T5EncoderModel

text_encoder_4bit = T5EncoderModel.from_pretrained(
    "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="text_encoder_2",torch_dtype=torch.float16,)

transformer_4bit = FluxTransformer2DModel.from_pretrained(
        "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="transformer",torch_dtype=torch.float16,)

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.float16,
                                    transformer=transformer_4bit,text_encoder_2=text_encoder_4bit)

pipe.load_lora_weights("je-suis-tm/aoi_sola_lora_flux_nf4",
                       weight_name='pytorch_lora_weights.safetensors')

prompt="A close-up 35mm film photograph of Aoi Sola sitting in a dimly lit room.  A red off-shoulder dress with intricate gold embroidery hugs her body. One hand rests delicately on her knee as she gazes directly into the camera lens. A layered gold necklace and long red earrings add to her mystique. The warm colors of the room create a moody atmosphere, while the dark background subtly blends into the shadows. The subject is the main focus, with the background blurred just enough to emphasize her beauty. High contrast between light and dark creates dramatic shadows across her face and body, making her appear almost ethereal in the photograph."

image = pipe(
            prompt,
            height=512,
            width=512,
            guidance_scale=5,
            num_inference_steps=20,
            max_sequence_length=512,
            generator=torch.Generator("cpu").manual_seed(0),            
        ).images[0]

image.save("aoi_sola_lora_flux_nf4.png")

Trigger words

You should use Aoi Sola to trigger the image generation.

Download model

Download them in the Files & versions tab.