Views
No views yet
vanessa_kirby_lora_flux_nf4 takes inspiration from this post (https://huggingface.co/blog/flux-qlora). The training was executed on a local computer with 1000 timesteps and the same parameters as the link mentioned above, which took around 6 hours on 8GB VRAM 4060. The peak VRAM usage was around 7.7GB. To avoid running low on VRAM, both transformers and text_encoder were quantized. All the images generated here are using the below parameters1import torch
2from diffusers import FluxPipeline, FluxTransformer2DModel
3from transformers import T5EncoderModel
4
5text_encoder_4bit = T5EncoderModel.from_pretrained(
6 "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="text_encoder_2",torch_dtype=torch.float16,)
7
8transformer_4bit = FluxTransformer2DModel.from_pretrained(
9 "hf-internal-testing/flux.1-dev-nf4-pkg", subfolder="transformer",torch_dtype=torch.float16,)
10
11pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.float16,
12 transformer=transformer_4bit,text_encoder_2=text_encoder_4bit)
13
14pipe.load_lora_weights("je-suis-tm/vanessa_kirby_lora_flux_nf4",
15 weight_name='pytorch_lora_weights.safetensors')
16
17prompt="Vanessa Kirby in a white kimono of cherry blossom patterns"
18
19image = pipe(
20 prompt,
21 height=512,
22 width=512,
23 guidance_scale=5,
24 num_inference_steps=20,
25 max_sequence_length=512,
26 generator=torch.Generator("cpu").manual_seed(0),
27 ).images[0]
28
29image.save("vanessa_kirby_lora_flux_nf4.png")Vanessa Kirby to trigger the image generation.