Loras extracted from finely tuned checkpoints. I did it primarily for myself because it’s easier to work with Lora, including mixing them in real time and setting different weights/strengths. It also saves space on the local computer. Choose a rank according to your taste and the capabilities of your hardware (the lower the rank, the less memory you need). After extraction, the weights were normalized. In most cases, ranks of 16, 32, or 64 will be sufficient. There is a correlation between rank and the acceptable Lora strength range. The strength range will be roughly the same for different ranks, but it also depends on the specific model: r8 80-120%; r32 80-120%; r64 80-160%; r128 100-160%
A lot of time, electricity, and compute went into this on my small and not particularly powerful Mac Mini M4. But it was an interesting research project.
