Llama-3-8B-NOLA is a fine-tuned variant of meta-llama/Meta-Llama-3-8B on OpenAssistant/oasst1
for 100 steps only. The goal of this experiment was to try out this new technique NOLA and see the the number of trainable parameters. Due to limited compute, the results of this experiment
might not be satisfactory. 1x A5000 was used for this experiment.
NOLA (Compressing LoRA using Linear Combination of Random Basis)
NOLA is a novel approach for fine-tuning large models such as LLMs and Vision Transformers. Similar to LoRA, NOLA uses a low-rank decomposition of weight matrices for the fine-tuning step. However, LoRA face two primary limitations:
The parameter count is lower-bounded by the rank one decomposition
The extent of reduction is heavily influenced by both the model architecture and the chosen rank.
NOLA brings parameter count felexiblity to LoRA. NOLA achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear
coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture.
