LoRI (LoRA with Reduced Interference) is a simple yet effective variant of LoRA for fine-tuning Large Language Models (LLMs). It addresses notable overhead and parameter interference in multi-task scenarios by freezing the projection matrices A as random projections and sparsifying the matrices B using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: this https URL

Model Details

Model Description

LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. This particular model, LoRI-S_nlu_llama3_rank_32, is an adapter fine-tuned for Natural Language Understanding (NLU) tasks, based on the Llama-3-8B model with an adapter rank of 32. It is an "LoRI-S" (Sparsified) adapter, indicating it has undergone an additional sparsity-inducing training stage after initial LoRI-D training.

Developed by: Juzheng Zhang, Jiacheng You, Ashwinee Panda, Tom Goldstein
Model type: Low-Rank Adaptation (LoRA) variant, Parameter-Efficient Fine-Tuning (PEFT) method for Causal Language Models
Language(s) (NLP): English
License: Apache 2.0
Finetuned from model: meta-llama/Meta-Llama-3-8B

Model Sources

Repository: https://github.com/juzhengz/LoRI/
Paper: https://huggingface.co/papers/2504.07448 (arXiv:2504.07448)
Project Page: https://juzhengz.github.io/
Hugging Face Collection: https://huggingface.co/collections/tomg-group-umd/lori-adapters-67f795549d792613e1290011

Uses

Direct Use

This model is intended to be used as a PEFT adapter for the meta-llama/Meta-Llama-3-8B base model to enhance its performance on Natural Language Understanding tasks. It allows for efficient fine-tuning and deployment of large language models without modifying the original base model's weights.

Downstream Use

LoRI adapters are designed to minimize cross-task interference, enabling effective adapter merging for multi-task scenarios and supporting continual learning by mitigating catastrophic forgetting. This makes them suitable for a wide range of downstream applications across NLU, code generation, mathematical reasoning, and safety alignment.

Out-of-Scope Use

As an adapter, this model is not intended for standalone use; it requires a compatible base language model (e.g., meta-llama/Meta-Llama-3-8B). Users should also be mindful of the ethical implications and potential biases inherited from the base model and training data when deploying in sensitive applications.

Bias, Risks, and Limitations

LoRI, like any LLM or PEFT method, may inherit biases present in its foundational Meta-Llama-3-8B model and the datasets it was fine-tuned on. While LoRI is designed to reduce cross-task interference and mitigate catastrophic forgetting, these are complex challenges in AI, and complete elimination cannot be guaranteed, especially in highly diverse or long-term continual learning scenarios.

Recommendations

Users should perform thorough evaluations for their specific use cases to identify and mitigate potential biases. Continuous monitoring for performance degradation or unintended behaviors is advised, particularly in critical applications. It is recommended to consult the original paper for detailed experimental results and discussions on limitations.

How to Get Started with the Model

Use the code below to get started with the model using the transformers and peft libraries. This example demonstrates loading the LoRI-S_nlu_llama3_rank_32 adapter and performing basic text generation.

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load the base model
base_model_name = "meta-llama/Meta-Llama-3-8B"
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16, # or torch.float16, depending on your hardware
    device_map="auto"
)

# Load the LoRI adapter
adapter_name = "tomg-group-umd/LoRI-S_nlu_llama3_rank_32" # This specific model
model = PeftModel.from_pretrained(base_model, adapter_name)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

# Example inference for text generation
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=20, temperature=0.7)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(output_text)

# Example for chat completion (if the base model supports chat templates)
# messages = [
#     {"role": "user", "content": "Explain the concept of quantum entanglement."},
# ]
# text = tokenizer.apply_chat_template(
#     messages,
#     tokenize=False,
#     add_generation_prompt=True
# )
# inputs = tokenizer(text, return_tensors="pt").to(model.device)
# output_ids = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
# output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
# print(output_text)

Training Details

Training Data

LoRI models were extensively evaluated across various domains and tasks. For the "LoRI-S_nlu_llama3_rank_32" adapter, it was fine-tuned for Natural Language Understanding (NLU) tasks. Other LoRI adapters in the collection were trained for:

Code Generation: Using datasets like CodeAlpaca.
Mathematical Reasoning: Using datasets like GSM8K.
Safety Alignment: Using datasets like Saferpaca.

Further details on specific datasets used for NLU tasks can be found in the original paper and the LoRI GitHub repository.

Training Procedure

LoRI training involves a two-stage process implemented using Fully Sharded Data Parallel (FSDP) for multi-GPU environments:

LoRI-D (Density/Distribution) training: Initial training where the projection matrices A are frozen as random projections.
LoRI-S (Sparsification) training: Continues training from the LoRI-D stage, where matrix B is sparsified using task-specific masks, typically at 90% sparsity.

Training Hyperparameters

Base Models: LLaMA-3-8B and Mistral-7B were used across experiments.
Adapter Ranks (r): 32 (for this model) and 64.
LoRA Alpha (lora_alpha): 64 (from adapter_config.json).
LoRA Dropout (lora_dropout): 0.05 (from adapter_config.json).
Sparsity: Typically 90% for LoRI-S adapters.
Training regime: Mixed precision (e.g., bf16).

Evaluation

Testing Data, Factors & Metrics

LoRI was evaluated across natural language understanding, mathematical reasoning, code generation (e.g., HumanEval), and safety alignment tasks. The paper presents comprehensive results comparing LoRI against full fine-tuning and other PEFT methods.

Results

Extensive experiments demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than standard LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. For detailed quantitative results, please refer to the official paper.

Technical Specifications

Model Architecture and Objective

LoRI modifies the standard LoRA architecture. The A matrices are frozen as random projections, and the B matrices are sparsified using task-specific masks. This design aims to promote monosemanticity (where adapters specialize in specific knowledge) and reduce interference between different adapters, especially in multi-task and continual learning settings. The objective remains to efficiently fine-tune the LLM for various downstream tasks.

Compute Infrastructure

The training scripts are designed to leverage multi-GPU setups using PyTorch's Fully Sharded Data Parallel (FSDP) for efficient distributed training.

Software

Python 3.10
PyTorch
Transformers library
PEFT library (version 0.12.0)

Citation

If you use LoRI in your work, please cite:

bibtex

@article{zhang2025lori,
  title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
  author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
  journal={arXiv preprint arXiv:2504.07448},
  year={2025}
}

Model Card Contact

For questions, feel free to reach out via the LoRI GitHub repository or contact the authors of the paper.

Framework versions

PEFT 0.12.0

juzhengz/LoRI-S_nlu_llama3_rank_32

Model Card for LoRI-S_nlu_llama3_rank_32