LoRI (LoRA with Reduced Interference) is a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference.

Model Details

Model Description

LoRI (LoRA with Reduced Interference) is a novel parameter-efficient fine-tuning (PEFT) method designed to address the overhead and parameter interference often encountered in multi-task scenarios for Large Language Models (LLMs). By freezing projection matrices A as random projections and sparsifying matrices B with task-specific masks, LoRI significantly reduces trainable parameters while preserving strong task performance and minimizing cross-task interference. This approach also supports continual learning by leveraging sparsity to mitigate catastrophic forgetting.

This specific model, LoRI-D_code_llama3_rank_32, is a LoRI adapter based on the meta-llama/Meta-Llama-3-8B model, specifically fine-tuned for code generation tasks.

Developed by: Juzheng Zhang, Jiacheng You, Ashwinee Panda, Tom Goldstein
Model type: Parameter-Efficient Fine-Tuning (PEFT) adapter; a variant of Low-Rank Adaptation (LoRA).
Language(s) (NLP): English (as the primary language for the base model and tasks such as NLU, mathematical reasoning, code generation, and safety alignment).
License: Apache-2.0
Finetuned from model: meta-llama/Meta-Llama-3-8B

Model Sources

Repository: https://github.com/juzhengz/LoRI/
Paper: https://huggingface.co/papers/2504.07448
Hugging Face Collection: https://huggingface.co/collections/tomg-group-umd/lori-adapters-67f795549d792613e1290011

Uses

Direct Use

LoRI adapters are intended for efficient fine-tuning of Large Language Models on various downstream tasks, including:

Natural Language Understanding (NLU)
Mathematical Reasoning
Code Generation
Safety Alignment

They can be loaded as PEFT adapters on top of a base LLM to enhance performance with reduced parameter overhead.

Downstream Use

LoRI enables effective adapter merging for multi-task applications, allowing a single model to perform well across several distinct tasks without significant performance degradation. It also supports continual learning, facilitating model adaptation to new tasks sequentially while mitigating catastrophic forgetting.

Out-of-Scope Use

This model is not intended for generating harmful, biased, or unethical content. Users should ensure compliance with the ethical guidelines of the base models and apply appropriate content filtering. It is not suitable for standalone use without a compatible base LLM or for tasks outside its trained domains without further adaptation.

Bias, Risks, and Limitations

As an adapter for Large Language Models, LoRI inherits potential biases and limitations from its base model (meta-llama/Meta-Llama-3-8B) and the datasets it was fine-tuned on. Users should be aware that results might reflect biases present in the training data. While LoRI aims to reduce cross-task interference, complete elimination of such interference or catastrophic forgetting is not guaranteed. Performance on highly out-of-distribution tasks may be limited.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. It is recommended to perform further evaluation on specific target datasets and contexts to understand its performance and potential biases relevant to the application. Implement appropriate safeguards and content filtering when deploying the model in production environments.

How to Get Started with the Model

Use the code below to get started with the model.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load the base model
base_model_path = "meta-llama/Meta-Llama-3-8B"
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16, # Use torch.float16 if bfloat16 is not supported
    device_map="auto" # Distributes the model across available devices
)

# Load the LoRI adapter (this model is for code generation)
lori_adapter_path = "tomg-group-umd/LoRI-D_code_llama3_rank_32"
model = PeftModel.from_pretrained(base_model, lori_adapter_path)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_path)

# Example usage for code generation
prompt = "def factorial(n):
    if n == 0:
        return 1
    else:
        return n * "

# Apply chat template if using a chat-tuned base model (Llama-3 is chat-tuned)
# Or directly tokenize if the base model is not primarily chat-tuned
messages = [{"role": "user", "content": prompt}]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    input_ids,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.7,
    top_p=0.9
)

# Decode and print the generated response (excluding the prompt for cleaner output)
response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
print(response)

Training Details

LoRI is implemented using Fully Sharded Data Parallel (FSDP) and can be executed in a multi-GPU environment. Training scripts are provided for various tasks covering Natural Language Understanding (NLU), Code Generation, Mathematical Reasoning, and Safety Alignment.

Training Data

LoRI adapters were trained on datasets corresponding to various tasks:

Code generation tasks: CodeAlpaca
Mathematical reasoning tasks: GSM8K
Safety alignment tasks: Saferpaca
Natural Language Understanding (NLU) tasks: Specific NLU datasets (refer to the paper for a full list).

More details on these datasets can be found in the LoRI GitHub repository.

Training Procedure

LoRI training typically involves two main stages:

LoRI-D (Decomposition): Initial training where projection matrices A are frozen as random projections, and matrices B are trained. This stage is used to extract sparse masks.
LoRI-S (Sparsification): Continues training from LoRI-D, where matrices B are sparsified using task-specific masks (e.g., 90% sparsity for LoRI-S).

Training Hyperparameters

PEFT Type: LORA
LoRA Alpha (lora_alpha): 64
LoRA Rank (r): 32 (for this specific model LoRI-D_code_llama3_rank_32)
LoRA Dropout (lora_dropout): 0.05
Target Modules: q_proj, o_proj, k_proj, v_proj, down_proj, gate_proj, up_proj
Training Regime: Mixed precision (e.g., bfloat16 commonly used with Llama-3).
Optimization: Fully Sharded Data Parallel (FSDP)

Evaluation

Extensive experiments were conducted across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks. The paper demonstrates that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than standard LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code generation performance on HumanEval is evaluated using the bigcode-evaluation-harness. For detailed quantitative results, please refer to the paper.

Technical Specifications

Model Architecture and Objective

LoRI introduces a novel architecture built upon existing Transformer-based LLMs, adapting the standard LoRA approach. It achieves efficiency and interference reduction by:

Freezing the low-rank projection matrix A as random projections.
Sparsifying the low-rank projection matrix B using task-specific masks. The primary objective is to minimize cross-task interference in multi-task low-rank adaptation, while significantly reducing the trainable parameter count and maintaining strong performance.

Compute Infrastructure

The training and evaluation were performed on multi-GPU environments, leveraging technologies like FSDP. Experiments were conducted with LLaMA-3-8B and Mistral-7B base models.

Software

The implementation relies on PyTorch, transformers, and peft libraries. It builds on the codebase of dpo-rlaif and incorporates code from lottery-ticket-adaptation. Evaluation for code generation uses the bigcode-evaluation-harness.

Citation

If you use LoRI in your work, please cite:

bibtex

@article{zhang2025lori,
  title={LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation},
  author={Zhang, Juzheng and You, Jiacheng and Panda, Ashwinee and Goldstein, Tom},
  journal={arXiv preprint arXiv:2504.07448},
  year={2025}
}

More Information

For further details on installation, training from scratch, adapter merging, continual learning, and customizing base models/losses, please refer to the official GitHub repository.

Framework versions

PEFT 0.12.0